MUFF WIGGLER Forum Index

 FAQ & Terms of UseFAQ & Terms Of Use   Wiggler RadioMW Radio   Muff Wiggler Blog & NewsMW News   Muff Wiggler StoreMW Store 
 SearchSearch   RegisterSign up   Log inLog in 
WIGGLING 'LITE' IN GUEST MODE

Why not 'merge' all these planners?
MUFF WIGGLER Forum Index -> RackPlanner Goto page 1, 2  Next [all]
Author Why not 'merge' all these planners?

synthcube

This is pretty solid functionality
https://www.muffwiggler.com/forum/viewtopic.php?p=956054#956054

Why not marry rack planner content with this web functionality ? Then the larger merged team of folks can work together to improve a single solution rather than reinventing?

I'll even volunteer to play project manager and communication facilitator if it gives more people web tools for module planning, etc.


dougcl

The most important issue at this moment, in my opinion, is that the module zips that have already been established be honored, with an eye toward carefully extending the xml in them if necessary. Because this is so important, I have established a schemas thread here in anticipation of this situation. Further, it would be best if all module planners used the same filenames to represent the same modules. For this reason, it is my hope that developers follow the lead set by bananaplug on the squiggletronics library.

All developers should permit import and export of modules.

Assuming we get this far (voluntary coordination? What are the chances?) the next thing to honor is the rack xml format that has already been established.

After that, there is a patch xml format that Richy Ho (I hope) will establish.

But first, can we see if everyone is on board with getting the module zips from bananaplug set as the standard?

Then John Noble merges in his additional attributes into the existing module zips?

I am not going to force anyone to do anything. Not interested. I have RackPlanner for my own uses, so I am happy. It is up to the developers involved to move forward together. I can help with that if needed. I see that happening in the schemas thread here. I can add attributes and assist with reconciliation of objectives.

Doug


synthcube

Thank you Doug
That is exactly the sort of collaboration I think (hope) this group is capable of... The best of everyone's contributions, combined.
Again, I will volunteer to spend my own time to help coordinate whatever meetings, conference calls or whatever is necessary to hash together the best solution. I can't contribute anything technically, but I can offer some project management and facilitation experience.
Simple math, really. Everyone has contributed module zips in all formats that are mostly complete. Desired functionality is clearly documented in at least four separate threads and some really talented folks have contributed a lot of thought to individual solutions. If this group were collaborating, every single existing and new synth user would gain.


a scanner darkly

In my opinion collaboration should be focused on 2 areas: 1) format standards 2) centralized storage. When you have those it doesn't matter that you have multiple developers and apps, as long as they can be used together and interchangeably. It's not the case of building one "ideal" app, but rather apps that might have little overlap in functionality or target different use cases and platforms.

(kinda like modular, we have particular standards that are supported by multiple developers - I doubt we'd have a better system if they all worked on one ideal solution :-)

I think it's already happening (and big thanks to dougcl for creating and maintaining the standard and to bananaplug and John Noble for supporting it). These are exciting times and hopefully we'll see more apps coming out soon (hopefully, mine included) particularly RichyHo's upcoming RackPlanner.


dougcl

a scanner darkly wrote:
2) centralized storage.


I value interoperability over centralization. The main thrust of RackPlanner was to make centralization optional through the use of a standardized, open module zip format. This removes the bottleneck of central management that plagued the previous planner.

But if centralization is all we can get, it's better than every developer writing their own database (seems to be happening). A central database that accepts user contributed modules and offers interfaces to client apps would be the way to go if my original idea of interoperability (as stated in my post above) proves unlikely.

I really think a decentralized system is the best way forward.


John Noble

I'm late to this, but here's my stance:

Export formats don't matter to me, within reason. If there are enough consumers to justify the trivial effort, I'll put together a file download tool for them. I suspect most of the cool kids are going to want JSON these days.

Import formats have no value for me since I don't trust the data unless it comes from an authoritative source. That source is frequently emails from the makers themselves and isn't available on the net.

If someone needs me to modify the existing RP export and can assure me that it doesn't break anything for existing users, I'll happily do it.

I'd rather pour bleach in my eyes than take on another pin-the-tail-on-the-donkey data merge project--I've done far too many of them over the years and the magic is gone. hihi Besides, I revise data fairly frequently so the merge would quickly go stale.

Feel free to PM or email when interoperability discussions come up. thumbs up


BananaPlug

The ModuleLibrary was created initially for RackPlanner zips. We could store additional module data there too. Anybody can contribute content to it and I'm open to suggestions. I hardly ever hear from anybody about how they use it or what they want. Doug says he misses the old everything-at-once interface so I'll bring that back as an alternate view.

I've reached out to some of the people working on planners and have offered an API, facilities for associating additional data with modules, etc. I'm open to that. Folks seem more interested in doing their own thing even if that means reinventing a wheel or two in the process. Collaboration is not always the easiest thing to do. I get that. But one of the simplest ways to collaborate is to agree on data formats and build interfaces to those specifications so different systems can talk to each other.

If anybody reading this would like to curate a portion of the library, working on consistency, etc. I would love to hear from you.

Thanks.


synthcube

I'll be the first to admit that a lot of the technical details are outside my domain, but it does seem to me that
a) lots of people like to experiment with web configurators and there are multiple 'sources' of basic module data, some more complete than others
b) no one has better access to rapidly changing module data than designers/manufacturers but they lack a common standard against which to supply and update data for users
c) format-centric solutions seem to take on their own lives-- but a simplified, common cross-format solution would benefit the entire synth community
d) 'best practices' exist in many of the solutions, but no overall architecture has been defined, nor has it been sorted out whcih features/functions ect comprise a consensus requirements definition


dougcl

BananaPlug wrote:

If anybody reading this would like to curate a portion of the library, working on consistency, etc. I would love to hear from you.


I am happy to help any way I can.


synthcube wrote:
I'll be the first to admit that a lot of the technical details are outside my domain, but it does seem to me that
a) lots of people like to experiment with web configurators and there are multiple 'sources' of basic module data, some more complete than others
b) no one has better access to rapidly changing module data than designers/manufacturers but they lack a common standard against which to supply and update data for users
c) format-centric solutions seem to take on their own lives-- but a simplified, common cross-format solution would benefit the entire synth community
d) 'best practices' exist in many of the solutions, but no overall architecture has been defined, nor has it been sorted out whcih features/functions ect comprise a consensus requirements definition


Please download RackPlanner and see how it supports all formats and uses the modules downloaded from the squiggletronics site. This should make it pretty clear what the current status is. I mention this as a starting point because it is the only case of a planner and database run separately and held together by an interface standard.

We have several things going on at the moment:
1) RichyHo: Offline planner using RackPlanner modules. All formats. Still not released, appears to be using a custom database in addition to RackPlanner standards. (RackPlanner does not specify standards for patch planning). Java only (no iPad or iPhone). No functioning patch planning software has hit the streets as of this writing, so standards not developed.
2) Modulargrid.net Online planner using custom database. Eurorack only. RackPlanner XML export, no import. User contributed. HTML5, iPhone, iPad supported. Leans toward user contributed database (at the moment). Quickly becoming a duplicate of eurorackdb.com.
3) Eurorackdb. Online database for eurorack only, no planner. Supports RackPlanner zip export, no import. Has more data on euro modules than squiggletronics. Moderated user contributions. Leans toward central management.
4) Squiggletronics (aka ModuleLibrary): Online database for all formats. Moderated user contributions. Exports RackPlanner zip standard. Leans toward user contributed content. Largest module collection available.
5) RackPlanner: offline planner supporting all formats. Unmoderated file based database. Java only (no iPad or iPhone).
6) There is another online planner/database that is scooping data from eurorackdb.com. I don't believe this activity is leading to an open interface or standards development. Appears to be a siloed effort. I think this one is HTML5 (iPhone/iPad supported).
7) ModularPlanner. Online Flash based with centrally managed database. Eurorack only. No iPad, iPhone. No user contributions. Project is on hold, apparently.


a scanner darkly

Leaving rack and patch definitions aside for a minute I think there is a need for a centralized "official" module definition source. Otherwise we'll end up with multiple versions of same modules and everybody maintaining their own versions, especially once we start adding more stuff to module definitions.

To me ModuleLibrary is such source, with eurorackdb taking care of the Eurorack part (because John makes the effort of maintaining that db up to date and like he said, he gets information directly from manufacturers). I think that there is nothing wrong with having one or two "official" source with possible mirrors (or one official source for Buchla, one for Serge and so on). Say, I want to add stuff that is very specific to my app and therefore there is no point in adding it to official definition - well, if that's the case then I'd be mad to recreate the whole thing. If I was in that position I would definitely take advantage of ModuleLibrary and eurorackdb (with BananaPlug and John's permissions, of course) and take care of somehow synchronizing my stuff to the official sources.

Think of this centralized storage as a reference library. I think there is a need for having "approved" module definitions that everybody could refer to (actually it would be great to have a field in module schema for "official" approval stamp that would specify that definition has been approved by the maker as having correct information).

Such centralized library could provide an API to retrieve module definition or to submit it - some of that functionality already exists in both ModuleLibrary and eurorackdb, and I talked to some of you about it as well.

As for other apps that are just coming out i haven't had a chance to check them out but I hope they re-use the existing RP files instead of creating a Babylon of module formats (looks like Doug already checked that while I was writing this). It's really up to good will of developers, I try to coordinate my stuff, I hope others do to

In terms of collaboration other than coordinating the formats I doubt it's that possible unless there is interest and trust from all participating parties (and who knows, we might have some open source projects down the road) but there is multitude of platforms and tools used which would make it hard for that to happen. If anything, realistically the only way to collaborate on development would be by releasing libraries, preferably open source, creating building blocks for apps. I could see writing something for Processing - but that would be in a pretty distant future for me personally, I have trouble finding time to work on PatchPad at the moment. However I will absolutely support and help with any efforts on coordinating format definitions and libraries and API for submitting/retrieving module definitions, and I already talked to some of you guys about that.


dougcl

The only thing that needs to be centralized is a shared module identifier. If everyone agrees that the 4MS Pedals Noise Swash should be identified as follows:
Manuf: 4MS
Model: Noise Swash
Format: E
Modifier: <null>

Then the data can be distributed in a hundred places. Applications could combine data from many sources if necessary. The problem reduces down to reliably identifying modules no matter who is hosting the data. This is good because one database might be great for euro, another for Serge, etc. This is already the case with euro in eurorackdb as you point out.

So two things are needed
1) A desire to coordinate module identification
2) An agreed data exchange format (module.xml)

I don't think this is that hard. Especially since we are talking about just a few developers with a shared interest. Squiggletronics has the broadest database available. Key identifiers are published in all the modules. If everyone would just conform to what has been done there, then we're more or less done, because that would take care of the identifiers and the adoption of the Module XML appears to be happening already.


a scanner darkly

I'm still not clear though on how would you deal with multiple versions of the same module definition? Leave it up to user to choose where to get the files? I guess you could also give the ability to specify multiple sources in order of preference (for downloading new modules). I guess online planners would have to choose what they use as their main source (or maintain their own...)


dougcl

a scanner darkly wrote:
I'm still not clear though on how would you deal with multiple versions of the same module definition? Leave it up to user to choose where to get the files? I guess you could also give the ability to specify multiple sources in order of preference (for downloading new modules). I guess online planners would have to choose what they use as their main source (or maintain their own...)


It's okay to have the same module defined in a hundred places. The problem is if that module has a hundred different identifiers.

Ideally we would have an industry wide identifier like an ISBN number that every manufacturer assigns to each module. But that isn't going to happen.

Instead, we have all the modules uniquely identified in the squiggletronics database, and the identifiers are published in the XML. John has the same modules defined in his system, but they have different identifiers. Modulargrid.net now has the same modules again with yet different identifiers.

STOP.

Use the squiggletronics "standard" for the identifiers. Start thinking about working together. There will be an identifier problem when a new module is created. Solve that problem together.

One answer would be to agree that squiggletronics be the only place online for users to create modules. Other databases would refer to squiggletronics for identifiers when modules are established locally. If this is followed, then nothing is wasted. Applications can browse the best of breed data across all the available databases.

There is value in having many databases, with each having a unique data focus, reflecting the particular interest of the guy running it. Shared identifiers supports this vision.


BananaPlug

Quote:
I think there is a need for a centralized "official" module definition source. Otherwise we'll end up with multiple versions of same modules and everybody maintaining their own versions

Quote:
"approved" module definitions that everybody could refer to (actually it would be great to have a field in module schema for "official" approval stamp that would specify that definition has been approved by the maker as having correct information).


There's a human element to this. The module's are often contributed to the library by non-official sources. The current version of the Squiggletronics Module Library continues the free access idea while adding a provision for interested parties, possibly the builder, to become registered users with editing capabilities. This approach allows lots of people to help with the initial data entry while also allowing people with some authority to make corrections. I haven't done a good job of recruiting moderators.

Official sources can get involved in a variety of ways. BugBrand Tom gave me a set of uniform images named by model number. I'll be importing these to replace the current set of his modules (I want to write a handy tool to streamline this process for moderators). The ones the users contributed have been doing the job for me and having that happen by itself was great for Tom. Makers have a lot on their plate.

Quote:
Manuf: 4MS
Model: Noise Swash
Format: E
Modifier: <null>

This naming spec which has evolved is not as wonderful as an ISBN number but it has some advantages. New stuff is coming in all the time, people use different names (Cynthia vs Cyndustries), makers put out a new version of something but don't name it as such, etc. In spite of all that, if you put them in a list people can generally find what they want. Within the ModuleLibrary each module has a unique ID. These IDs are not part of the XML spec but are necessary for internal housekeeping and will be exposed through the API. If an app used the API to get module data from the library it has the option of storing those ID numbers. If later the module in the library was renamed (there are reasons) that app could access the module by number and thus get updated. If we are tidying up the library, find several versions of a module and cull one or two of them, the app could use the Manuf, Model, Format, Modifier fields to locate the alternates to the module which was removed. Both approaches have their place.


a scanner darkly

There is definitely a lot of value in allowing anybody to enter new modules. I think a wiki like system would really be ideal, where people could also edit it afterwards and there would be a history of revisions. Adding new modules however should be moderated, exactly to avoid having multiple versions of same module. This would also give the freedom for makers to participate as much as they would like (they could have special "maker" status, I think there is a concept of "trusted" user in Wikipedia? not sure but you get the idea).

As for the naming convention my preference (as a potential consumer of the API) would be the unique ID (especially if it's exposed by the API) and a method that would return list of makers for given format and a method that would return a list of all modules for given maker and format, including module names and unique ID. Then I could cache that info and periodically check if there is a new version of a specific module available (using the unique ID) or if there are new modules added since specified date.


solitud

Hi, I am the modulargrid guy,

I saw this thread late but I wanted to drop in with some thoughts.

I am in modulars since one year only and I developed the idea and website very quick, without doing any big researches. If I had done some researches I probably would not have made the site because of the many other tools available. That would have put me off.

So I started fresh from scratch without thinking about apis, xmls, exchange whatsoever and focused on the user interface.

The simple idea: provide a planer with content from the users that can be instantly updated. Provide additional dynamic living data like matching youtube movies, statistics about favorite modules, etc.

I have received many emails and comments since the launch last friday, with mostly positive feedback, the not so good ones came with considerations about unsufficient compatibility to existing systems (which I didn´t know that those even exist).

I didn´t used content from the existing platforms and databases, because I thought it would be unfair/stealing.
But I recognized that people here are very open minded and want to share content and thats a good thing I would like to participate.

BUT:
I found out that the needs of the offline planers differ from the needs of web based community apps. It´s not so much formats, it´s structure.

The modulargrid has received 350+ module submits in three days, the quality of the submits differs strongly. There are several double submits of modules, diy panels, graphics bad, spelling mistakes, wrong assignments, etc.
Astonishingly people like to upload content to complete their racks.

That means content on the site is CHAOTIC but this is the way it has to be to get a vivid web community going.

To make data exchange easier I agree with Doug there has to be a kind of logical unique identifier for the modules, but even in this case I probably can not deliver "clean" data.

I CAN deliver well formated data of any kind and I am glad to do it, if someone is interested!

Knut


dougcl

solitud wrote:

Hi, I am the modulargrid guy,


Sorry it got complicated. What you are doing is great. As it turns out, a lot of what you are going through has been dealt with already at squiggletronics. They also have a module builder for users and it's been up for years. There has been a lot of effort already to clean those submissions. The work is ongoing. If there were a way for you to turn off your module builder and instead direct people to squiggletronics to build modules, you could avoid some hassle of data management. If you populated your database using squiggletronics modules, taking only what fields you want (probably just a few things), your database would be pre-populated with moderated module data. Your users could then add information (as many new attributes as you care to support) to those modules on your site leaving the identifiers intact. For this, a REST query to squiggletronics daily to check for new modules could keep your system pre-populated with clean (moderated) data. I believe bananaplug is prepared (or preparing) to offer these services to you or anyone else. Please contact him if you are interested.

You may want to have your own garbage dump to let users do their thing with DIY modules. But that's different I think, because it does not require moderation.

Worst case, nothing happens, and we just keep going the way we are going. I think that's okay but it means a lot of work for you that never ends. That might get old.

For now I would ignore the needs of the offline planners. We are doing just fine with squiggletronics. The other issue here is more important.


solitud

Sorry for late reply, have a lot to do, modulargrid bugfixes, daytimejob, impatient girlfriend, you name it.

dougcl wrote:

If there were a way for you to turn off your module builder and instead direct people to squiggletronics to build modules, you could avoid some hassle of data management.

No, this would break the whole concept of the site.

dougcl wrote:

If you populated your database using squiggletronics modules, taking only what fields you want (probably just a few things), your database would be pre-populated with moderated module data.

This would be a cool thing. I have already a mechanism where every dataset is assigned to an owner. I could assign squiggletronics data to a user squiggletronics so users can see where the data is coming from. This way it´s also easy to update the data. I can also provide a backlink to the original database.

dougcl wrote:

You may want to have your own garbage dump to let users do their thing with DIY modules. But that's different I think, because it does not require moderation.

I already implemented a function "private module". Users can declare their DIY modules private. Those modules are not listed in the search index.

At the moment I am busy fixing bugs and content has not highest priority. Users seem to be happy with what the upload themselves.

But I will contact you guys later to discuss the data exchange topics.
What is not completely clear to me: is John Nobles database also related to the squiggletronics db or is this an independent project?


dougcl

solitud wrote:

No, this would break the whole concept of the site.


Not at all, have a look at the Module Maker on squiggletronics if you haven't already. Yours is a duplicate effort.

http://squiggletronics.com/files/RackPlannerModules/moduleMaker.php


solitud wrote:

This would be a cool thing. I have already a mechanism where every dataset is assigned to an owner. I could assign squiggletronics data to a user squiggletronics so users can see where the data is coming from. This way it´s also easy to update the data. I can also provide a backlink to the original database.


Use the squiggletronics data for your clean library, because it is already cleaned.

solitud wrote:

I already implemented a function "private module". Users can declare their DIY modules private. Those modules are not listed in the search index.


Maintain two areas. A clean "library" area based on cleaned squiggletronics modules, and let each user build their own modules as unmoderated data. This means no more moderation for you, unless you want to help out at squiggletronics.


solitud wrote:

At the moment I am busy fixing bugs and content has not highest priority. Users seem to be happy with what the upload themselves.


Yes, this will have more meaning to you when (if) you are interested in cleaning the contributions to build a shared library.

solitud wrote:

But I will contact you guys later to discuss the data exchange topics.
What is not completely clear to me: is John Nobles database also related to the squiggletronics db or is this an independent project?


Unfortunately John has not coordinated with squiggletronics.


BananaPlug

Quote:
Use the squiggletronics data for your clean library, because it is already cleaned.


Well, it's not yet as clean as I would like it to be but we do at least have a quick look at things before they get added and cull obvious junk and duplicates. Since July we've been doing some minor corrections as things come in but we've always leaned towards inclusion rather than perfection. The plan is to recruit people to maintain and tidy up specific areas. We have some older modules with bad spelling, some cases where people had different ideas about abbreviations, that sort of thing.

The evolving facilities for moderation and data sharing allow for ongoing maintenance and improvement of the collection with the results percolating out to end users.


solitud

dougcl wrote:
The only thing that needs to be centralized is a shared module identifier. If everyone agrees that the 4MS Pedals Noise Swash should be identified as follows:
Manuf: 4MS
Model: Noise Swash
Format: E
Modifier: <null>


I would like to recommend a unique key as identifier which would make querying the database a lot easier if you guys planing to make a kind of rest interface.
1. Concatenate the values and remove spaces:
4MSNoiseSwashE

2. Put them lowercase
4msnoiseswashe

3. Hash it with md5
9c0d025eace67eb966f966f32b42c18a

Thats the unique identifier for data interchange.

This way we could also remap existing data to the identifier and chances are identifiers will match.

Put also a modified date as unix timestamp or mysql timestamp in the dataset, than we know when the data was changed.

Advanced thoughts:
For even more fault tolerance brave developers would apply a soundex function to the manufacturer name before concatenatening:

soundex("Pitsburgh Modular") = P321
soundex("Pittsburgh Modular") = P321
soundex("4MS") = M200

id = md5("m200noiseswashe");

But I don´t know if soundex is available in all programming languages, so it´s more experimental nature wink

Just some ideas to play with.


dougcl

These are all great ideas around key generation, but they all must still rely on an agreed set of identifier strings if they are to lead to universal keys.

So we are back where we started, coordinating identifier strings.

I think it's actually better to have completely meaningless keys, so that the identifier strings can be changed if needed.

The plan is to have one main key generating authority, who is responsible for scrubbing user contributed data, so that we don't have duplicate efforts on many databases. I am proposing that the key assigning authority is squiggletronics.

I have added a structure for such a key (or keys) to the schema.

At this point please consider the schema a requirements document rather than a commitment to implement a web service. The same ideas can be implemented with REST or whatever technology is most convenient.

The identifier strings will still be there to assist in identification to systems that are not participating (eurorackdb, for example). It's less reliable, but it's better than nothing.


John Noble

dougcl wrote:
The identifier strings will still be there to assist in identification to systems that are not participating (eurorackdb, for example). It's less reliable, but it's better than nothing.


Correction: pending feedback, The Eurorack Database *is* participating as noted in the update to the schema thread. IDs are now included in the module files since there is now a place to put them.

As noted elsewhere, including email, I'm happy to share what I have. I don't have any use for automated data import, however, so there's not a powerful reason for me to spend days hand matching data from another database--especially when no one has made any real case for how that data might be consumed.

I run an authoritative module reference site that has a courtesy download function for users of RackPlanner and compatible software, not a module library for planners. The server logs bear this notion out. There is room for both kinds of sites and more besides, and I certainly applaud anyone who goes to the trouble to serve the community. Squiggletronics and Rackplanner were there for me last night when I was goofing around with Buchla Tetris and needed new modules. applause


dougcl

Hi John, hopefully the meaning is clear. A database that is participating in this collaborative effort is deferring the creation and moderation of modules to squiggletronics.

This means that the database honors the squiggletronics identifiers:

identifier (the new one, with authority=squiggletronics)

... and/or at least these:

manuf
model
format
modifier

It means nothing more nor less than this.

No matter what the ultimate use of your data is or will be, it should be clear that a consistent use of identifiers by all databases is a benefit to the community.

Thanks,
Doug


solitud

Article in Wikipedia:
There are 3 competing standards ...

IT Guy:
3 competing standards, that´s ridiculous! Let´s combine them into one standard!

Article in Wikipedia:
There are 4 competing standards ...

MUFF WIGGLER Forum Index -> RackPlanner Goto page 1, 2  Next [all]
Page 1 of 2
Powered by phpBB © phpBB Group