As I mentioned in the last post, the new Markov engine in XIAS really required a new data structure system to run efficiently.  It required a data structure with the intuitive read/write functions of Object System but the powerful substructure storage system of GDS.  Enter Object System 2.

In developing the second version of Object System, the primary goal was to take a write statement like “movement.progression.chord1=Amin7″ and make it correspond to a storage address rather than just a variable name.  In other words, instead of just being stored as a variable with the name movement.progression.chord1, it would be stored as the variable chord1 (having the value Amin7) within the substructure progression as part of a larger structure movement.  This kind of structural nesting provides significant improvements in access times for large data structures that have been properly segmented.

On top of a true substructure storage system like that of GDS, Object System 2 has the powerful ability to nest an infinite number of substructures, as opposed to the 10-sublevel limit imposed by GDS.  Furthermore, OS2 uses only 3 string replacement commands to encode a substructure, while GDS uses around 20.  In short, OS2 now has all the power of GDS, all the ease-of-use of OS, and a significant efficiency boost over both.

To prove the benefits of OS2, I’ve replaced all of the XIAS calls to OS1 with identical calls to OS2 and graphed the performance results.  While the two data systems perform almost equally for low-order Markov analyses, the runtime for OS1 quickly gets out of hand for high-level analyses.  For a 10th level analysis, OS2 is more than twice as fast as OS1.  This is to be expected, since the performance gains are magnified as the data structures get larger and OS2’s substructure system saves the CPU expensive string parsing loops.