Now that I've got the engines out of the way, there's really only one major thing standing in the way of a new GGrewve: data handling between sessions. All these well-organized data structures look great inside a compiler...but one glaring downside to using object-oriented programming over my previous "hack-ish" data system is that loading and saving are no longer as easy as writing the data structure to a file - because such an operation isn't well-defined for arbitrary structures like it is for a data structures that consists of one large string (GDS and OS structures from the AHK code). Loading and saving the structures from the c++ library requires a custom interface for each structure which is, to put it plainly, no fun at all.
For backwards compatibility, I've chosen to write all structures in Object System's encoding format. It's not nearly as efficient as binary data, but it takes a lot less work to handle the data in lesser languages like AHK. After writing a c++ version of the OS data system, to which most of my day today was devoted, reading and writing arbitrary data structures now consists of implementing an OS conversion interface for each structure. That is, each data structure must be given a function for converting its data to Object System format as well as a function for loading its data from an Object System structure.
It sounds pretty straightforward, but in complex engines like gMSE in which structures are nested within structures that are, in turn, nested within superstructures, the task gets daunting. Nonetheless, I've finished implementing loading/saving for all MSE structures and am in the process of writing the function for the gMSE space.
After loading and saving are out of the way, it should be relatively smooth sailing to the new GGrewve.
As I mentioned in the last post, the new Markov engine in XIAS really required a new data structure system to run efficiently. It required a data structure with the intuitive read/write functions of Object System but the powerful substructure storage system of GDS. Enter Object System 2.
In developing the second version of Object System, the primary goal was to take a write statement like "movement.progression.chord1=Amin7" and make it correspond to a storage address rather than just a variable name. In other words, instead of just being stored as a variable with the name movement.progression.chord1, it would be stored as the variable chord1 (having the value Amin7) within the substructure progression as part of a larger structure movement. This kind of structural nesting provides significant improvements in access times for large data structures that have been properly segmented.
On top of a true substructure storage system like that of GDS, Object System 2 has the powerful ability to nest an infinite number of substructures, as opposed to the 10-sublevel limit imposed by GDS. Furthermore, OS2 uses only 3 string replacement commands to encode a substructure, while GDS uses around 20. In short, OS2 now has all the power of GDS, all the ease-of-use of OS, and a significant efficiency boost over both.
To prove the benefits of OS2, I've replaced all of the XIAS calls to OS1 with identical calls to OS2 and graphed the performance results. While the two data systems perform almost equally for low-order Markov analyses, the runtime for OS1 quickly gets out of hand for high-level analyses. For a 10th level analysis, OS2 is more than twice as fast as OS1. This is to be expected, since the performance gains are magnified as the data structures get larger and OS2's substructure system saves the CPU expensive string parsing loops.
Picking XIAS up again, I'm now working on developing a flexible and lightweight Markov engine to give the XIAS library stochastic abilities.
Potential features for the Markov engine:
- Hierarchical storing of event probabilities for fast access time
- Will require hefty upgrade of Object System data structure
- Similar to GDS
- Improve access times for complex data structures
- Overlapping of orders
- Markov chains of different orders can have different "weights" that contribute to the overall probability of an event occurring
- Gives the user the ability to control how deeply the engine considers the past states when calculating probabilities
- Arbitrary event-handling
- Events don't have to take any particular structure - in fact, events could even have other data encoded into them to further enhance the predictive capabilities of the engine when outside variables may be influencing the future
Unfortunately, as mentioned in the notes above, making the engine efficient is going to require an overhaul of the Object System data structure created recently. It will require the ability to store substructures as single elements of larger structures to speed up access times (which I'm quite certain would get out of hand quickly for large systems in which many events contribute to the weightings). This ability will bring OS closer to the functionality of GDS. The main difference, however, will lie in the syntax. I still intend for OS to have easy, OOP-like syntax (Parent.sublevel1.sublevel2 ... variable = blah). All previous OS code will need to function after the rewrite.
When the Markov engine is finished, XIAS will boast a truly wide range of algorithms. I can only imagine the hybrid possibilities.
Here's what I'm thinking: a genetic algorithm that dynamically evolves Markov probability spaces that are based on an underlying grammatical system analysis, wherein the words of the grammar come from a fractal cutting engine whose parameters are fed by an L-system. Talk about one hybrid algorithm to rule them all!
Introducing a new structure plugin: Crystal Network. Although it has absolutely nothing to do with crystals, this plugin will be based off of a freshly-rewritten fractal cutting engine similar to that of Fraccut. The engine is undergoing a total rewrite using some different techniques as well as the OS data structure instead of GDS. The new engine should be more robust and easier to handle than the Fraccut engine.
If fractal cutting works half as well with structures as it does with generative modules, mGen will be cranking out even more fantastic pieces in no time.
Though Fraccut only has a very basic variation system right now, it's enough to almost ruin the coherence that, during the early stages, sounded so promising. But there are plenty of ways to fix that. Repetition of ideas has been almost totally destroyed, especially when multiple Fraccut parts come together. There are several problems with the method of generation right now:
- Each variation is pseudorandom, so there exists an infinite number of "permutations" of the idea. Does this not completely defeat the purpose of the seeding engine?
- Other than the single "styleclass" parameter that the generative modules get each movement, there is essentially no way of coordinating ideas across multiple modules. The lack of coordination module functionality is starting to bring everything else down. This avoidance won't be able to go on much longer, as it's quickly becoming the system's bottleneck.
- Variation is based on seeds right now...maybe it would be smarter to base variation on a separate variation engine? Perhaps some kind of sub-module that could be used across multiple generative modules and offers a variety of variation algorithms? Sounds like it'd be worth a try.
- Variation is performed on both seed strings. It would probably be more coherent to only perform variation on the offset string. I'll see to this once the new fractal cutting engine is in place.
On the side, the new cutting engine (still without a name) is coming along nicely. The OS data structure is performing well and code readability is better than ever.
It'll be exciting to see how the fractal structure module will perform in comparison to Manual and Easy Contoured Structure.
A new data structure was conceived today. While GDS is very powerful, it's not very efficient, and code readability can become an issue when several structures are nested. It's difficult to retrieve multiple-nested variables because that requires first retrieving the nested structures sequentially, which can get annoying.
Object System, to be abbreviated OS, is designed to be a speedy and easy-to-use alternative to GDS. Syntax for OS is as follows:
// Initializing an OS structure and writing some variables
OS("myvar3=a third value!",OSstruct)
OS("myvar1.substruct.nestedvar=a nested value!",OSstruct)
// Reading a variable
As one can see, the syntax is very concise and designed to offer maximum code readability with minimum wasted coding time. Also, nested variables are as easy as using the typical object-oriented dot operator. Of course it isn't true "nesting," like GDS, but it still offers good readability.
Finally, the performance benefits of OS were measured against GDS to make sure that OS is more efficient. Though the graphs don't display much gain, the performance tests did NOT measure the access times on nested variables, which would have taken GDS MUCH longer to retrieve and OS would have pulled ahead significantly.
A graph of performance follows
I hope to redesign the fractal cutting engine soon with the help of OS.