New tricks for old MUDs: using JSON for flexible player data

In the beginning (1990), the creators of Diku had many programming challenges.  One of the very least interesting of these was how player data would be written to and read from permanent disk storage.  This was before the days of structured data formats and object-oriented databases – some subset of Staerfeldt, Madsen, Nyboe, Seifert, and Hammer had to roll their own format.  Sometimes I wonder if they thought that their work would still be around nearly 30 years later.

As it turns out, the original method they came up with for storing players was a little terrible.  Diku stored all players in one binary file, and all of their possessions in another binary file.  Altering either of these would corrupt your entire player base. A year later Merc 1.0 would take a big leap to fix this, by storing players (with their stuff) in individual binary files.  You still couldn’t modify them unless you were handy with a hex editor, but at least only one player could be messed up at a time.

The Almighty pFile

It was Merc 2.0 (1992) that would introduce the format that most of the Merc derivatives still use: the human readable “pfile.”  For the first time, game administrators could search and modify the files as needed.  Here’s an example, straight from a fresh save on Merc:

What we see here are strings terminated by a tilde character, simple integers for Sex, Class, Race, and Level, and an array of integers for HpManaMove.  The reading behavior is defined for each key – whether to expect a string, int, array, or something else.

This was a huge improvement over previous games – and not just because the admin could tinker with it.  Merc 2 introduced treating the file as a key-value store, where reading the key (such as “Name”) could be mapped to the correct location to store the value (“Montrey”).  Keys could come in any order as well, to prevent additions or removals from making files unreadable.  This was accomplished with a nifty macro to hide a huge ugly bunch of “if” conditions:

The inflexible pFile

If you’ve ever worked on a Merc derivative (like ROM), you’ve seen all this in save.c.  It works well, to an extent.  Why would anyone want to change it?

Rather than list all the abstract reasons why a portable, standardized structured file format is better than the Merc standard, I’ll describe the use case that led me to the decision to change.  We at Legacy are planning an equipment enhancement system based on placing “gem” objects in a piece of equipment to add stat boosts.  Simple enough, I thought.  I’ll just make any piece of equipment able to contain a list of “gems”, add up and cache their effects, and done.  It was all clear-cut… until I faced save.c.

In Merc muds, objects (of the container type) can hold a list of other objects.  When the game saves the player’s inventory, it writes the first object, followed by the first thing (if any) that it contains, writing the nest level for each object so that they can be read correctly in reverse order.  (Actually, because of how singly linked lists work, it uses a recursive function to write the list in reverse order, so that it can be loaded in the correct order without having to iterate to the tail of the list.  But that’s not important.)  The pfile ends up looking like this:

So how to add a new savable list of objects that can be “contained” in another object? What if you put gems in a bag?  How about a bag encrusted with gems?  Hrm.  You could write one list after another… but then you’d have to remember the nest level for “gems” and for “contents.”  You could come up with all kinds of convoluted ways that would make the code harder to read, harder to maintain, and easier to break.  The limitations of the pfile format were starting to become obvious.  The format was meant to store sets of values, but only barely supports nested objects.  I wanted a more elegant solution.

Enter JSON

JSON (JavaScript Object Notation) is a fairly modern structured storage format, similar to XML without all the <>; nonsense.  A JSON file can store 3 types of things:

  • values (strings, integers, floats, etc)
  • lists (of values, like a C array), separated by commas, enclosed within square brackets ([])
  • maps (a set of named values, like a C struct), with keys : value pairs separated by commas and enclosed within braces ({})

Most importantly for my purposes, these can be nested – you can store, say, an array of maps.  Some examples can be found at json.org, but it might be easier to just show a Character expressed in JSON:

The root of the JSON file is a map, with named values contained in it.  Name, Description, Sex, Class, Race, and Level are string or integer values.  HpManaMove is a list of integer values.  Inventory is nested map of the top-level inventory objects, with Contents and Gems optional lists of more objects.  Obviously, you probably don’t want to write complex JSON files by hand, but it is still pretty straightforward if you need to occasionally grep, sed, or fix something in a text editor.

Serializing MUD objects with cJSON

So how do we make it happen?  I won’t lie, it did take a while to convert Legacy to use JSON for pfiles, but once I muddled my way through it, it turned out to be pretty simple.  I decided to use the cJSON project, because it is lightweight (just a single include and source file) and had the reasonably simple semantics I wanted.  Here’s a shortened example of writing a character to file:

In this code, we simply create a root object (which is a JSON map) and add some named objects to it.  Then we let cJSON convert the whole thing to a string and write it to the file.  cJSON takes care of the parsing for us too, so at the high level, reading a pfile is essentially the reverse:

Constructing and iterating through cJSON objects

Of course, the real meat of it in the fread/write_char and inventory functions.  Here is how the “character” section is written – actually not that different from the default fprintf statements:

And next is the corresponding read section. This is a little more involved because of how ROM uses a memory pool for strings. We also use a variety of integer lengths and some strings that convert to bitpacked integers, so I defined some functions to handle the conversions:

With that out of the way, reading data isn’t much different from the original Merc 2 way:

Wasn’t this about nested objects?

The original problem was that I was having trouble storing different types of nested lists in the Merc format. This turns out to be trivial with JSON, because the structure is already similar to the inventory hierarchy of the player. An object can contain other objects, or other named lists (“gems”, etc) as necessary. We simply create those lists within the cJSON object representing the game object.

What about that spiffy recursion to write the list in reverse order? Turns out we don’t even need it – since cJSON stores children of a cJSON node as a linked list, we can efficiently insert new children at the head (index 0). This way, the list is written backwards, so we can naturally load it forwards. Next up is writing the object itself, which can write its own lists of contents as necessary:

We’re in the home stretch! All that’s left to implement is reading those objects back into the game. Here things get a little weird – we do a check to see if the object loaded correctly. What if we decided to remove the object from the game since last time the player logged on? Rather than blow up the contents, we put them into the player’s inventory.

And finally, the code to load one game object:

What’s the downside?

“Wow!” you might say.  “Why not put all the game’s data in JSON?”

Why not?  JSON is easy to read, easy to write.  Players are probably the most complex thing you would ever use them for in a MUD.  Databases have their purposes (we use sqlite3 for some data) but files are good for human-accessible things.

I would say that the one thing I would not consider storing in JSON is the area files, unless you are wholly committed to OLC.  JSON is human readable, human changeable… but actually writing complex data files by hand would be tedious at best, and error-prone at worst.

You also might think that performance is an issue, because a JSON file is more verbose than the Merc format.  In the early 1990s, this was probably true.  I may have been a little hard on the developers of Merc in the introduction – they faced resource constraints and performance challenges that only embedded systems programmers think about now.  But in 2017, the extra 20-30% bytes in a JSON player file is going to amount to zero measurable difference.

Putting it all together

By now, I’ll bet you’re just dying to crack open your ROM source code for a marathon coding session, chugging Mountain Dew and pretending it’s 1994 again, dreaming up the new features made possible by the slick JSON player files. I tried not to make it too exciting, but it’s exciting stuff after all.

But wait. How do you get those old player files into the new format?

You could implement the new system in two stages, since your game already reads the old files.  Create the writer, and then come up with some way to batch load and save all of your current files.  I did that once to build a MySQL database pfile implementation, and it worked fine.  The parsing, anyway – SQL storage for complex game objects is a terrible idea.  Or, you could take the conversion script I wrote in Python and spend a few minutes adapting it to your particular flavor of pfiles.  I’ve included it in the source linked below.

I hope you’ve enjoyed this article, and for those of you still running old Merc/ROM derivatives, maybe it will be helpful.  You can find a copy of Legacy’s save.c here, and if you’d like to see this and our other projects in action, please visit us at legacy.xenith.org:3000.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">