What’s going on here?
If you’re lost about what I’m trying to do here, the beginning can be found here: Ticket to Graph: modeling a board game in Neo4j. The TL;DR is that Ticket to Ride is a popular board game that asks players to build the best rail network across North America by buying routes to connect cities. I’m building out a model that will eventually be able to track the state of the game and help suggest the best routes and most valuable cards to pick up. Above is a list that includes all the posts in this series in order.
Improving our loading situation.
I want our model to be able to be quickly loaded and flexible enough to accommodate the multiple maps expansions that exist in Ticket to Ride. Now that I have a good start on the very basic elements of the nodes that make up the board, I should be in a good place to start thinking about making it easier to load the entire graph of the boards.
Neo4j makes it possible to load in CSV, JSON, and other common data types. I’m going to have to create the raw data, and then create a procedure that can read the data into nodes and relationships for us.
Creating raw data.
Since I’m in the enviable position of creating my own raw data, I’m going to do what I can to make it easy on myself. I want to have separate files describing the city nodes, the route nodes, and the relationships between it all. I’m also going to go ahead and add a mapName element, just for later reference, since we might have multiple options later. I’m also going to group all the files in their own subfolder, looking ahead to a time where I might have multiple sets of map data that can be loaded.
[
{
name: 'Pittsburgh',
code: 'PIT',
mapName: 'USA'
},
{
name: 'Toronto',
code: 'TOR',
mapName: 'USA'
},
{
name: 'New York',
code: 'NYC',
mapName: 'USA'
},
{
name: 'Montreal',
code: 'MTL',
mapName: 'USA'
},
{
name: 'Boston',
code: 'BOS',
mapName: 'USA'
},
{
name: 'Washington',
code: 'DC',
mapName: 'USA'
},
{
name: 'Raleigh',
code: 'RNC',
mapName: 'USA'
},
{
name: 'Charleston',
code: 'CLS',
mapName: 'USA'
},
{
name: 'Miami',
code: 'MIA',
mapName: 'USA'
},
{
name: 'Atlanta',
code: 'ATL',
mapName: 'USA'
}
]I created the above JSON and saved it in the project directory. This is not all cities, but I wanted to check I could load this much before proceeding, and I’m already hitting a roadblock. Note that the file is an array, containing many cities to create.
The Roadblock, AKA the configuration rabbit hole.
Warning, this is just a gripe and contains no useful information about the project itself. Feel free to skip to the next heading.
So, up until this point, I’ve been able to get by copying and pasting code from my text editor into the command prompt for Neo4j, but now that I’ve gotten to a point where I need the system to look at a file I’ve defined, I found that I needed to begin reckoning with the intended workflow of Neo4j. I had to figure out how to configure Neo4j to accept local files to load.
I’m about to complain about this, and I’m aware that it’s not something I fully understand, so I’m definitely part of the problem in this situation. But I found out that not only was disabling local files disabled by default, but that in order to change it, I had to find and alter the configuration files for the project. Now, when I created my default project in Neo4j, it created me a directory to hold it, which appeared empty to me. However, unbeknownst to me there were actually subfolders that were hidden by default that contained the config files, as well as plenty of other things. So when I went and created a folder with the name the documentation was requesting, and added the config file to that, nothing happened. When I tried to add the line of configuration to a file I was able to open in Neo4j Browser, that caused the entire DB to fail to load. What I ACTUALLY had to do in order to get the config to stick I wound up finding 2/3 of the way down in a forum thread.
So once I got the configuration added, I also had to activate the plugin that contained the file load processes themselves, which was buried in a different Neo4j Browser menu.
Then when I got that working, I found that one of those hidden folders, “Import” was actually the ONLY place that Neo4j was allowed to read data from by default, so I had to shift the data into that directory. Once I got all that figured out, the actual load process was very simple. However, I was really discouraged in the moment, as none of this is readily explained. You’re locked into using Neo4j’s default project template as a setup for your project, which would be perfectly fine, but I found the Desktop setup to be quite poorly documented, and not great UI. There’s a place for files within your project that I still cannot figure out how to get my scripts and files to display within. I wish something were different about how this worked, but I’m not sure I can explain what. It’s entirely possible it just takes a little bit to learn.
Back on Track- Pun Intended
Okay, so you’ve either skipped or indulged me in my rant. Now we’re back to loading data. Once I enabled local files and added my json data to the file, I was able to quite easily adapt the examples to do what I needed it to do.
match(m)
Detach delete m // Clear the Database
;
call apoc.load.jsonArray("MapData/UnitedStates/Cities.json")
YIELD value //read json data into "value"
MERGE (c:City
{
name: value.name, // traverse json to find properties
code: value.code,
mapName: value.mapName
}
)
;
As you can see, we’re back to having just cities with no relationship since we dropped all our nodes before reloading. That’s okay, though as I’m planning on creating JSON to represent all the missing routes and relationships as well.
Route JSON
I’m reviewed my plan, and I believe that I should be able to combine the rest of everything into one JSON file. I’m not sure if this is a good idea or not, but it should save me a step or two in the load process, so I’m going to see if I can get it to work. Here’s a look at my JSON for the routes and their relationships with the cities and one another.
You might spot the mistake in the above JSON. The parallelRoute for PIT_NYC_B points to itself. Take my word that I’ve fixed it in my JSON so that I don’t have to take another screenshot. It took me a few tries to get the script right to load the routes since there are a few steps here.
- Create each individual Route node.
- Unwind the cities that are connected.
- Create relationships between the route and those two cities
- Match the route mentioned in parallelRoute
- Create PARALLEL_TO relationship
So here is the Cypher code that I wound up with that accomplishes all of that:
// Load Routes from Routes.json
call apoc.load.jsonArray("MapData/UnitedStates/Routes.json")
YIELD value //read json data into "value"
// create node with "Route" label for each cities.json array element.
MERGE (r:Route{
// Creating Route Node
code: value.code,
mapName: value.mapName,
length: value.length
}
)
With r, value
UNWIND value.connectsCities as cityCode //unpack array.
MATCH (c:City{code:cityCode}) // Find the city node matching cityCode...
merge (c)-[:CONNECTED_BY]->(r) //... and merge the new relationship
;
MATCH (r:Route)
WHERE r.parallelRoute IS NOT NULL
WITH r
Match (r2:Route {code: r.parallelRoute})
REMOVE r.parallelRoute
MERGE (r)-[:PARALLEL_TO]->(r2)
;
So a few minutes of setting up some additional JSON data sees us with this:
Now that we have a JSON definition and a programmatic way to get the nodes loaded in, we’re well on our way to having a functional data model. I have to add the additional cities and routes to the JSON.
On the backlog for this, we’ve got a few things that we can continue working on aside from just adding more data.
- Make this load into a proper procedure in the database, rather than just a script.
- Start working on a few pathfinding algorithms to find ways around the map.
- Start designing player nodes and figuring out ownership of different routes.
If anybody is actually reading this, Thanks for tagging along! I’m really enjoying this chance to work on a side project and flex my muscles. If you want any more information on me or working with me, feel free to reach out. Watch this space for more updates on this project when they happen.
Leave a Reply