I bet that storing data in accessible, understandable formats results in better programs and more creativity than optimizing exclusively for the computer. I'm trying this idea out by writing a tool to convert the Minetest world map format into one that can be understood by a person with basic SQL skills.
Last week, minetest-cereal could turn an individual Minetest block 'pos' value into an x, y, z position. I also learned how to open a SQLite3 database and access command line arguments using Racket.
This week, all those parts are working together. When I run
program converts the 'pos' values from my DemoLand Minetest map into
x,y,z based positions and stores the results in the new
So that's a little progress.
I'd expect to be much farther along if I were writing in a more familiar language or style; functional programming and Racket are new to me. Luckily this is about learning and having fun, so no artificial deadlines. I don't know if my speed will pick up, or whether there is something to those claims that functional programming is actually just harder than more conventional styles.
SQLite3 is much, much faster if you run inserts inside of transactions. The first draft of the script was taking well over an hour to complete, even for fairly small maps. Luckily I found some instructions on how to optimize SQLite performance. Using transaction batches of 1,000 records at a time cut the time down to around 45 seconds for DemoLand. This may be worth further discussion.
The new database format doesn't have much use. The tool can't even convert it back to a Minetest world yet! But I can show you where this is going.
Here are 5 'pos' positions from the DemoLand map:
sqlite> select pos from blocks limit 5; -1140883449 -1140883448 -1140883447 -1140883446 -1140883445
Do those numbers mean anything to you? Me neither.
Here are 5 x,y,z coordinates from the new format:
sqlite> select x,y,z from blocks limit 5; -68|-8|32 -68|-8|33 -68|-8|34 -68|-8|35 -68|-8|36
I can tell you that all of those are 68 blocks west, 8 blocks below, and between 32 and 36 blocks east of 0,0,0. I could also look for a few blocks that are above 0,0,0:
sqlite> select x,y,z from blocks where y > 0 limit 5; -68|1|32 -68|1|33 -68|1|34 -68|1|35 -68|1|36
Lot's of numbers near -68,Y,35! I wonder whether those got stored first because of something in the mathematics of pos values, or if that's just the area where the game actually started?
Anyway - next step is to reverse the conversion. That will tell me whether I've mangled anything in either the positions or the actual block data itself.