Update doc/mapformat.txt

doc/mapformat.txt

@@ -1,24 +1,22 @@
-=========================================
+===============================================
-         Minetest World Format
+Minetest World Format used as of 0.4.dev-120322
-=========================================
+===============================================
 
-Format used as of 0.4.dev-120322
+This applies to a world format carrying the block serialization version 22
-==================================
+which is used at least in version 0.4.dev-120322.
 
-This applies to a world format carrying the serialization version 22 which
+The map data serialization version used is 22. It does not fully specify every
-is used at least in version 0.4.dev-120322.
+aspect of this format; if compliance with this format is to be checked, it
+needs to be done by detecting if the files and data indeed follows it.
 
-The serialization version used is 22. It does not fully specify every aspect
+Legacy stuff
-of this format; if compliance with this format is to be checked, it needs to
+=============
-be done by detecting if the files and data indeed follows it.
-
-Legacy stuff:
--------------
 Data can, in theory, be contained in the flat file directory structure
-described below in Version 17, but it is not officially supported.
+described below in Version 17, but it is not officially supported. Also you
+may stumble upon all kinds of oddities in not-so-recent formats.
 
-Files:
+Files
-------
+======
 Everything is contained in a directory, the name of which is freeform, but
 often serves as the name of the world.
 
@@ -33,36 +31,367 @@ World
 |-- map.sqlite --- Map data
 |-- players ------ Player directory
 |   |-- player1 -- Player file
-|   '-- player2 -- Player file
+|   '-- Foo ------ Player file
 `-- world.mt ----- World metadata
 
 auth.txt
 ---------
 Contains authentication data, player per line.
-<name>:<password hash as <name><password> SHA1 base64>:<privilege1,...>
+  <name>:<password hash>:<privilege1,...>
+Format of password hash is <name><password> SHA1'd, in the base64 encoding.
+
 Example lines:
-Player "celeron55", no password, privileges "interact" and "shout":
+- Player "celeron55", no password, privileges "interact" and "shout":
     celeron55::interact,shout
-Player "Foo", password "bar", privileges "interact" and "shout":
+- Player "Foo", password "bar", privilege "shout":
-foo:iEPX+SQWIR3p67lj/0zigSWTKHg:interact,shout
+    foo:iEPX+SQWIR3p67lj/0zigSWTKHg:shout
+- Player "bar", no password, no privileges:
+    bar::
 
 env_meta.txt
 -------------
---- Example content ---
+Simple global environment variables.
+Example content (added indentation):
   game_time = 73471
   time_of_day = 19118
   EnvArgsEnd
------------------------
 
 ipban.txt
 ----------
+Banned IP addresses and usernames.
+Example content (added indentation):
+  123.456.78.9|foo
+  123.456.78.10|bar
 
+map_meta.txt
+-------------
+Simple global map variables.
+Example content (added indentation):
+  seed = 7980462765762429666
+  [end_of_params]
 
-Format used as of 2011-05 or so
+map.sqlite
-================================
+-----------
+Map data.
+See Map File Format below.
+
+player1, Foo
+-------------
+Player data.
+Filename can be anything.
+See Player File Format below.
+
+world.mt
+---------
+World metadata.
+Example content (added indentation):
+  gameid = mesetint
+
+Player File Format
+===================
+
+- Should be pretty self-explanatory.
+- Note: position is in nodes * 10
+
+Example content (added indentation):
+  hp = 11
+  name = celeron55
+  pitch = 39.77
+  position = (-5231.97,15,1961.41)
+  version = 1
+  yaw = 101.37
+  PlayerArgsEnd
+  List main 32
+  Item default:torch 13
+  Item default:pick_steel 1 50112
+  Item experimental:tnt
+  Item default:cobble 99
+  Item default:pick_stone 1 13104
+  Item default:shovel_steel 1 51838
+  Item default:dirt 61
+  Item default:rail 78
+  Item default:coal_lump 3
+  Item default:cobble 99
+  Item default:leaves 22
+  Item default:gravel 52
+  Item default:axe_steel 1 2045
+  Item default:cobble 98
+  Item default:sand 61
+  Item default:water_source 94
+  Item default:glass 2
+  Item default:mossycobble
+  Item default:pick_steel 1 64428
+  Item animalmaterials:bone
+  Item default:sword_steel
+  Item default:sapling
+  Item default:sword_stone 1 10647
+  Item default:dirt 99
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  EndInventoryList
+  List craft 9
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  Empty
+  EndInventoryList
+  List craftpreview 1
+  Empty
+  EndInventoryList
+  List craftresult 1
+  Empty
+  EndInventoryList
+  EndInventory
+
+Map File Format
+================
+
+Minetest maps consist of MapBlocks, chunks of 16x16x16 nodes.
+
+In addition to the bulk node data, MapBlocks stored on disk also contain
+other things.
+
+History
+--------
+We need a bit of history in here. Initially Minetest stored maps in a
+format called the "sectors" format. It was a directory/file structure like
+this:
+  sectors2/XXX/ZZZ/YYYY
+For example, the MapBlock at (0,1,-2) was this file:
+  sectors2/000/ffd/0001
+
+Eventually Minetest outgrow this directory structure, as filesystems were
+struggling under the amount of files and directories.
+
+Large servers seriously needed a new format, and thus the base of the
+current format was invented, suggested by celeron55 and implemented by
+JacobF.
+
+SQLite3 was slammed in, and blocks files were directly inserted as blobs
+in a single table, indexed by integer primary keys, oddly mangled from
+coordinates.
+
+Today we know that SQLite3 allows multiple primary keys (which would allow
+storing coordinates separately), but the format has been kept unchanged for
+that part. So, this is where it has come.
+</history>
+
+So here goes
+-------------
+map.sqlite is an sqlite3 database, containg a single table, called
+"blocks". It looks like this:
+
+  CREATE TABLE `blocks` (`pos` INT NOT NULL PRIMARY KEY,`data` BLOB);
+
+The key
+--------
+"pos" is created from the three coordinates of a MapBlock using this
+algorithm, defined here in Python:
+
+  def getBlockAsInteger(p):
+      return int64(p[2]*16777216 + p[1]*4096 + p[0])
+  
+  def int64(u):
+      while u >= 2**63:
+          u -= 2**64
+      while u <= -2**63:
+          u += 2**64
+      return u
+  
+It can be converted the other way by using this code:
+
+  def getIntegerAsBlock(i):
+      x = unsignedToSigned(i % 4096, 2048)
+      i = int((i - x) / 4096)
+      y = unsignedToSigned(i % 4096, 2048)
+      i = int((i - y) / 4096)
+      z = unsignedToSigned(i % 4096, 2048)
+      return x,y,z
+  
+  def unsignedToSigned(i, max_positive):
+      if i < max_positive:
+          return i
+      else:
+          return i - 2*max_positive
+
+The blob
+---------
+The blob is the data that would have otherwise gone into the file.
+
+See below for description.
+
+MapBlock serialization format
+==============================
+NOTE: Byte order is MSB first (big-endian).
+NOTE: Zlib data is in such a format that Python's zlib at least can
+      directly decompress.
+
+u8 version
+- map format version number, this one is version 22
+
+u8 flags
+- Flag bitmasks:
+  - 0x01: is_underground: Should be set to 0 if there will be no light
+    obstructions above the block. If/when sunlight of a block is updated
+    and there is no block above it, this value is checked for determining
+    whether sunlight comes from the top.
+  - 0x02: day_night_differs: Whether the lighting of the block is different
+    on day and night. Only blocks that have this bit set are updated when
+    day transforms to night.
+  - 0x04: lighting_expired: If true, lighting is invalid and should be
+    updated.  If you can't calculate lighting in your generator properly,
+    you could try setting this 1 to everything and setting the uppermost
+    block in every sector as is_underground=0. I am quite sure it doesn't
+    work properly, though.
+  - 0x08: generated: True if the block has been generated. If false, block
+    is mostly filled with CONTENT_IGNORE and is likely to contain eg. parts
+    of trees of neighboring blocks.
+
+u8 content_width
+- Number of bytes in the content (param0) fields of nodes
+- Always 1
+
+u8 params_width
+- Number of bytes used for parameters per node
+- Always 2
+
+zlib-compressed node data:
+- content:
+  u8[4096]: param0 fields
+  u8[4096]: param1 fields
+  u8[4096]: param2 fields
+
+zlib-compressed node metadata list
+- content:
+  u16 version (=1)
+  u16 count of metadata
+  foreach count:
+    u16 position (p.Z*MAP_BLOCKSIZE*MAP_BLOCKSIZE + p.Y*MAP_BLOCKSIZE + p.X)
+    u16 type_id
+    u16 content_size
+    u8[content_size] (content of metadata)
+
+u16 mapblockobject_count
+- Always 0
+- Should be removed in version 23 (TODO)
+
+u8 static object version:
+- Always 0
+
+u16 static_object_count
+
+foreach static_object_count:
+  u8 type (object type-id)
+  s32 pos_x_nodes * 10000
+  s32 pos_y_nodes * 10000
+  s32 pos_z_nodes * 10000
+  u16 data_size
+  u8[data_size] data
+
+u32 timestamp
+- Timestamp when last saved, as seconds from starting the game.
+- 0xffffffff = invalid/unknown timestamp, nothing should be done with the time
+               difference when loaded
+
+u8 name-id-mapping version
+- Always 0
+
+u16 num_name_id_mappings
+
+foreach num_name_id_mappings
+  u16 id
+  u16 name_len
+  u8[name_len] name
+
+EOF.
+
+Node metadata format
+---------------------
+
+1: Generic metadata
+  serialized inventory
+  u32 len
+  u8[len] text
+  u16 len
+  u8[len] owner
+  u16 len
+  u8[len] infotext
+  u16 len
+  u8[len] inventory drawspec
+  u8 allow_text_input (bool)
+  u8 removal_disabled (bool)
+  u8 enforce_owner (bool)
+  u32 num_vars
+  foreach num_vars
+    u16 len
+    u8[len] name
+    u32 len
+    u8[len] value
+
+14: Sign metadata
+  u16 text_len
+  u8[text_len] text
+
+15: Chest metadata
+  serialized inventory
+
+16: Furnace metadata
+  TBD
+
+17: Locked Chest metadata
+  u16 len
+  u8[len] owner
+  serialized inventory
+
+Inventory serialization format
+-------------------------------
+- The inventory serialization format is line-based
+- The newline character used is "\n"
+- The end condition of a serialized inventory is always "EndInventory\n"
+- All the slots in a list must always be serialized.
+
+Example (format does not include "---"):
+---
+List foo 4
+Item default:sapling
+Item default:sword_stone 1 10647
+Item default:dirt 99
+Empty
+EndInventoryList
+List bar 9
+Empty
+Empty
+Empty
+Empty
+Empty
+Empty
+Empty
+Empty
+Empty
+EndInventoryList
+EndInventory
+---
+
+==============================================
+Minetest World Format used as of 2011-05 or so
+==============================================
 
 Map data serialization format version 17.
 
+0.3.1 does not use this format, but a more recent one. This exists here for
+historical reasons.
+
 Directory structure:
 sectors/XXXXZZZZ or sectors2/XXX/ZZZ
 XXXX, ZZZZ, XXX and ZZZ being the hexadecimal X and Z coordinates.