Point guard age of empires game recording files (.mgx)

Question

I am a fan of the outdated game Age of Empires II (AoE). I want to write AoE write parser (.mgx files) using Python.

I did a few searches on GitHub and found few projects on this, the most useful aoc-mgx-format , which provide some information about the .mgx recording files .

Here is the problem:

according to the link, the structure of the .mgx file is as follows:

| header_len (4byte int) | next_pos (4byte int) | header_data | ...... |

The hexadecimal order of the data in mgx format is a bit endian .

header_len stores the data length of the header part (header_len + next_post + header_data)

header_data strong> stores useful information that I need, but compressed with zlib

I tried to unzip the data in header_data by the zlib module as follows:

import struct import zlib with open('test.mgx', "rb") as fp: # read the header_len bytes and covert it to a int reprents length of Header part header_len = struct.unpack("<i", fp.read(4))[0] # read next_pos (this is not important for me) next_pos = struct.unpack("<i", fp.read(4))[0] # then I can get data length of header_data part(compressed with zlib) header_data_len = header_len - 8 compressed_data = fp.read(header_data_len)[::-1] # need to be reversed because byte order is little endian? try: zlib.decompress(compressed_data) print "can be decompressed!" except zlib.error as e: print e.message 

but I got this after running the program:

Error -3 while unpacking data: incorrect header check

ps: examples of .mgx files can be found here: https://github.com/stefan-kolb/aoc-mgx-format/tree/master/parser/recs

Answer 1

Your first problem is that you should not modify the data; just get rid of [::-1] .

But if you do, instead of getting this -3 error, you will get another -3 error, usually about an unknown compression method.

The problem is that this is data without a zlib header, similar to what gzip uses. Theoretically, this means that information about the compression method, window, start dict, etc. Must be provided somewhere else in the file (in the case of gzip according to the information in the gzip header). But in practice, everyone uses deflate with the maximum window size and no dict beginning, so if I were developing a compact format for the game on the days when every byte counted, I would just hard code them. (Nowadays, this is exactly what was standardized in the RFC as the "DEFLATE Compressed Data Format", but most computer games of the 90s did not match the RFC in design ...)

So:

 >>> uncompressed_data = zlib.decompress(compressed_data, -zlib.MAX_WBITS) >>> uncompressed_data[:8] # version b'VER 9.8\x00' >>> uncompressed_data[8:12] # unknown_const b'\xf6(<A' 

So, it’s not only unpacked, it looks like a version and ... well, I think something looks like an unknown constant, but it is the same unknown constant in the specification, so I think we are good.

As decompress shows, MAX_WBITS is the standard / most common window size (and the only size used by what is commonly called "zlib deflate" as opposed to "zlib"), and passing a negative value means that the header is suppressed; other arguments that we can leave as default.

See also this answer , Advanced Features section in zlib docs and RFC 1951 . (Thanks to OP for finding links.)

Answer 2

Old, but here is an example of what I did:

 class GameRecordParser: def __init__(self, filename): self.filename = filename f = open(filename, 'rb') # Get header size header_size = struct.unpack('<I', f.read(4))[0] sub = struct.unpack('<I', f.read(4))[0] if sub != 0 and sub < os.stat(filename).st_size: f.seek(4) self.header_start = 4 else: self.header_start = 8 # Get and decompress header header = f.read(header_size - self.header_start) self.header_data = zlib.decompress(header, -zlib.MAX_WBITS) # Get body self.body = f.read() f.close() # Get players data sep = b'\x04\x00\x00\x00Gaia' pos = self.header_data.find(sep) + len(sep) players = [] for k in range(0, 8): id = struct.unpack('<I', self.header_data[pos:pos+4])[0] pos += 4 type = struct.unpack('<I', self.header_data[pos:pos+4])[0] pos += 4 name_size = struct.unpack('<I', self.header_data[pos:pos+4])[0] pos += 4 name = self.header_data[pos:pos+name_size].decode('utf-8') pos += name_size if id < 9: players.append(Player(id, type, name)) 

Hope this helps future programmer :)

On the way, I plan to write such a library.