Game engine runtime decompression

Question

I'm trying to figure out how game engines deal with asset compression. Obviously, they compress all assets when creating a game. But how do they unpack them at runtime. The only thing I could think of was unpacking into memory, but it should be very intense. If they are unpacked to the hard drive, will the folder fill up during the game? It is not very effective.

Using a library like zlib (or any other) with C ++, how is this execution decompression performed?

David

Answer 1

Be that as it may, you have a game with so much data that it does not fit in a reasonable amount of memory, so you cannot load them all at once in memory, so you define some buffers for the data that you use as caches.

The memory is now in order of speed from lowest to highest:

• DVD
• HDD
• RAM

Ideally, you don’t have to transfer data from a DVD, but this should be taken into account if you need to make a console game, for example. Therefore, for each of these available storage spaces, you define a buffer that will be used as a cache.

When the game engine decides that it may need an asset, it must first look in the fastest cache to see if the asset is loaded. If so, you're in luck, you can immediately send it to draw. If this is not the fastest cache, you need to go to the cache level of the hard drive. This is a file in which you save assets that have already been unpacked and ready to be loaded into memory. If the fastest cache is not fully occupied, you can simply start loading the data and use it when it is ready. If there is not enough space, then you will first have to unload other assets, I would recommend deleting the least recently used assets until you have enough space to load a new one.

Now, if there is no downloaded data in the cache of the hard drive, you need to go one more level to the archive, you will need to use the zip format for compression, because the zip file format does not force you to unzip the entire archive in order to have access to one file, so all you have to do is find the offset of the specified file in the archive and unzip it into the cache of your hard drive. Again, if the cache is full, you will first have to unload some other assets, and I would recommend deleting the last one, but you can also try other algorithms if you think this will improve performance.

John Carmack had a key report from QuakeCon 2011, where he explained that the whole process can be a little better than I can at the post (among other amazing things), you can find it here

Answer 2

Compression can occur in different ways at different levels, and when, where and how its use depends entirely on its goals and objectives (compression does not always mean saving disk space).

Basically, at the top level, all assets will / can be compressed into a massive archive (this speeds up reading, since reading from a hard disk is less, but you sacrifice processing power for this, using DMA to read uncompressed files that don't use a processor at all ), reading is almost always performed in memory, memory reading back on the HDD will lead to performance degradation and will cause many problems (and in some cases it will be impossible, for example, on older generation consoles).

The second level can / can be done on the asset itself, as an example, textures can be compressed in MANY different ways, but mainly block compression (S3TC / DXTn, BCn) is used, since its decompression is supported in (or emulated by the driver), therefore When reading it for an archive / disk, further decompression is not required.

Compression strategies also depend on the platform, especially on consoles that are very sensitive to memory layout, have limited resources and have small caches, etc.

Using a library like zlib (or any other) with C ++, how is this execution decompression performed?

usually you want to use memory mapped files in the archive and unzip it directly to RAM, which is a good example of an AAA archive system that is well documented, which makes it MPQ format (used by Blizzard Entertainment, more details here ), it uses a lot of compression algo, such as deflation for Diablo I, zlib for Warcraft III, bzip2 in World of Warcraft, and they recently added LZ and sparse compression for their new games, such as SCII.

Jan Wassenberg ( Optimizing File Accesses via Ordering and Caching ) has a good file management structure, which may also be of interest.

Answer 3

game engines deal with asset compression

There are various game engines, such as geometric models (triangulation / polygonal model), textures, etc. Each asset class has a different compression strategy. Can you be more specific?

To compress the model Check this