Date: 20180804 (Aug 4th - 2018)
Playing with assembly acceleration, I found that the problem is reproducible in the (currently) latest version: v1.2.11 ( [GitHub]: madler / zlib - a very compact, but delicately unobtrusive compression library (http://zlib.net) ) .
This error only occurs (obviously OS: Win, build assembly toolkit: VStudio and build acceleration included) for:
- cmake build (works fine for "$ {ZLIB_SRC_DIR} /win32/Makefile.msc")
- x64 architecture (AMD64) (works fine for x86)
The following is the call stack (top → down is equivalent to external → inner) during unpacking.
Normal case:
- inflate (inflate.c)
- inflate_fast (inffast.c)
- ...
Assembler body:
- inflate (inflate.c)
- inflate_fast (contrib / masmx64 / inffast8664.c)
- inffas8664fnc (contrib / masmx64 / inffasx64.asm)
- ...
Problem:
# 2. missing ("$ {ZLIB_SRC_DIR} /CMakeLists.txt" knows nothing about inffast8664.c), so the chain is broken, which leads to an invalid library.
Decision:
Make CMakeLists.txt aware of this file by adding:
set(ZLIB_SRCS ${ZLIB_SRCS} contrib/masmx64/inffas8664.c )
in line ~ # 158 (enclosed in conditional expressions if(MSVC) and elseif (AMD64) ).
Posting full changes as well.
Zlib-1.2.11-msvc_x64_asm_speedups.diff:
The above is the differential. See [SO]: Run / debug UnitTests of a Django application from the right-click context menu in PyCharm Community Edition? (Reply @CristiFati) (section " Patch on utrunner ") on how to apply patches on Win (basically, each line starting with one "+" sign is entered, and each line starting with one "-" character , disappears). I am using Cygwin, by the way.
I also posted this patch to [GitHub]: madler / zlib - Ms VisualStudio - x64 assembler acceleration , but I'm not sure what its fate is, since more than 100 receive requests are pending.
Output :
e:\Work\Dev\StackOverflow\q029505121\build\x64>"c:\Install\Google\Android_SDK\cmake\3.6.4111459\bin\cmake.exe" -G "NMake Makefiles" -DAMD64=ON "e:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11" -- The C compiler identification is MSVC 19.0.24215.1 -- Check for working C compiler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/cl.exe -- Check for working C compiler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/cl.exe -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Looking for sys/types.h -- Looking for sys/types.h - found -- Looking for stdint.h -- Looking for stdint.h - found -- Looking for stddef.h -- Looking for stddef.h - found -- Check size of off64_t -- Check size of off64_t - failed -- Looking for fseeko -- Looking for fseeko - not found -- Looking for unistd.h -- Looking for unistd.h - not found -- Renaming -- E:/Work/Dev/StackOverflow/q029505121/src/zlib-1.2.11/zconf.h -- to 'zconf.h.included' because this file is included with zlib -- but CMake generates it automatically in the build directory. -- The ASM_MASM compiler identification is MSVC -- Found assembler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/ml64.exe -- Configuring done -- Generating done -- Build files have been written to: E:/Work/Dev/StackOverflow/q029505121/build/x64 e:\Work\Dev\StackOverflow\q029505121\build\x64>"c:\Install\Google\Android_SDK\cmake\3.6.4111459\bin\cmake.exe" --build . --target zlibstatic Scanning dependencies of target zlibstatic [ 5%] Building C object CMakeFiles/zlibstatic.dir/adler32.obj adler32.c [ 10%] Building C object CMakeFiles/zlibstatic.dir/compress.obj compress.c [ 15%] Building C object CMakeFiles/zlibstatic.dir/crc32.obj crc32.c [ 21%] Building C object CMakeFiles/zlibstatic.dir/deflate.obj deflate.c Assembler code may have bugs -- use at your own risk [ 26%] Building C object CMakeFiles/zlibstatic.dir/gzclose.obj gzclose.c [ 31%] Building C object CMakeFiles/zlibstatic.dir/gzlib.obj gzlib.c [ 36%] Building C object CMakeFiles/zlibstatic.dir/gzread.obj gzread.c [ 42%] Building C object CMakeFiles/zlibstatic.dir/gzwrite.obj gzwrite.c [ 47%] Building C object CMakeFiles/zlibstatic.dir/inflate.obj inflate.c [ 52%] Building C object CMakeFiles/zlibstatic.dir/infback.obj infback.c [ 57%] Building C object CMakeFiles/zlibstatic.dir/inftrees.obj inftrees.c [ 63%] Building C object CMakeFiles/zlibstatic.dir/inffast.obj inffast.c Assembler code may have bugs -- use at your own risk [ 68%] Building C object CMakeFiles/zlibstatic.dir/trees.obj trees.c [ 73%] Building C object CMakeFiles/zlibstatic.dir/uncompr.obj uncompr.c [ 78%] Building C object CMakeFiles/zlibstatic.dir/zutil.obj zutil.c [ 84%] Building C object CMakeFiles/zlibstatic.dir/contrib/masmx64/inffas8664.obj inffas8664.c [ 89%] Building ASM_MASM object CMakeFiles/zlibstatic.dir/contrib/masmx64/gvmat64.obj Microsoft (R) Macro Assembler (x64) Version 14.00.24210.0 Copyright (C) Microsoft Corporation. All rights reserved. Assembling: E:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11\contrib\masmx64\gvmat64.asm [ 94%] Building ASM_MASM object CMakeFiles/zlibstatic.dir/contrib/masmx64/inffasx64.obj Microsoft (R) Macro Assembler (x64) Version 14.00.24210.0 Copyright (C) Microsoft Corporation. All rights reserved. Assembling: E:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11\contrib\masmx64\inffasx64.asm [100%] Linking C static library zlibstatic.lib [100%] Built target zlibstatic
Notes :
- I am using VStudio 2015
- Regarding the above output:
- To keep the output as small as possible, I only build a static version
- For the same reason (and also to leave it as text), I create "NMake Makefiles" (cmdline assembly)
- inffas8664.c is being built (somewhere near the end)
- You can also disable assembler acceleration ( unchecking AMD64 in cmake-gui), but this will just be a workaround
- I conducted several rough tests (of course, I do not pretend that these results were general), and there was an improvement in the performance of the assembler implementation compared to the standard (debug versions) (a percentage lower is the ratio between the time spent on the same operation ( with / without) accelerations):
- Compress: ~ 86%
- Unpacking: ~ 62%
@ EDIT0 :
Comment by @MarkAdler ( [GitHub]: madler / zlib - building ASM zlib on Windows gives erroneous results ) states:
What assembly code is used? In the zlib contrib directory there are several. By the way, the material in the contrib directory is not part of zlib. This is just for convenience and is supported (or not) by third parties. What I will do is just remove the offensive code from the next release.
Just like a compilation warning (that everyone must have seen (and most likely ignored)):
Assembler code may have bugs -- use at your own risk
Obviously, accelerating assembler and VStudio do not get along very well. Moreover, on x86 there are several problems:
[SO] offers one fix : the module is unsafe for SAFESEH C ++ image (answer by @NayanaAdassuriya) (although this is not directly related to the question). In short, inffas32.asm and the linker option [MS.Docs]: / SAFESEH (the image has safe exception handlers) do not match. To get rid of this, either:
- Disable this option (enabled by default in x86 Release)
- Pass the / safeseh option to the collector (ml.exe)
- Declare one in asm
Since I use cmake to build for cmdline, I found a workaround for this. After generating CMakeFiles (but before the build) I specify this:
I'm sure cmake offers a way to do the above properly, but I did not find it (nor did I fully investigate it).
The unpleasant thing is segfault (access violation) during decompression. This requires [GitHub]: madler / zlib - inffas32.asm struct / enum to bind to zlib 1.2.9 .
After the fix, everything works fine, and the performance improvements are similar to x64.