0. Two approaches
In principle, this problem can be solved in two different ways or by a combination of two:
- Build your teams as smart as possible.
- Speeding up sales for quality loss.
The next two sections discuss both approaches.
1. Check who has ImageMagick: “Q8”, “Q16”, “Q32” or “Q64”?
First check your exact version of ImageMagick and run:
convert -version
If your ImageMagick has Q16 (or even Q32 or Q64 , which is possible, but overkill!) In the version line: This means that all internal ImageMagick functions treat all images as having 16-bit (or 32 or 64-bit) channel depths . This gives you the best quality image processing. But it also requires dual memory compared to the Q8 . Therefore, at the same time, this means poor performance.
Therefore: you can check what benefits you will achieve by switching to Q8 -build. ( Q is the "quantum depth" symbol supported by the ImageMagick layout.)
Nevertheless, you pay for possible loss of Q8 -performance with loss of quality. Just check what speed you achieve with Q8 over Q16 , and what quality losses you suffer. Then decide whether you can live with disabilities or not ...
In any case, Q16 will use twice as much RAM for each image for processing, and Q32 will again use twice as much Q16 . This is independent of the actual bits on the pixels that are visible in the input files. When saved, 16-bit image files will also consume more disk space than 8-bit ones.
If Q16 or Q32 requires more memory, you always need to make sure that you have enough. Because exceeding your physical memory would be very bad news. If a larger Q does the swap to disk process, performance will drop. A 1074 x 768 pixel image ( width x height ) will require the following amounts of virtual memory depending on the quantum depth:
Quantum Virtual Memory Depth (consumed by 1 image 1024x768) ------- ------------------------------ 8 3.840 kiB (=~ 3,75 MiB) 16 7.680 kiB (=~ 7,50 MiB) 32 15.360 kiB (=~ 14,00 MiB)
Also keep in mind that some “optimized” processing pipelines (see below) must contain multiple copies of the image in virtual memory! When virtual memory cannot be satisfied with available RAM, the system will begin to replace and will require "memory" from the disk. In this case, all the smart optimization of the command pipeline, of course, is gone, and begins to move on to the very opposite.
ImageMagick's birthday was in aera, when processors could only process 1 bit at a time. That was decades ago. Since then, the processor architecture has changed a lot. 16-bit operations performed twice as much as 8-bit operations or even longer. Then came the 16-bit processors. 16-bit operators have become standard. CPUs have been optimized for 16-bit: Suddenly, some 8-bit operations may take even more than 16-bit equivalents.
Currently 64-bit processors are shared. Therefore, the argument of Q8 vs. Q16 vs. Q32 in real conditions may even be invalid. Who knows? I do not know about any serious benchmarks about this. It would be interesting if someone (with really deep know-how about processors and about testing real programs) would one day work with such a project.
Yes, I see that you are using Q16 on Windows. But I still wanted to mention this, for completeness ... In the future, other users reading this question and answers will be read.
Most likely, since your input TIFFs are only black + white, the output image quality of the Q8 assembly will be good enough for your workflow. (I just don't know if it will also be much faster: it pretty much also depends on the hardware resources you are running this on ...)
In addition, if your sports support the installation of HDRI (high dynamic resolution images), this may also cause some speed limitation. Who knows? Thus, creating IM with the settings --disable-hdri --quantum-depth 8 may or may not lead to an improvement in speed. No one has ever tested this seriously ... The only thing we know about it: these settings will reduce image quality. However, most people will not even notice this if they do not show a very close look and do not make direct comparisons in the image ...
2. Test Your ImageMagick Capabilities
Then check if your ImageMagick installation is installed with OpenCL and / or OpenMP Support :
convert -list configure | grep FEATURES
If so (like mine), you should see something like this:
FEATURES DPC HDRI OpenCL OpenMP Modules
OpenCL (for C omputing L anguage) uses ImageMagick parallel computing functions (if compiled). This will allow your GPU to be used in addition to the processor for image processing operations.
OpenMP (for M completion P ) does something similar: it allows ImageMagick to run in parallel on all the cores of your system. Therefore, if you have a quad-core system and image resizing, resizing occurs on 4 cores (or even 8 if you have hyper-threading).
Command
convert -version
Displays basic information about supported features. If OpenCL / OpenMP is available, you will see one of them (or both) at the output.
If none of the two appears: look at getting the latest version of ImageMagick, which supports OpenCL and / or OpenMP support.
If you yourself create a package from sources, make sure that OpenCL / OpenMP are used. Do this by including the appropriate options in your configure step:
./configure [...other options-] --enable-openmp --enable-opencl
ImageMagick's documentation on OpenMP and OpenCL is here:
- Concurrent execution using OpenMP . Read carefully. Since OpenMP is not a silver bullet, and it does not work under any circumstances ...
- Concurrent execution with OpenCL . It applies the same as above. In addition, not all ImageMagick operations are supported by OpenCL. The link here contains a list of those that are.
-resize is one of them.
Tips and instructions for creating ImageMagick from sources and customizing the assembly, explaining the various options, are here:
This page also contains a brief description of the --with-quantum-depth configure option.
3. Control your ImageMagick
Now you can use the built-in -bench option to force ImageMagick to run a test for your team. For instance:
convert logo: -resize 500% -bench 10 logo.png [....] Performance[4]: 10i 1.489ips 1.000e 6.420u 0:06.510
The above command with -resize 500% tells ImageMagick to run the convert command and scale the embedded image IM logo: by 500% in each direction. The -bench 10 reports that it runs the same command 10 times in a loop, and then prints the performance results:
- Since I have OpenMP enabled, I have 4 threads (
Performance[4]: . - It is reported that he performed 10 iterations (
10i ). - The speed was almost 1.5 iterations per second (
1.489ips ). - The total battery life is 6.420 seconds.
If your result includes Performance[1]: and only one line, then your system does not support OpenMP. (You may be able to enable it if your assembly supports it: run convert -limit thread 2 )
4. Change ImageMagick Resource Limits
Find out how your ImageMagick system is configured for resource limits. Use this command:
identify -list resource
File Area Memory Map Disk Thread Time
-------------------------------------------------- ------------------
384 8.590GB 4GiB 8GiB unlimited 4 unlimited
My current system settings are shown above (and not the default values - I already configured them in the past). Numbers is the maximum amount of each resource that ImageMagick will use. You can use each of the keywords in the column headings to pimp your system. To do this, use convert -limit <resource> <number> to set it to a new limit.
Perhaps your result would look like this:
identify -list resource
File Area Memory Map Disk Thread Time
-------------------------------------------------- ------------------
192 4.295GB 2GiB 4GiB unlimited 1 unlimited
files defines the maximum simultaneously open files that ImageMagick can use.- The resource,
memory , map , area and disk resource limits are defined in bytes. To set them to different values, you can use SI prefixes, for example 500 MB).
If you do have OpenMP for ImageMagick on your system, you can run.
convert -limit thread 2
Include 2 parallel threads as a first step. Then run the test and see if it really matters, and if so, how much. After that, you can set the limit to 4 or even 8 and repeat the exercise ....
5. Use the Magic Pixel Cache (MPC) and / or the Magic Permanent Register (MPR)
Finally , you can experiment with ImageMagick's special internal pixel cache format. This format is called MPC (Magick Pixel Cache). It exists only in memory.
When an MPC is created, the processed input image is saved in RAM as an uncompressed raster format. Thus, MPC is ImageMagick's own uncompressed memory format in memory. This is just a direct dump of memory to disk. Reading is a quick memory card from disk to memory as needed (similar to exchanging a memory page). But image decoding is not required.
(Additional technical details: MPC is not portable as a format. It is also not suitable as a long-term archive format. Its only suitability is an intermediate format for high-performance image processing. Two files are required to support one image.)
If you still want to save this format to disk, keep this in mind:
- Image attributes are written to a file with the extension .mpc.
- Image icons are written to a file with a .cache extension.
Its main advantage arises when ...
- ... processing very large images or when
- ... applying multiple operations on the same image in "control pipelines".
MPC was designed specifically for workflow templates that meet the criteria for "read many times, write once."
Some say that for such operations, performance improves here, but I have no personal experience with it.
First convert the base image to MPC:
convert input.jpeg input.mpc
and only then run:
convert input.mpc [...your long-long-long list of crops and operations...]
Then see how much this will save your time.
Most likely, you can use this MPC format even "inline" (using the special mpc: notation, see below).
The MPR format (persistent memory register) does something similar. It reads the image into a named memory register. The process pipeline can also read the image from this register again if it needs to access it several times. The image is saved in the register from which the current instruction pipeline exits.
But I never applied this technique to a real problem, so I can’t say how it works in real life.
6. Build a suitable IM processing pipeline to complete all tasks in one go
As you describe your process, it consists of 4 highlighted steps:
- Convert TIFF to JPEG.
- Resize JPEG image to xx (? What value?)
- Crop JPEG to 200px.
- Add a text watermark.
Tell me, did I understand your intentions from reading code fragments correctly:
- You have 1 input file, TIFF.
- You want to get 2 final output files:
- 1 thumbnail JPEG, size 200x200 pixels;
- 1 tagged JPEG, with a width of 1024 pixels (aspect ratio while maintaining the input TIFF);
- 1 (untagged) JPEG is just an intermediate file that you really don't want to store.
In principle, each step uses its own team - only 4 teams. This can be greatly accelerated by using a single command pipeline that performs all the steps on its own.
In addition, you apparently do not need to save unlabeled JPEG as the final result, but your one command to create it as an intermediate temporary file saves it to disk. We can try to skip this step as a whole and try to achieve the final result without additional writing to disk.
Different approaches are possible for this change. I will show you (and other readers) only one - and only for the CLI, not for PHP. I'm not a PHP guy - it is your own job to “translate” my CLI method to the appropriate PHP calls.
(But, anyway, please check my commands first, really using the CLI to see if the translation approach is worth the PHP!)
But first, make sure you really understand the architecture and structure of the more complex ImageMagick commands! For this purpose, please refer to this other answer :
Your 4 steps are converted to the following separate ImageMagick commands:
convert image.tiff image.jpg convert image.jpg -resize 1024x image-1024.jpg convert image-1024.jpg -thumbnail 200x200 image-thumb.jpg convert -background white image-1024.jpg label:12345 -append image-labelled.jpg
Now, to convert this workflow into one pipeline command ... The following command is executed. It should run faster (no matter what your results are, following my previous steps 0 .-- 4. ):
convert image.tiff \ -respect-parentheses \ +write mpr:XY \ \( mpr:XY +write image-1024.jpg \) \ \( mpr:XY -thumbnail 200x200 +write image-thumb.jpg \) \ \( mpr:XY -background white label:12345 -append +write image-labelled.jpg \) \ null:
Explanations:
-respect-parentheses : to really make subcommands executed inside brackets \( .... \) independent of each other.+write mpr:XY : used to write the input file to the MPR memory register. XY is just a label (you can use anything) that is needed for subsequent recall of the same image.+write image-1024.jpg : writes the result of a subcommand executed inside the first pair of parentheses to disk.+write image-thumb.jpg : writes the result of a subcommand executed in the second pair of parentheses to disk.+write image-labelled.jpg : writes the result of a subcommand executed inside a pair of third parentheses to disk.null: terminates the command pipeline. Required, because otherwise we would end the last copied sub-command bracket.
7. Benchmarking 4 separate teams compared to a single pipeline
To get a rough idea of my proposal, I ran the commands below.
The first one starts a sequence of 4 separate commands 100 times (and saves all the resulting images under different file names).
time for i in $(seq -w 1 100); do convert image.tiff \ image-indiv-run-${i}.jpg convert image-indiv-run-${i}.jpg -sample 1024x \ image-1024-indiv-run-${i}.jpg convert image-1024-indiv-run-${i}.jpg -thumbnail 200x200 \ image-thumb-indiv-run-${i}.jpg convert -background white image-1024-indiv-run-${i}.jpg label:12345 -append \ image-labelled-indiv-run-${i}.jpg echo "DONE: run indiv $i ..." done
My result for 4 separate commands (repeated 100 times!):
real 0m49.165s user 0m39.004s sys 0m6.661s
The second command starts one pipeline:
time for i in $(seq -w 1 100); do convert image.tiff \ -respect-parentheses \ +write mpr:XY \ \( mpr:XY -resize 1024x \ +write image-1024-pipel-run-${i}.jpg \) \ \( mpr:XY -thumbnail 200x200 \ +write image-thumb-pipel-run-${i}.jpg \) \ \( mpr:XY -resize 1024x \ -background white label:12345 -append \ +write image-labelled-pipel-run-${i}.jpg \) \ null: echo "DONE: run pipeline $i ..." done
The result for one pipeline (repeated 100 times!):
real 0m29.128s user 0m28.450s sys 0m2.897s
As you can see, a single pipeline is approximately 40% faster than four separate commands!
Now you can also invest in multiprocessor, a lot of RAM, fast hardware SSD, to speed up the work :-)
But first, translate this CLI approach into PHP code ...
There are a few more things to say about this topic. But the time has come. I will probably return to this answer in a few days and update it more ...
Update: I had to update this answer with new benchmarking numbers: initially I forgot to include the -resize 1024x operation (dumb me!) In the pipelined version. By turning it on, the performance gain still exists, but not so big.
8. Use -clone 0 to copy the image to memory
Here is another alternative to try instead of the mpr: approach mpr: with a named memory register, as suggested above.
It uses (again inside "side processing in parentheses") the -clone 0 operation. How it works:
convert reads the input TIFF from the disk once and loads it into memory.- Each
-clone 0 statement creates a copy of the first loaded image (since it has index 0 in the current image stack). - The sub-category "inside-brackets" of the complete pipeline of commands performs some operation on the clone.
- Each
+write operation saves the corresponding result to disk.
So here is the command to compare:
time for i in $(seq -w 1 100); do convert image.tiff \ -respect-parentheses \ \( -clone 0 -thumbnail 200x200 \ +write image-thumb-pipel-run-${i}.jpg \) \ \( -clone 0 -resize 1024x \ -background white label:12345 -append \ +write image-labelled-pipel-run-${i}.jpg \) \ null: echo "DONE: run pipeline $i ..." done
My result:
real 0m19.432s user 0m18.214s sys 0m1.897s
, , , mpr: !
9. -scale -sample -resize
, , . , , ( , ).
-resize , -sample -scale . :
:
time for i in $(seq -w 1 100); do convert image.tiff \ -respect-parentheses \ \( -clone 0 -thumbnail 200x200 \ +write image-thumb-pipel-run-${i}.jpg \) \ \( -clone 0 -scale 1024x \ -background white label:12345 -append \ +write image-labelled-pipel-run-${i}.jpg \) \ null: echo "DONE: run pipeline $i ..." done
My result:
real 0m16.551s user 0m16.124s sys 0m1.567s
( +clone ).
, , 4 .
10. Q8 -depth 8 .
, .
time for i in $(seq -w 1 100); do convert image.tiff \ -respect-parentheses \ \( -clone 0 -thumbnail 200x200 -depth 8 \ +write d08-image-thumb-pipel-run-${i}.jpg \) \ \( -clone 0 -scale 1024x -depth 8 \ -background white label:12345 -append \ +write d08-image-labelled-pipel-run-${i}.jpg \) \ null: echo "DONE: run pipeline $i ..." done
" 4 " - .
11. GNU parallel ,
, , , .
. - , ...
time for i in $(seq -w 1 100); do \ cat <<EOF convert image.tiff \ \( -clone 0 -scale 1024x -depth 8 \ -background white label:12345 -append \ +write d08-image-labelled-pipel-run-${i}.jpg \) \ \( -clone 0 -thumbnail 200x200 -depth 8 \ +write d08-image-thumb-pipel-run-${i}.jpg \) \ null: echo "DONE: run pipeline $i ..." EOF done | parallel --will-cite
Results:
real 0m6.806s user 0m37.582s sys 0m6.642s
user real : user , 8 .
, , : 10 .
12.
- :