Binding phase in distcc

Question

Is there any special reason that the binding phase when creating a project with distcc is performed locally and not sent to other computers that need to be executed, for example, to compile? Reading the white pages of distcc did not give a clear answer, but I assume that the time taken to link the object files is not very significant compared to compiling. Any thoughts?

Answer 1

The way distcc works is local preprocessing of input files until a single file conversion unit is created. This file is then sent over the network and compiled. At this point, the distcc remote server only needs the compiler, it doesn’t even need the header files for the project. The compilation result is then returned back to the client and saved locally as an object file. Please note that this means that not only binding, but also preprocessing is performed locally. This separation of work is common to other build tools such as ccache (preprocessing is always performed, then it tries to allow input with previously cached results, and if it successfully returns the binary without recompiling).

If you plan to implement a distributed linker, you will either have to make sure that all the hosts on the network have the same configuration, or you will have to send all the necessary inputs for the operation in one batch. This implies that distributed compilation will result in a collection of object files, and all these object files must be redirected over the network for the remote system to link and return the linked file. Please note that this may require system libraries that are referenced and present in the linker path, but are not present on the linker command line, so a “preliminary link” will need to determine which library set is required for sending. Even if this is possible, it will require the local system to guess / calculate all real dependencies and send them with a big impact on network traffic and can actually slow down the process, since the cost of sending can be more than the cost of linking - if the cost of getting the dependencies is It’s not as expensive as a snap.

The project I'm working on now has one statically linked executable file over 100M. Static libraries have a size in size, but if the distributed system believes that the final executable should be deleted remotely, this will probably require three to five times more network traffic than the final executable (templates, built-in ... all this appears in all translation units that include them, so there will be many copies on the network).

Answer 2

Linking, by definition, requires all the object files in one place. Since distcc discards the object files on the computer, which primarily cause distcc, the local machine is the best place to execute the link because the objects are already present.

In addition, remote linking will become especially attractive when you start throwing libraries into the mix. If the remote machine linked your program to its local version of the library, you discovered potential problems when the linked binary was returned to the local machine.

Answer 3

The reason that complication can be sent to other machines is because each source file is compiled independently of the others. Simply put, for each input .c file there is an output .o file at compile time. This can be done on any other computer as you like.

Binding, on the other hand, collects all .o files and creates a single binary output file. There is not much to spread.