The caller and the caller must accept what the registers and the stack contain. This is called a calling convention , which is part of a larger concept called the binary application interface (ABI) . The call determines how it wants to be called (i.e., Should the arguments be on the stack, in the register, etc.), And the compiler ensures that the code it generates matches the calling convention.
As for your specific question, it depends on the ABI. Sometimes, if the return value is more than 4 bytes, but not more than 8 bytes, it can be divided into EAX and EDX. But most of the time, the calling function simply allocates some memory (usually on the stack) and passes a pointer to this area of the called function.
Please also note that the role of the OS is not as important as you think. Binary groups with different calling conventions can coexist on the same system, and binary files can even use different calling conventions within the country. ABI OS only matters when binary calls its system libraries.
sam hocevar
source share