Conventional Call Convention

Question

I am debugging an x86 DLL in windbg, in particular a function that supposedly has the following signature:

bool __cdecl func(LPVOID p1, LPVOID p2, wchar_t* p3, size_t p4, LPVOID p5)

The function is not exported. AFAIK __cdeclmust receive all arguments on the stack, and the caller must clear the stack.

But that is not what is happening. Windbg says the calling convention __cdecl, but the first 2 arguments are passed in ecxand edx, as a function __fastcall. Also, the function itself clears the stack, which (I think) should not be executed using the function __cdecl.

I try to use this function without success. I tried to make a detour function like __cdecland __fastcall, and both of them have caused a failure.

Any suggestions?

Answer 1

It is possible that if the function is in the executable file (and not in the library) or is called from only one DLL, then the compiler can optimize the calling convention at its discretion.

If the compiler knows both ends (the calling and the called) and knows that the function will not be exported to another block (as it would be with the library), then it can be optimized in any way.

So: Is the function part of the finished executable? Do you have any optimization?

I recommend disabling optimization and try again.