I have a DLL that I am inserting into other processes using SetWindowsHookEx. Inside the DLL, I increment the module reference count by invoking GetModuleHandleExso that I can control when the module is unloaded.
At this point, the module reference count should be “2” of both of these API calls. When the calling process shuts down, it calls UnhookWindowsHookEx, reducing the number of links to 1. There is a thread in the DLL that waits for several things, one of which is a handle to the process that is called SetWindowsHookEx. When the process leaves, the DLL does some cleanup, terminates all threads, cleans up memory and processes, and then calls FreeLibraryAndExitThread. This decreases the counter, and the DLL is unloaded.
Here is my problem. There are several processes, especially without a user interface, where the DLL is never unloaded. I'm pretty sure that I cleaned everything. And I know that none of my threads work.
First of all, if you have troubleshooting tips to help uncover the cause, this will be helpful. Otherwise, I thought about using some kind of API, such as NtQueryInformationProcessto get the module address and confirm that the number of module handlers is actually zero, then call CreateRemoteThreadto call LdrUnloadDllit to unload the module address from the inside to process. What are your thoughts on this approach? Does anyone have some sample code? I am having difficulty figuring out how to get the module counter.