How to make my program DEP compatible?

Question

I have a windows project (.net 3.0) that will not run on my Vista computer for the user due to a DEP error. It works on my Vista machine and on a clean version of vista sp1 in a virtual machine. I'm having trouble finding paths for the compatibility of my DEP program, preventing data execution. I really can’t do anything to finish the custom machines, it just needs to be started. Is there any way out of this last nightmare of development perspectives? My program uses devexpress, sql express and .net controls, i.e. Manage your web browser. I already mixed tag control, but to no avail. I have another program that uses devexpress and sql express on the same machine, and they work fine. I find it difficult to debug this on the user's computer.

Answer 1

DEP starts in one of two modes:

1) DEP hardware is designed for processors that may mark memory pages as unreachable. This helps prevent some exploits, such as buffer overflows.

2) DEP software is for processors that do not have DEP hardware support. This does not prevent code from executing on data pages, but instead cancels overwriting SEH (another type of attack).

In Windows XP with processors supporting it, DEP hardware is enabled by default only for certain Windows system binaries, as well as for programs that prefer to “opt out”.

In Vista with processor support, hardware DEP is enabled by default for almost all processes. This can sometimes be problematic, usually for older programs and drivers, as well as for independent software vendors who have not tested Vista.

Therefore, I suspect that the first step is to find out if you are dealing with software or hardware DEP. Also, are you using C # / VB or managed C ++? And do you use any native code or components? If your application uses its own component or ActiveX control that was created using the old ATL infrastructure, it is possible that your application terminates with hardware DEP.

Starting with the .NET Framework 2.0 SP1, I believe that the C # compiler emits DEP compatible managed code. But if your application throws DEP exceptions, you can try to clear the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag for your executable. To do this, you use EDITBIN.EXE from the VC toolkit as follows:

editbin.exe /NXCOMPAT:NO <your binary> 

If you are using Visual Studio, you can add a post-build step to your executable project. You need to set up your environment so that you can solve problems with EDITBIN changes. When I use native code as part of my application, the post-build step is as follows:

 call $(DevEnvDir)..\tools\vsvars32.bat editbin.exe /NXCOMPAT:NO $(TargetPath) 

Answer 2

Older versions of ATL are not recognized by DEP, so if you use any ActiveX controls created using ATL and were built on this version of ATL (version 7.1 and below, I think), you will get DEP errors.

As a last resort, you can actually disable DEP for the process by calling the API function: SetProcessDEPPolicy .

Additional Information about SetProcessDEPPolicy

Answer 3

Compilers shipped with .NET 2.0 SP1 include the NXCOMPAT flag in the header of the executable file. You can disable this flag in the Post Build phase by running EditBin.exe with the / NXCOMPAT: NO switch.

Answer 4

FWIW, it’s worth mentioning directly that in many cases the applications are not “incompatible with DEP”, but most likely will crash and DEP will “drop by to save the day”. Very often, as soon as you turn off DEP, you will find that you are pressing "normal" AV.

If your project is written exclusively in .NET 3.0, it is almost certain, because .NET does not do any “crazy” things that run DEP (for example, the thunking function, etc.).

To debug, install a debugger, or activate Watson to create a .DMP file, then take this .DMP file to the developer's machine and find out what went wrong.

Answer 5

Start by trying to figure out where and how your program fails. Can you reproduce the problem on your system? With DEP support for an application on your system? When you can replicate a problem and receive an error message (access violation), you can look at fixing your program.

See the MSDN article for information on DEP .