Data Execution Prevention protects a system (e.g. Windows XP SP2 or later) by disabling code execution from data pages. Most of the malicious programs like viruses, spyware etc use buffer overrun flaws in programs to inject and execute their own code. Malicious programs copy executable code into data memory and manipulate the program counter to start executing that code. DEP uses Hardware support (in the CPU) and some software based techniques to prevent code execution from stack and heap memory as both are supposed to store only data.
When an application is launched, OS allocates it a Virtual Address Space (VAS) consisting of memory pages. Each of the pages in a VAS are marked either as code or data. Size of each page depends on the processor. Hardware DEP monitors if a program tries to execute instructions from pages that are marked as data only page. Whenever this happens the CPU generates an exception which is handled by the OS which in turn terminates the offending application.
Intel x86 based processors provide XD bit (Execute Disable bit) for operating systems to implement hardware DEP. AMD has similar bit named NX bit (No Execute bit).
Software-enforced DEP is another form for protection implemented in software (as in Windows XP SP2). It doesn’t need NX bit support in the CPU. This mechanism only provide protection from malicious code which uses flaws in Structured Exception Handling support available in the Windows OS.
To explore and play with DEP settings on your Windows system, go to Start -> Settings -> Control Panel -> System -> System Properties -> Advanced Tab -> (Performance) Settings -> Data Execution Prevention.
Have safe computing!