NICKLE1) is a rootkit prevention system that works by prevented unauthorized code from executing with kernel privilege. It does this by creating a Shadow Memory where it stores authenticated (i.e., verified with a hash) kernel code. Instruction fetches while the processor is in kernel mode can be routed to the shadow memory while data accesses go to the standard memory. This means that a rootkit author cannot execute any code he injects into the kernel, as his injected code will exist only in the standard memory. In the event an attempt is made to execute malicious code, NICKLE will rewrite it in order to allow the operating system to respond to the failure gracefully.
We've built NICKLE in QEMU, VirtualBox, and VMWare Workstation. We're able to protect Linux 2.4, Linux 2.6, and Windows 2000/XP. Linux 2.4 has full support (we can even handle valid kernel modules while denying malicious ones) while Linux 2.6 and Windows have a more limited support.
There are two publications corresponding to NICKLE: The conference paper and the technical report. When in doubt, read the conference paper. (The tech report has a few more experiments described and a bit more detail about the VirtualBox report.)
The QEMU source is now available! If you aren't sure which file to get, get the distribution. It includes the source, virtual machine image, binaries, and instructions to run it.