libemu + fnstenv = no detection

Today the blog post «Shellcode detection using libemu» reminded me of looking into libemu's detection capabilities. Currently, I'm writing my Bachelor Thesis about this topic, or more specifically a faster alternative to libemu (with better detection rates, I hope). Hence, I've got quite a repository of shellcode at my disposal.

A current checkout of libemu failed at detecting the first shellcode I've tried:

$ udcli -32 sc | head -n 3
0000000000000000 d9ea             fldl2e            
0000000000000002 33c9             xor ecx, ecx      
0000000000000004 d97424b4         fnstenv [esp-0x4c]     
0000000000000008 8b5c24c0         mov ebx, [esp-0x40]
$ sctest -gS -s 100000 < sc
verbose = 2
[emu 0x0x16f1560 debug ] in  emu_shellcode.c:470>
failed
[emu 0x0x16d60e0 debug ] cpu state    eip=0x00416fff
[emu 0x0x16d60e0 debug ] eax=0x00000000  ecx=0x00000000  edx=0x00000000  ebx=0x00000000
[emu 0x0x16d60e0 debug ] esp=0x00416fce  ebp=0x00000000  esi=0x00000000  edi=0x00000000
[emu 0x0x16d60e0 debug ] Flags: 
[emu 0x0x16d60e0 debug ] 00D9                            add cl,bl
[emu 0x0x16d60e0 debug ] EA33C9D97424B4                  jmpf 0xb424:0x74d9c933
cpu error error accessing 0x00000004 not mapped

stepcount 1
[emu 0x0x16d60e0 debug ] cpu state    eip=0x00417002
[emu 0x0x16d60e0 debug ] eax=0x00000000  ecx=0x00000000  edx=0x00000000  ebx=0x00000000
[emu 0x0x16d60e0 debug ] esp=0x00416fce  ebp=0x00000000  esi=0x00000000  edi=0x00000000
[emu 0x0x16d60e0 debug ] Flags: PF ZF
$

So libemu fails to even recognize the fnstenv instruction as potential GetPC sequence. This irritaded me quite a bit, since I remembered that libemu normally has support for fnstenv, so I took a quick look into emu_getpc.c and around line 121, I saw the reason:

                if ( c->instr.fpu.ea == emu_cpu_reg32_get(c, esp) - 0xc )
                {
//                      printf("found fnstenv with ea = esp - 0xc\n");
                        return 1;
                }

There is some serious overspecialization, libemu will only consider fnstenv [esp-0xc] sequences as potential GetPC. Due to the above example this is obviously not enough. For example, libscizzle detects these shellcodes without problems:

$ ./libscizzle-test < sc 
[*] Filtering / scanning over 171.0 B of data took 1 ms.
[*] Verifying 1 shellcode candidate offsets...
  00014000[            d9ea] > fldl2e 
  00014002[            33c9] > xor ecx,ecx
  00014004[................] <
Emulating 00014004: fstenv [esp-0x4c]
FSTENV fixup to 11eb4+c = 14000
  00014008[        8b5c24c0] ! mov ebx,[esp-0x40] 1 ->  [11ec0] [stack]
...
[+] Found verified shellcode at offset 0
[*] Verification over 171.0 B of data took 48 ms.
[*] Everything over 171.0 B of data took 49 ms.