Anti-unpacker tricks – part nine

New anti-unpacking tricks continue to be developed as older ones are constantly being defeated. Last year, a series of articles described some tricks that might become common in the future, along with some countermeasures [1], [2], [3], [4], [5], [6], [7], [8], [9]. Now, the series continues with a look at tricks that are specific to debuggers and emulators.

In this article we look at anti-debugging tricks including self-modifying code, selectors, RDTSC and Syser plug-ins.

Unless stated otherwise, all of the techniques described here were discovered and developed by the author.

If a debugger uses the common ‘CC’ opcode (short-form ‘INT 3’ instruction) to place breakpoints during step-over, then it is vulnerable to self-modifying code that removes the breakpoint. As a result, the debugger’s control of the process will be lost. Example code looks like this:

 mov al, 90h
 xor ecx, ecx
 inc ecx
 mov edi, offset l1
 rep stosb
l1: nop

Of course, there are a couple of variations, such as using ‘rep movs’ instead of ‘rep stos’. The direction flag can be involved, too, in such a way that at a glance, the overwrite might be overlooked. Example code looks like this:

 mov al, 90h
 push 2
 pop ecx
 mov edi, offset l1
 std
 rep stosb
 nop 
l1: nop

As noted in a previous paper [1], single-stepping is also vulnerable to a variation of this technique, if the overwrite includes the string instruction itself.

The solution to this problem is to use hardware breakpoints instead, though this workaround has its own set of problems. What is not immediately obvious in this example is that the debugger has no way of knowing if the breakpoint that it places at a location is the one that is executed. If the application removes the breakpoint, it can restore it afterwards, and then jump to the address to execute that breakpoint. The debugger will see the breakpoint exception that it was expecting, and behave as normal. Example code looks like this:

 mov al, 90h
l1: xor ecx, ecx
 inc ecx
 mov edi, offset l3
l2: rep stosb
l3: nop
 cmp al, 0cch
l4: mov al, 0cch
 jne l1
l5: ...

In this example, stepping over the instruction at l2 will allow the code to reach l4. This will cause the breakpoint to be replaced by l2 and executed by l3. The debugger will then regain control. At that time, the only obvious difference will be that the AL register will hold the value 0xCC instead of 0x90, which will allow l5 to be reached in what appears to be one pass instead of two. Of course, much more subtle variations are possible, including the execution of entirely different code-paths.

A variation of the technique can be used as a simple method to detect the presence of a debugger. Example code looks like this:

 xor ecx, ecx
 inc ecx
 mov esi, offset l1
 lea edi, [esi + 1]
 rep movsb
l1: mov al, 90h
 cmp al, 0cch
 je being_debugged

Selector values look stable, but they are actually volatile. Specifically, a selector value can be set within a thread, but it might not hold its value for very long. Certain events will cause the value to be changed back. One such event is an exception. In the context of a debugger, the single-step exception can cause some unexpected behaviour. Example code looks like this:

 xor  eax, eax
 push fs
 pop  ds
l1: xchg [eax], cl
 xchg [eax], cl

Single-stepping through this code will cause an access violation exception at l1 because the DS selector will be restored to its default value even before l1 is reached.

A variation of this technique detects the single-step event in a less obvious fashion, simply by checking if the assignment was successful. Example code looks like this:

push 3
pop gs
mov ax, gs
cmp al, 3
jne being_debugged

This technique is used by Zlob. However, this code is vulnerable to a race condition caused by a thread-switch event, because a thread-switch event also results in the selectors being restored to their default values.

A variation of this technique waits intentionally for a thread-switch event to occur, in order to trigger the effect. Example code looks like this:

 push 3
 pop gs
l1: mov ax, gs
 shr ax, 1
 jb l1

This technique is used by Zlob. The code expects the GS selector to become zero again when a thread-switch occurs. This technique works only on the 32-bit versions of Windows. It is invalid for 64-bit versions of Windows because the GS value on those platforms has bit 0 set, so the loop never exits.

This technique is actually a variation of an anti-emulation trick first seen in 2000, which has been rediscovered. At that time, selectors were not well-supported, so assignments often misbehaved or were ignored completely. Example code looks like this:

mov eax, ds
xor ebx, ebx
mov ds, bx
mov ecx, ds
cmp ecx, eax
;detect selector not updated
je  being_debugged

This technique was first used by Moridin, but Moridin simply checked if the selector held its value when assigned the same value. Example code looks like this:

mov edi, ds
push edi
pop ds
mov eax, ds
cmp edi, eax
jnz being_debugged

When the system is powered-on, a timer starts to run, whose value can be queried by the RDTSC instruction. Given how long a typical system takes to boot, and how long it takes for an arbitrary application to be launched, the value that is returned by the RDTSC instruction should be at least x, where x is a quite large value. Unfortunately, it is common for some hiding tools to intercept the RDTSC instruction, and to return a small incrementing value instead, which reveals their presence. It might be better to begin with the real value, but there is a problem with that, too. The problem is that if code knows that it is running at start-up, then too large a value reveals that it was not executed in the usual way. This technique could be used to determine the execution state, instead of checking from which directory the application was launched, for example.

Data-Execution Prevention is intended to disallow the execution of code from pages that are not marked explicitly as executable. However, for compatibility reasons, the protection is not as secure as it sounds. If code begins in an executable section, and jumps into a non-executable section with DEP enabled, then an exception will occur as expected. However, if execution begins in a non-executable section, then the file will run with DEP silently disabled. This is true even if the code jumps into an executable section, and then back into a non-executable section.

Turbo Debug32 (and possibly other debuggers) allows breakpoints to be executed in non-executable pages, even in cases where the execution of any other instructions would cause a DEP exception.

Some packers have been written to detect Syser, so a plug-in (only one so far) has been written to attempt to hide Syser from those packers. The following is a description of that plug-in, along with its very serious bug.

mov  edi, edi
push ebp
mov  ebp, esp
mov  edx, [ebp+10]
...
push ebx
push esi
push edi
...
push d [ebp+30]
push d [ebp+2c]
push d [ebp+28]
push d [ebp+24]
push d [ebp+20]
push d [ebp+1c]
push d [ebp+18]
push d [ebp+14]
push edx
push d [ebp+C]
push d [ebp+8]
call ntcreatefileplus5
pop  edi
pop  esi
pop  ebx
pop  ebp
retn 2Ch

mov  edi, edi
push ebp
mov  ebp, esp
...
xor  eax, eax
jmp  ntcreatefileplus5

OllyDbg was described in a previous paper [6]. The following is a description of a bug that had been discovered since that paper was published.

 xor   ebx, ebx
 push  40h
 mov   eax, esp
 push  3000h
 push  esp
 push  ebx
 push  eax
 push  -1 ;GetCurrentProcess()
 call  NtAllocateVirtualMemory
 mov b [ebx], 0c3h
 call  d [bx+1]
l1: ...