How to Hook Thiscall Functions?

Yesterday I found a really interesting way to hook functions compiled with thiscall calling convention.
In the following post I will try to explain the method.

 A Bit About x86 Calling Conventions

According to Wikipedia:

Calling conventions describe the interface of called code:

• The order in which atomic (scalar) parameters, or individual parts of a complex parameter, are allocated
• How parameters are passed (pushed on the stack, placed in registers, or a mix of both)
• Which registers may be used by the callee without first being saved (i.e. pushed)
• How the task of setting up for and restoring the stack after a function call is divided between the caller and the callee

First, lets review some of the most important calling conventions:

• stdcall – Parameters are pushed right-to-left. The callee is responsible for cleaning up the stack.
• cdecl – Parameters are pushed right-to-left. The caller is responsible for cleaning up the stack.
• thiscall – Parameters are pushed right-to-left. Either caller or callee is responsible for cleaning up the stack – depending if variable number of arguments is being used. A this pointer is passed in the ECX register.
• fastcall – Passes the first two arguments(left-to-right) that fit into ECX and EDX. Remaining arguments are pushed onto the stack from right to left. The callee is responsible for cleaning up the stack.

Why it’s different to hook a thiscall function?

Well, let’s say you want to declare a function that will get called instead of the original detoured function.
You probably want your detour function prototype to be exactly the same as the original – so the stack won’t get corrupted. For example, let’s say we want to perform hook on a function with the following prototype:

char* __thiscall Foo(unsigned int firstParam, unsigned int secondParam);

One would try to declare a detour function that looks like this:

char* __thiscall myFoo(unsigned int firstParam, unsigned int secondParam);

But the problem is that VC++ won’t let you to explicitly define a thiscall function. If you’ll try to do it, the following error will be raised:

error C3865: '__thiscall' : can only be used on native member functions

This is rational – thiscall calling convention needs a “this” pointer which is available only when defining a method within a class.

So, how can we hook functions compiled with thiscall?

We can, of course, declare our function as __naked__  and write the function’s prologue and epilogue ourselves, but this is not fun at all – with every change to the function we’ll need to write assembly code to fix the stack.

But there is a better way to do it. Look again at the definition of thiscall: The this pointer should be passed in the ECX register and the callee should clean the stack.
Now, let’s look at the definition of fastcall: The two first parameters that fits into ECX and EDX are passed in these registers.
If we declare a function that uses the fastcall calling convention with the first parameter as This and the seconds parameter as a trash(not used) parameter, we could use this calling convention instead of thiscall, and the compiler won’t raise any error!
So, you can declare a function pointer(with a reference to the This pointer) and a detour function(with a reference to the This pointer and to a dummy seconds parameter), and the compiler will do all the rest for you:

typedef int (__thiscall *myFooPtr)(void*, unsigned int, unsigned int);
char* __fastcall myFoo(void* This, void* notUsed, unsigned int firstParam,
unsigned int secondParam) { }

The compiler will place the This pointer into ecx and the second dummy parmeter into EDX, as these are the first and the second parameters and we’re using fastcall calling convention.
The remaining parameters will be pushed into the stack.

This way, we can take advantage of the fastcall calling convention, and the compiler will do all the dirty stack work for us. Cool, huh? 🙂

Cy’a,