x86 Disassembly/Calling Convention Examples

From Wikibooks, the open-content textbooks collection

[edit] Microsoft C Compiler

Here is a simple function in C:

int MyFunction(int x, int y)
 {
        return (x * 2) + (y * 3);
 }

Using cl.exe, we are going to generate 3 separate listings for MyFunction, one with CDECL, one with FASTCALL, and one with STDCALL calling conventions. On the commandline, there are several switches that you can use to force the compiler to change the default:

• /Gd : The default calling convention is CDECL
• /Gr : The default calling convention is FASTCALL
• /Gz : The default calling convention is STDCALL

Using these commandline options, here are the listings:

[edit] CDECL

int MyFunction(int x, int y)
 {
        return (x * 2) + (y * 3);
 }

becomes:

PUBLIC        _MyFunction
 _TEXT  SEGMENT
 _x$ = 8                                               ; size = 4
 _y$ = 12                                              ; size = 4
 _MyFunction    PROC NEAR
 ; Line 4
        push   ebp
        mov    ebp, esp
 ; Line 5
        mov    eax, DWORD PTR _y$[ebp]
        imul   eax, 3
        mov    ecx, DWORD PTR _x$[ebp]
        lea    eax, DWORD PTR [eax+ecx*2]
 ; Line 6 
        pop    ebp
        ret    0
 _MyFunction    ENDP
 _TEXT  ENDS
 END

As you can clearly see, parameter y was pushed first, because it has a higher offset from ebp than x does. Both x and y are accessed as offsets from ebp, so we know they are located on the stack. For that matter, the function sets up a standard stack frame as well. The function does not clean it's own stack, as you can see from the "ret 0" instruction at the end. It is therefore the callers duty to clean the stack after the function call.

As a point of interest, notice how lea is used in this function to simultaneously perform the multiplication (ecx * 2), and the addition of that quantity to eax. Unintuitive instructions like this will be explored further in the chapter on Unintuitive Instructions.

[edit] FASTCALL

int MyFunction(int x, int y)
 {
        return (x * 2) + (y * 3);
 }

becomes:

PUBLIC        @MyFunction@8
 _TEXT  SEGMENT
 _y$ = -8                                              ; size = 4
 _x$ = -4                                              ; size = 4
 @MyFunction@8 PROC NEAR
 ; _x$ = ecx
 ; _y$ = edx
 ; Line 4
        push   ebp
        mov    ebp, esp
        sub    esp, 8
        mov    DWORD PTR _y$[ebp], edx
        mov    DWORD PTR _x$[ebp], ecx
 ; Line 5
        mov    eax, DWORD PTR _y$[ebp]
        imul   eax, 3
        mov    ecx, DWORD PTR _x$[ebp]
        lea    eax, DWORD PTR [eax+ecx*2]
 ; Line 6
        mov    esp, ebp
        pop    ebp
        ret    0
 @MyFunction@8 ENDP
 _TEXT  ENDS
 END

This listing is very interesting. I want the reader to keep one important point in mind before I start talking about this function: This function was compiled with optimizations turned off. With that point in mind, let's examine it a little bit. First thing we notice is that on line 4, a standard stack frame is set up, and then ebp is decremented by 8. Why does it do this? The function might not receive the parameters on the stack, but the cl.exe code generation back-end is expecting the parameters to be on the stack anyway! This means that space needs to be allocated on the stack, and the parameters need to be moved out of ecx and edx, and moved onto the stack. This is made even more ridiculous by the fact that parameter x is moved out of ecx in the beginning of the function, and is moved back into ecx on line 5. Hopefully the optimizer would catch this nonsense if the optimizer was turned on.

It is difficult to determine which parameter is passed "first" because they are not put in sequential memory addresses like they would be on the stack. However, the Microsoft documentation claims that cl.exe passes fastcall parameters from left-to-right. To prove this point, let's examine a simple little function with only one parameter, to see which register it is passed in:

int FastTest(int z)
 {
        return z * 2;
 }

And cl.exe compiles this listing:

PUBLIC        @FastTest@4
 _TEXT  SEGMENT
 _z$ = -4                                              ; size = 4
 @FastTest@4 PROC NEAR
 ; _z$ = ecx
 ; Line 2
        push   ebp
        mov    ebp, esp
        push   ecx
        mov    DWORD PTR _z$[ebp], ecx
 ; Line 3
        mov    eax, DWORD PTR _z$[ebp]
        shl    eax, 1
 ; Line 4
        mov    esp, ebp
        pop    ebp
        ret    0
 @FastTest@4 ENDP
 _TEXT  ENDS
 END

So it turns out that the first parameter passed is passed in ecx. We notice in our function above that the first parameter (x) was passed in ecx as well. Therefore, parameters really are passed left-to-right, unlike in CDECL.

Notice 2 more details:

• The name-decoration scheme of the function: @MyFunction@8.
• The "ret 0" function seems to show that the caller cleans the stack, but in this case, there is nothing on the stack to clean. It is unclear who will clean the stack, from this listing. If we take a look at yet one more mini-example:

int FastTest(int x, int y, int z, int a, int b, int c)
 {
     return x * y * z * a * b * c;
 }

and the corresponding listing:

PUBLIC        @FastTest@24
 _TEXT  SEGMENT
 _y$ = -8                                              ; size = 4
 _x$ = -4                                              ; size = 4
 _z$ = 8                                               ; size = 4
 _a$ = 12                                              ; size = 4
 _b$ = 16                                              ; size = 4
 _c$ = 20                                              ; size = 4
 @FastTest@24 PROC NEAR
 ; _x$ = ecx
 ; _y$ = edx
 ; Line 2
        push   ebp
        mov    ebp, esp
        sub    esp, 8
        mov    DWORD PTR _y$[ebp], edx
        mov    DWORD PTR _x$[ebp], ecx
 ; Line 3
        mov    eax, DWORD PTR _x$[ebp]
        imul   eax, DWORD PTR _y$[ebp]
        imul   eax, DWORD PTR _z$[ebp]
        imul   eax, DWORD PTR _a$[ebp]
        imul   eax, DWORD PTR _b$[ebp]
        imul   eax, DWORD PTR _c$[ebp]
 ; Line 4
        mov    esp, ebp
        pop    ebp
        ret    16                                     ; 00000010H

We can clearly see that in this case, the callee is cleaning the stack, which we can safely assume will happen every time. An important point to notice about this function is that only the first 2 parameters are passed in registers. The first of which is passed in ecx, and the second of which is passed in edx. All the remaining arguments are clearly passed on the stack, in right-to-left order. It seems that the first 2 arguments are passed left-to-right, but all the remaining arguments are passed right-to-left.

[edit] STDCALL

int MyFunction(int x, int y)
 {
        return (x * 2) + (y * 3);
 }

becomes:

PUBLIC        _MyFunction@8
 _TEXT  SEGMENT
 _x$ = 8                                               ; size = 4
 _y$ = 12                                              ; size = 4
 _MyFunction@8 PROC NEAR
 ; Line 4
        push   ebp
        mov    ebp, esp
 ; Line 5
        mov    eax, DWORD PTR _y$[ebp]
        imul   eax, 3
        mov    ecx, DWORD PTR _x$[ebp]
        lea    eax, DWORD PTR [eax+ecx*2]
 ; Line 6
        pop    ebp
        ret    8
 _MyFunction@8 ENDP
 _TEXT  ENDS
 END

Notice that y is a higher offset from ebp, which indicates that these arguments are passed on the stack from left-to-right, instead of right-to-left as the Microsoft documentation claims. The proof is in the pudding, it would seem. The STDCALL listing is almost identical to the CDECL listing except for the last instruction, which says "ret 8". This function is clearly cleaning it's own stack. Notice the name-decoration scheme, with an underscore in front, and an "@8" on the end, to denote how many bytes of arguments are passed. Lets do an example with more parameters:

int STDCALLTest(int x, int y, int z, int a, int b, int c)
 {
        return x * y * z * a * b * c;
 }

Let's take a look at how this function gets translated into assembly by cl.exe:

PUBLIC        _STDCALLTest@24
 _TEXT  SEGMENT
 _x$ = 8                                               ; size = 4
 _y$ = 12                                              ; size = 4
 _z$ = 16                                              ; size = 4
 _a$ = 20                                              ; size = 4
 _b$ = 24                                              ; size = 4
 _c$ = 28                                              ; size = 4
 _STDCALLTest@24 PROC NEAR
 ; Line 2
        push   ebp
        mov    ebp, esp
 ; Line 3
        mov    eax, DWORD PTR _x$[ebp]
        imul   eax, DWORD PTR _y$[ebp]
        imul   eax, DWORD PTR _z$[ebp]
        imul   eax, DWORD PTR _a$[ebp]
        imul   eax, DWORD PTR _b$[ebp]
        imul   eax, DWORD PTR _c$[ebp]
 ; Line 4
        pop    ebp
        ret    24                                     ; 00000018H
 _STDCALLTest@24 ENDP
 _TEXT  ENDS
 END

Notice the name decoration, and how there is now "@24" appended to the name, to signify the fact that there are 24 bytes worth of parameters. Notice also how x has the lowest offset, and how c has the highest offset, indicating that c (the right-most parameter) was passed first, and that x (the left-most parameter) was passed last. Therefore it's clearly a right-to-left passing order. The "ret 24" statement at the end cleans 24 bytes off the stack, exactly like one would expect.

[edit] GNU C Compiler: GCC

We will be using 2 example C functions to demonstrate how GCC implements calling conventions:

int MyFunction1(int x, int y)
 {
        return (x * 2) + (y * 3);
 }

and

int MyFunction2(int x, int y, int z, int a, int b, int c)
 {
        return x * y * (z + 1) * (a + 2) * (b + 3) * (c + 4);
 }

GCC does not have commandline arguments to force the default calling convention to change from CDECL (for C), so they will be manually defined in the text with the directives: __cdecl, __fastcall, and __stdcall.

[edit] CDECL

The first function (MyFunction1) provides the following assembly listing:

_MyFunction1:
        pushl   %ebp
        movl    %esp, %ebp
        movl    8(%ebp), %eax
        leal    (%eax,%eax), %ecx
        movl    12(%ebp), %edx
        movl    %edx, %eax
        addl    %eax, %eax
        addl    %edx, %eax
        leal    (%eax,%ecx), %eax
        popl    %ebp
        ret

First of all, we can see the name-decoration is the same as in cl.exe. We can also see that the ret instruction doesnt have an argument, so the calling function is cleaning the stack. However, since GCC doesnt provide us with the variable names in the listing, we have to deduce which parameters are which. After the stack frame is set up, the first instruction of the function is "movl 8(%ebp), %eax". One we remember (or learn for the first time) that GAS instructions have the general form:

instruction src, dest

We realize that the value at offset +8 from ebp (the last parameter pushed on the stack) is moved into eax. The leal instruction is a little more difficult to decipher, especially if we don't have any experience with GAS instructions. The form "leal(reg1,reg2), dest" adds the values in the parenthesis together, and stores the value in dest. Translated into Intel syntax, we get the instruction:

lea ecx, [eax + eax]

Which is clearly the same as a multiplication by 2. The first value accessed must then have been the last value passed, which would seem to indicate that values are passed right-to-left here. To prove this, we will look at the next section of the listing:

movl   12(%ebp), %edx
 movl   %edx, %eax
 addl   %eax, %eax
 addl   %edx, %eax
 leal   (%eax,%ecx), %eax

the value at offset +12 from ebp is moved into edx. edx is then moved into eax. eax is then added to itselt (eax * 2), and then is added back to edx (edx + eax). remember though that eax = 2 * edx, so the result is edx * 3. This then is clearly the y parameter, which is furthest on the stack, and was therefore the first pushed. CDECL then on GCC is implemented by passing arguments on the stack in right-to-left order, same as cl.exe.

[edit] FASTCALL

.globl @MyFunction1@8
        .def    @MyFunction1@8;      .scl    2;      .type   32;     .endef
 @MyFunction1@8:
        pushl   %ebp
        movl    %esp, %ebp
        subl    $8, %esp
        movl    %ecx, -4(%ebp)
        movl    %edx, -8(%ebp)
        movl    -4(%ebp), %eax
        leal    (%eax,%eax), %ecx
        movl    -8(%ebp), %edx
        movl    %edx, %eax
        addl    %eax, %eax
        addl    %edx, %eax
        leal    (%eax,%ecx), %eax
        leave
        ret

Notice first that the same name decoration is used as in cl.exe. The astute observer will already have realized that GCC uses the same trick as cl.exe, of moving the fastcall arguments from their registers (ecx and edx again) onto a negative offset on the stack. Again, optimizations are turned off. ecx is moved into the first position (-4) and edx is moved into the second position (-8). Like the CDECL example above, the value at -4 is doubled, and the value at -8 is tripled. Therefore, -4 (ecx) is x, and -8 (edx) is y. It would seem from this listing then that values are passed left-to-right, although we will need to take a look at the larger, MyFunction2 example:

.globl @MyFunction2@24
        .def    @MyFunction2@24;     .scl    2;      .type   32;     .endef
 @MyFunction2@24:
        pushl   %ebp
        movl    %esp, %ebp
        subl    $8, %esp
        movl    %ecx, -4(%ebp)
        movl    %edx, -8(%ebp)
        movl    -4(%ebp), %eax
        imull   -8(%ebp), %eax
        movl    8(%ebp), %edx
        incl    %edx
        imull   %edx, %eax
        movl    12(%ebp), %edx
        addl    $2, %edx
        imull   %edx, %eax
        movl    16(%ebp), %edx
        addl    $3, %edx
        imull   %edx, %eax
        movl    20(%ebp), %edx
        addl    $4, %edx
        imull   %edx, %eax
        leave
        ret    $16

By following the fact that in MyFunction2, successive parameters are added to increasing constants, we can deduce the positions of each parameter. -4 is still x, and -8 is still y. +8 gets incremented by 1 (z), +12 gets increased by 2 (a). +16 gets increased by 3 (b), and +20 gets increased by 4 (c). Let's list these values then:

z = [ebp + 8]
a = [ebp + 12]
b = [ebp + 16]
c = [ebp + 20]

c is the furthest down, and therefore was the first pushed. z is the highest to the top, and was therefore the last pushed. Arguments are therefore pushed in right-to-left order, just like cl.exe.

[edit] STDCALL

Let's compare then the implementation of MyFunction1 in GCC:

.globl _MyFunction1@8
        .def    _MyFunction1@8;      .scl    2;      .type   32;     .endef
 _MyFunction1@8:
        pushl   %ebp
        movl    %esp, %ebp
        movl    8(%ebp), %eax
        leal    (%eax,%eax), %ecx
        movl    12(%ebp), %edx
        movl    %edx, %eax
        addl    %eax, %eax
        addl    %edx, %eax
        leal    (%eax,%ecx), %eax
        popl    %ebp
        ret    $8

The name decoration is the same as in cl.exe, so STDCALL functions (and CDECL and FASTCALL for that matter) can be assembled with either compiler, and linked with either linker, it seems. The stack frame is set up, then the value at [ebp + 8] is doubled. After that, the value at [ebp + 12] is tripled. Therefore, +8 is x, and +12 is y. Again, these values are pushed in right-to-left order. This function also cleans it's own stack with the "ret 8" instruction.

Looking at a bigger example:

.globl _MyFunction2@24
        .def    _MyFunction2@24;     .scl    2;      .type   32;     .endef
 _MyFunction2@24:
        pushl   %ebp
        movl    %esp, %ebp
        movl    8(%ebp), %eax
        imull   12(%ebp), %eax
        movl    16(%ebp), %edx
        incl    %edx
        imull   %edx, %eax
        movl    20(%ebp), %edx
        addl    $2, %edx
        imull   %edx, %eax
        movl    24(%ebp), %edx
        addl    $3, %edx
        imull   %edx, %eax
        movl    28(%ebp), %edx
        addl    $4, %edx
        imull   %edx, %eax
        popl    %ebp
        ret    $24

We can see here that values at +8 and +12 from ebp are still x and y, respectively. The value at +16 is incremented by 1, the value at +20 is incremented by 2, etc all the way to the value at +28. We can therefore create the following table:

x = [ebp + 8]
y = [ebp + 12]
z = [ebp + 16]
a = [ebp + 20]
b = [ebp + 24]
c = [ebp + 28]

With c being pushed first, and x being pushed last. Therefore, these parameters are also pushed in right-to-left order. This function then also cleans 24 bytes off the stack with the "ret 24" instruction.

[edit] Example: C Calling Conventions

int MyFunction(int a, int b)
 {
    return a + b;
 }

[edit] Example: Named Assembly Function

:@MyFunction@0
 push ebp
 mov ebp, esp
 ...
 pop ebp
 ret 12

[edit] Example: Unnamed Assembly Function

push ebp
 mov ebp, esp
 add eax, edx
 pop ebp
 ret

[edit] Example: Another Unnamed Assembly Function

push ebp 
 mov ebp, esp
 mov eax, [ebp + 8]
 pop ebp
 ret 16

[edit] Example: Name Mangling

move ecx, x
 push eax
 mov eax, ss:[ebp - 4]
 push eax
 mov al, ss:[ebp - 3]
 call @__Load?$Container__XXXY_?Fcii

class Container
{
  void Load(char, int, int);
}