I was messing around with optimizations and IDA Pro today in an attempt to get some dead code compiled into a program intentionally. I thought this would be a 10 second task, and in some ways it was. However, it took a while because I ended up finding something odd out about Visual Studio 2008 (untested on older versions).

Let's take the following code:

#include "stdafx.h"
#include "windows.h"
#include "stdio.h"

static void foo()
{
printf("Inside function foo. This should never be called\n");
return;
}

int _tmain(int argc, _TCHAR* argv[])
{
printf("This is main. It doesn't do squadoosh!\n");
return 0;
}

Under unix world with gcc I can simply compile this program using the flag -O0 to disable optimizations. I then can execute objdump -d to determine that indeed my code is present in the binary.

*SNIP*

08048374 foo:
8048374: 55 push %ebp
8048375: 89 e5 mov %esp,%ebp
8048377: 83 ec 08 sub $0x8,%esp
804837a: c7 04 24 98 84 04 08 movl $0x8048498,(%esp)
8048381: e8 2a ff ff ff call 80482b0 printf@plt
8048386: c9 leave
8048387: c3 ret

08048388 main:
8048388: 55 push %ebp
8048389: 89 e5 mov %esp,%ebp
804838b: 83 ec 08 sub $0x8,%esp
804838e: 83 e4 f0 and $0xfffffff0,%esp
8048391: b8 00 00 00 00 mov $0x0,%eax
8048396: 83 c0 0f add $0xf,%eax
8048399: 83 c0 0f add $0xf,%eax
804839c: c1 e8 04 shr $0x4,%eax
804839f: c1 e0 04 shl $0x4,%eax
80483a2: 29 c4 sub %eax,%esp
80483a4: c7 04 24 cc 84 04 08 movl $0x80484cc,(%esp)
80483ab: e8 00 ff ff ff call 80482b0 printf@plt
80483b0: b8 00 00 00 00 mov $0x0,%eax
80483b5: c9 leave
80483b6: c3 ret
80483b7: 90 nop

However in Visual Studio it's not quite as cut and dry. I disabled all optimizations and spent what amounted to nearly two hours trying to disable whatever it was that was keeping the compiler from adding my deadcode to the binary. I ended up chatting with a coworker of mine and he suggested I remove the "static" modifier from the foo function. Sure enough, as soon as I did this I was able to compile the dead code into the function. So after much face banging into desk I figured out that Visual Studio will always optimize out static functions regardless of the optimization settings configured in the project properties.

This makes no sense since the word static is intended to only allow the local file access to the function, it should have no impact on the optimizations of the binary as a whole. If anyone with more programming expertise than I can explain a valid reason for this optimization, please help me out.

I absolutely loved this post by a colleague at "the anti-shazzzam" blog. She has some excellent incites and wrote a fantastic piece on the unwritten laws of the security researcher. These laws can be expanded to encompass just about any research related field, but has specific twists and caveats to the nerd side of security research. I highly recommend that anyone doing security research read and digest these unwritten rules on how to make your (and everyone's) life quite a bit easier.