‘Shellcode’ challenges from MalwareTech

The ‘shellcode’ is relatively a small piece of code used as the payload in the exploitation of a software vulnerability. It is mostly written in machine code which is a list of carefully organized assembly instruction opcodes that can be executed once loaded into executing program. Wikipedia greatly defines about this…

Using this shellcode technique, this time we’ll see two challenges from MalwareTech. The rules are, both of two challenges should be solved using static analysis only & neither of those given executables need to run nor the debugger should be used.

In the last article (called ‘Hide & Seek’ String challenges from MalwareTech), we found the flags from string challenges & this time we’ll dig into the shellcode!

Analysis of ‘shellcode1.exe_’

• Once opened the binary in IDA & decompiled the start() function (which is main entry point of binary), we get the assembly instructions listing in the “IDA View-A” & the decompiled C-like source for this function in the “Pseudocode-A” (for easy understanding & comparison, here both views opened one after other):

Picture 1.1 — The start() entry function of shellcode1.exe_ binary.

• If we follow the C-like source from “Pseudocode-A” (in picture 1.1), it will be easier to see the general flow of this function.
• After some variable declarations, MD5 initialization the call goes to GetProcessHeap() function that returns a handle to the default heap of calling process.
• Using this retrieved heap handle called ProcessHeap, the HeapAlloc() function allocates the 0x10 (i.e., 16) bytes block of memory from heap. IDA showing the pointer to this allocated memory being stored in v4 variable.
• Then the line numbers 12 & 13 (in picture 1.1), are assigning some starting address of offset Str & also calculating the length of that Str to store on allocated heap respectively. Offset Str defines bytes as:

Picture 1.2 — Bytes at offset Str.

• Next, line number 14 (in picture 1.1), is calling the VirtualAlloc() function reserves, commits, or changes the state of a region of pages in the virtual address space of the calling process. It allocates the size of 0xD (i.e., 13) bytes of memory, using 2nd parameter. The 1st, 3rd & 4th parameters are starting address of region, type of memory allocation & memory protection for region of pages respectively.
• The memcpy() function call on the line 15 (in picture 1.1) is copying the 0xD (i.e., 13) bytes from a location defined by unk_404068. Now let’s check what bytes defined at this unknown, unk_404068:

Picture 1.3 — Total 0xD (i.e., 13) bytes defined at unk_404068.

• So, picture 1.3 of unk_404068 is showing the shellcode bytes! We can convert these bytes in IDA to assembly instructions by selecting all these bytes & pressing ‘c’ shortcut key or by selecting from menu Edit -> Code to convert. Here’re the converted instructions:

Picture 1.4 — Bytes to code converted instructions.

• Converted code instructions (in picture 1.4) clearly describes that it is first copying some value from source address pointed by ESI register to destination EDI register. Then setting some counter value in ECX register. After counter setup, the ROL (rotate left) instruction rotates the bits pointed by EDI+ECX-1 expression, by 5 positions. Then the LOOP instruction decrements ECX register by 1 & continues again from loc_40406D label above.
• But where is the source address pointed by ESI register pointing to? 🤔 That’s where the offset Str comes in. So, the bytes from offset Str are used in the shellcode loop.
• To make this looping over all Str bytes & left rotating bits easier, below Python script will be helpful:

Picture 1.5 — Python script to loop, left rotate bits of each byte in Str & getting the flag!

• Above Python code (in picture 1.5), is well documented using comments. The str_bytes_list is the bytes defined by offset Str, then two variables max_bits & rotate_bits define maximum number of bits for a value to consider & number of positions to left rotate bits respectively. The for loop is where ROL operation performed & flag characters are printed. When executed this script, we get resulting flag as:

Captured the flag!

And the flag is correct once submitted! 🙃

Analysis of ‘shellcode2.exe_’

• We’ll open the second binary in IDA but there’s one thing we should do while loading the binary is to enable the Manual Load option from IDA’s “Load a new file” window. Later it will be clear why this selection is necessary:

Picture 2.1 — IDA option to select while loading the binary file.

• Once opened binary, we’ll get at start() function like before:

Picture 2.2 — The start() entry function of shellcode2.exe_ binary.

• In “Pseudocode-A” of picture 2.2, after some variable declarations & MD5 initialization there are lot of constant values getting assigned for array called v4. The size of this v4 array is 36 defined at line number 7 & there are indeed 36 constant values assigned afterwards:

Picture 2.3 — Remaining pseudocode after 36 constant values of v4 array in start() function.

• From above pseudocode (in picture 2.3), the lines 47 to 55 are the part of our interest & analysis to get the flag.
• Just like last binary, the GetProcessHeap() function returns a handle to the default heap of calling process in ProcessHeap & it is used by the HeapAlloc() function to allocate the 0x10 (i.e., 16) bytes block of memory from heap. IDA showing the pointer to this allocated memory is stored in v5 variable. So, handle to allocated heap is pointed by v5.
• Next, there’re assignments of different things to this v5 allocated heap memory:

1. At v5[0], the LoadLibraryA() function’s address is stored. (Line 49, picture 2.3)
2. At v5[1], the GetProcAddress() function’s address is stored. (Line 50, picture 2.3)
3. At v5[2], the previously defined v4 address is stored which is array of pre-defined constant values. (Line 51, picture 2.3)
4. At v5[3], some another constant value 36 is stored. (Line 52, picture 2.3)

• The line 53 (in picture 2.3) has VirtualAlloc() function that allocates the size of 0x248 (i.e., 584) bytes of memory (using 2nd parameter) in the virtual address space of the calling process, similar to our observation in previous binary.
• By using all the setup so far, line 54 (from picture 2.3), use memcpy() to copy the subroutine sub_404040 to the v1. Then on the line 55, this v1 gets called. So, this should be the next thing to look & understand!
• Let’s check the de-compiled code of subroutine sub_404040:

Picture 2.4 — De-compiled code of subroutine sub_404040.

• If we observe this code (from picture 2.4), it’s kind of difficult to understand because there are several variables with names from v & they have suffix of some indices like v2, v3, v4, v5 & so on…
• To make this code more readable we need to rename & refactor those variables & function calls to what they really correspond with the help of Microsoft documentation. After refactoring, this code looks like:

Picture 2.5 — Subroutine sub_404040 after renaming & refactoring. Type casts are also hidden to make it clear.

• Now the code (from picture 2.5) looks much better & clear to understand. 😊
• From line numbers 28 to 35 (in picture 2.5), there are multiple strcpy() calls to copy names of DLL files, function names into variables.
• Next two lines 36 & 37 (in picture 2.5), gets the LoadLibraryA() function’s address & the GetProcAddress() function’s address from current sub_404040's parameter a1.
• The first function call used to load two DLL files called msvcrt.dll & kernel32.dll. The second function call used to load several procedure addresses from those loaded DLL file handles. Lines 38 to 44 (in picture 2.5) do this stuff.
• From lines 45 to 49 (in picture 2.5), there’s file handling operation to read the bytes. The same current executable file ‘shellcode2.exe_’ will be read. In short, this read operation will start at offset 78 in file & it will read the 38 bytes of data from this file only 1 time. We need to check the offset that comes under HEADER section of this binary. So, the reason to select Manual Load option at first is to tell IDA about loading the HEADER section also.
• The HEADER section offset starts at 0x00400000 & so by adding 78 (i.e., 0x4E) to this will go to offset 0x0040004E, that offset location have string as “This program cannot be run in DOS mode” (assigned to renamed variable as bufferStorage):

Picture 2.6 — The HEADER section at offset 0x0040004E will be stored in bufferStorage variable.

• Back to line numbers 50 & 51 within pseudocode (in picture 2.5), it basically stores constant value 36 (renamed variable as const_36) & address of pre-defined array values (renamed variable as preDefinedArray).
• Then the following lines 52 to 58 (in picture 2.5) are looping over to get each byte from bufferStorage, then those bytes are XORed with corresponding byte in preDefinedArray. This is the logic of loop & will be the resulting flag we’re looking for…
• Once again, some Python scripting can help here to operate over these bytes:

Picture 2.7 — Python script to loop over & do XOR operation for each byte of bufferStorage & preDefinedArray.

• This Python code (in picture 2.7) is also well documented using the comments. But the only thing is at first both bufferStorage have string in form of bytes & preDefinedArray have values in form of bytes. That’s because if we don’t do this, then XOR operation between string type & some value type will not be permitted due to different types of data.
• Let’s run this script & we’ll get the resulting flag:

Captured the flag!

Again, the flag is correct once submitted! 🙃

That’s it! We extracted the flags from both shellcode challenges… Here I intentionally marked some portions of resulting flags in red color because those who’re interested, can follow this process to generate resulting flags or can follow the Python scripts & try to understand.

Reverse engineering gets more interesting as we explore, understand more & more! These shellcode challenges explained things about how this technique can possibly be used. Also, the Wikipedia says,

Writing good shellcode can be as much an art as it is a science.

Again, big thanks to MalwareTech for making these challenges available. I hope you enjoyed this analysis. Please consider sharing, following & subscribing to notifications for more upcoming writeups… ✌️🚀

— Amey Chavan