x86 Assembly Language and Shellcoding on Linux

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification.

Student ID: PA-7449


The goal of this assignment is to create a custom encoder/decoder with the execve payload.


For the encoding I chose the ROT-13 cipher, which shifts each byte by 13. To implement the encoding part of the exercise I wrote a small Python script. The encoded shellcode then needs to be inserted into the nasm file and compiled to work.

Encoder - encode.py

The encoder is a really simple script, it has a few hardcoded parameters, which needs to be changed directly in the code. In order to change the shellcode, the value of the byte shift or the badchars, you will need to edit the script.

The script is really simple. The main function prints out the input shellcode in hex format and then the encoded shellcode in hex and NASM friendly hex format.

The encoding is done in an encode function which takes the shellcode as an input parameter. The function breaks down the shellcode into a bytearray and iterates through each character and shifts it by 13. There is a small bad character detection function as well, which alerts the user if a bad character is encountered.

encode.py - source


The decoder is simple as well. First we use a JMP-CALL-POP to place the shellcode address into the ESI register. After this we load the length of the shellcode into the cl register to act as a counter.

The decode loop is simple. We substract 13 byte from the value stored at the ESI address, then we increase ESI and loop until the shellcode ends. After the loop finishes, we jump to the shellcode and execute it.

The only really - spicy - solution in the shellcode is the fck_null junk instruction. Without it the shellcode introduces a null byte during compilation, so I broke it up with a simple xor eax,eax instruction.

source and binaries



Final Thoughts

Although this kind of encoding was fun as a mental exercise, I’m still 95% sure that I will still rely on Metasploit encoders in the future instead. 仕方がない is love, 仕方がない is life.

Again, all the code is on my Github and if you want to be informed about new posts, just follow me on Twitter at @fuzboxz.