The fuzzing of satellite firmware will be the focus of this article. In a previous article, I described how AFL++ can be added to a custom build of QEMU. Now, I will explain how that setup can be used to target the firmware of a satellite.

Like the previous article, this text is based on my master’s thesis. However, some segments were changed and improved. I learned a lot along the way, and some approaches from back then are optimized here.

Finding a fuzzing target

The first step when fuzzing a firmware image is to identify an interesting code segment that handles input data. As explained in my last article, the goal of fuzzing is to trigger a crash of the target application by sending automatically generated input to it. A good starting point for fuzzing are functions that directly handle incoming data.

If you have access to the source code of the firmware or at least can use the ELF headers, you can search for function names that imply a connection to input handling. For example, in the OPS-SAT, there are functions like can_rx_task_gmv, csp_i2c_rx or CSP_ServerCycle. Another interesting function is the TCTA_Cycle, which works on input data without a direct hardware interaction. The function was selected because hardware interaction would unnecessarily increase the complexity. TCTA_Cycle handles the processing of so-called telecommands. Telecommands are sent to the satellite from the ground and can be used to trigger various actions, like file changes on the satellite’s filesystem or sending telemetry data.

The TCTA_Cycle is implemented as a FreeRTOS task and consists of 4 steps:

void TCTA_Cycle(void)
{
  TCMA_CleanTCBuffers();
  TCMA_ReadCommand();
  TCMA_VerifyCommand();
  TCMA_ExecuteCommand();
  return;
}

The first function just overwrites the telecommand buffer with zeros. The second function reads a telecommand from an external interface, for example, from the CAN bus. The function also calculates a cyclic redundancy check (CRC) from any received telecommand. Such checks are usually a hurdle for fuzzing, as the generated input will likely not pass the check. Therefore, the fuzzzing-input should be inserted after the check.

The third function checks if the received telecommand is supported. A telecommand has a service area, an operation, and a few other meta-information. The OPS-SAT only supports a defined combination of them, and TCMA_VerifyCommand checks if the telecommand holds one of those combinations.

In the last function, the telecommand is executed if it is valid. TCMA_VerifyCommand sets a flag in the telecommand buffer to indicate if the command is valid. Only commands with that flag are executed.

Because I did not want to deal with the CRC, but still wanted to only execute valid telecommands, I decided to insert the fuzzing test cases directly into the telecommand buffer. The insertion was done just before TCMA_VerifyCommand was called. This, of course, only works in a laboratory setting and may circumvent security measures that might exist. However, there needs to be a consideration between realism and workload for a researcher. If any vulnerabilities are found by such a simplified approach, more research is needed to decide if they are relevant in the real world.

The telecommand buffer is 1000 bytes long and can hold 20 telecommands. Every telecommand can be up to 0x14c bytes long. By reverse engineering the firmware, I noticed that the buffer starts at address 0xd0ec2f10. Of course, this will be different for any other fuzzing target. If you want to use fuzzing on a firmware image, you need to have a deep understanding of how the relevant parts work. Without the source code, a lot of reverse engineering will be necessary.

Inserting test cases

To insert a test case, we will use a QEMU helper function. Inside the translate.c file is the avr32_tr_tb_start function, which is executed at the start of every translation block. We will patch this instruction to call our fuzzing integration.

First, we need to define a QEMU helper in the fuzzerbinding.h file:

DEF_HELPER_1(fuzzing_mng, noreturn, env)

Now we can call the helper by editing avr32_tr_tb_start:

static void avr32_tr_tb_start(DisasContextBase *db, CPUState *cs)
{
    gen_helper_fuzzing_mng(cpu_env);
}

The implementation of our helper will be placed inside the fuzzerbinding.c file:

void helper_fuzzing_mng(CPUAVR32AState *env){
    CPUState *cs = env_cpu(env);
    uint32_t pc_address = cs->env_ptr->r[AVR32A_PC_REG];
    
    switch(pc_address){
        case 0xd00a0652:{
            insert_fuzzing_data(env, 0x14c, 0xd0ec2f10);            
            break;
        }
        default:{
            return;
        }
    }
}

Now, every time TCMA_VerifyCommand is called, AFL generates a new test case, and QEMU inserts it into the memory location of the teleommand buffer.

Ending test cases

Every inserted test case needs to end at some point. Either in a crash or in a regular end.

Finding a point at which QEMU should report the regular end of the execution to AFL is fairly simple: the return instruction of the TCTA_Cycle. We can expand our switch statement from above with a new case:

//The address of the return instruction from the text segment of the firmware file
case 0xd00a065a:{
    test case_active= false;
    __afl_end_test case(0x0);
    break;
}

The difficult part is finding crash points. When fuzzing a regular application on a computer, the operating system notices when the application has crashed. AFL then receives the corresponding signal and creates a crash report. But we are not executing a regular application, and there is no operating system that tells us that the emulated firmware crashed. So, how can we tell AFL or QEMU when a crash occurs?

The firmware has multiple functions that are called when an error occurs. For example, there is vApplicationStackOverflowHookthat is called if a stack overflow is detected.

There are also the _exit and the cpu_reset functions that are not called during a normal execution but only when a critical error occurs.

If one of these functions is executed, we can assume that a test case provoked some behavior that was not expected and the firmware fell to an undefined state. We can again expand the switch statement and add the following cases:

//Addresses of 'error points' inside the firmware
case 0xd00c307e:
case 0xd00c460e:
case 0xd00c4682:
{
    report_crash();
    //Do reset of emulated cpu
    //...
    break;
}

If the firmware is at any of the relevant addresses, we report a crash to AFL. As the firmware is in an undefined state, we also need to reset the emulated CPU to restart the firmware.

Illegal Instructions

It is possible that a test case triggers the execution of non-existent code. For example, if a classic buffer overflow occurs and a return pointer is overwritten with values that do not represent a real code address. If such a pointer is loaded into the program counter register, execution will continue at that address. However, as there is no valid instruction, QEMU will exit with an ìllegeal instruction message.

Because this message could indicate a crash, we should also define such an event as a ‘crash point’. To do so, we can modify the avr32_tr_translate_insn function in translate.c:

insn = decode_insn_load(ctx);
    if (!decode_insn(ctx, insn)) {
        error_report("[AVR32-TCG] avr32_tr_translate_insn, illegal instr, pc: 0x%04x\n", ctx->base.pc_next);
        report_crash();
        
        //As this error could also appear when a non-implemented instruction is loaded,
        //we still want to stop QEMU and restart from the outside.
        gen_helper_raise_illegal_instruction(cpu_env);
    }

Now the fuzzing integration is complete and adapted for a specific fuzzing target.

Seeds

AFL requires a seed to generate its test cases. The seed should consist of a valid input for the application that does not trigger a crash. AFL then changes the seed with a few algorithms and tries to generate an input that triggers a crash.

For the OPS-SAT, I started with some random bytes. As one would expect, this did not result in the execution of any telecommand, because no supported operation was generated by AFL. Therefore, I created a few seeds that had valid combinations for the operation and service fields. This led to various test cases with valid telecommands in a short time.

Starting AFL

Fuzzing with AFL will result in a huge number of small files that represent the current fuzzing state. AFL will create files with future test cases and other information. Therefore, the fuzzing should take place in a separate folder.

To start AFL with QEMU and the target firmware, we can use the following command:

AFL_SKIP_CPUFREQ=1 afl-fuzz -t 5000 -i input -o output ./qemu-system-avr32 -machine nanomind-a3200 -bios [path to firmware]

The AFL_SKIP_CPUFREQ parameter is needed because some CPU optimizations may not work on every system. You can try to start AFL without it.

In some cases, QEMU crashed during the fuzzing, for example, because an unimplemented instruction was reached or because some unexpected error occurred. In other cases, the execution speed of AFL went down to less than one execution per second. Because fuzzing is a time-consuming process, I wanted AFL to automatically restart if QEMU crashed or if I manually killed a slow instance.

To do so, I wrapped the AFL start command with a while loop:

while true;
  do AFL_SKIP_CPUFREQ=1 afl-fuzz -t 5000 -i input -o output ./qemu-system-avr32 -machine nanomind-a3200 -bios [path to firmware];
  sleep 1;
done

And now, we are automatically fuzzing a satellite:

AFL output in terminal

As you can see on the left side under stage progress, the performance is quite good. More than 180 executions per second are a more than ok throughput.

Hurdles and issues

Fuzzing satellite firmware comes with a few hurdles and lessons learned. In my thesis, I have a longer discussion about them. Here, I will briefly name a few.

First, the low stability

Because the OPS-SAT firmware needs regular timer interrupts to function, the execution flow is often stopped to handle the timer interrupt. Hence, the execution path for one test case is likely not the same for two separate runs.

Second, the static state

We do not start over the firmware after a test case ends. This is because the firmware needs a few seconds to start up. We would fall to a very low execution speed if the firmware would restart after every test case.

Third, later effects

By fuzzing the TCTA_Cycle, I actually found an insecure memory operation in the firmware. But this operation only resulted in a noticeable effect after the TCTA_Cycle ended. By restarting the firmware after the TCTA_Cycle ended, I would have never noticed this issue.

Fourth, unknown side effects

Some of the AFL instances I had running fell to a really low execution speed after some time. For now, I have not identified the exact reason for this. Maybe some telecommands trigger a timed operation or hardware interaction. Maybe there are other issues in the firmware that do not trigger a crash but provoke other undefined behaviors. When fuzzing satellite firmware, you should keep in mind that such issues can happen and be prepared to work around them.

Analyzing a crash

If you are lucky, AFL will trigger a crash at some point. AFL will then store the exact test case that triggered the crash and show a notification in the terminal. You then have to manually analyze why the test case triggered a crash.

In a future article, I will show you how this was done for one vulnerability in the OPS-SAT.