An Introduction to Binary Analysis

Sam Thomas / xorpse
AFiniteNumberOfMonkeys ~ University of Birmingham

Introduction

Given an executable, how do we understand what it does without access to the source code?

Motivation

Why might we want to do this?
  • Find vulnerabilities in programs
  • Reverse engineer algorithms (such as license checks, etc.)

Plan for today

  • Overview of various tools and techniques
  • Look into how compiled programs are represented in x86 assembly language (using IDA Pro)
  • Reverse engineering competition

Tools

Static analysis

  • IDA Pro
  • radare2

Dynamic analysis

  • edb
  • gdb

From Executable to Assembly

What is IDA?

IDA is a state of the art disassembler:
  • Transforms a compiled program into a “human readable” representation.

Variables

#include <stdio.h>

int main(int argc, char **argv)
{
    int a = 10, b = 20;

    printf("%d\n", a + b);

    return 0;
}

Conditionals

#include <stdio.h>

int main(int argc, char **argv)
{
    int a = 10, b = 20;

    if (a <= b) {
        puts("a <= b");
    } else {
        puts("a > b");
    }

    return 0;
}

Loops

#include <stdio.h>

int main(int argc, char **argv)
{
    int i = argc * 10;

    while (i != 0) {
        puts("Looping...");
        i--;
    }

    return 0;
}

Function calls & the call stack

#include <stdio.h>

int add2(int a, int b)
{
    return a + b;
}

int main(int argc, char **argv)
{
    printf("%d + %d = %d\n", 10, 20, add2(10, 20));

    return 0;
}

A simple challenge

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
    unsigned int password = 0;

    printf("enter password: ");

    if (scanf("%u", &password) != 1) {
        puts("error: invalid password!");
        exit(EXIT_FAILURE);
    }

    if (password == 0xdeadbeef) {
        puts("success: valid password!");
    } else {
        puts("error: invalid password!");
        exit(EXIT_FAILURE);
    }

    return EXIT_SUCCESS;
}

Competition!

Recap & more!

  • Solution to ex2.
  • Modern Binary Exploitation course:
    • Slides for reverse engineering (lecture 1 & 2) (link)
    • Exercises for reverse engineering (link)