Dennis Yurichev

Computers & Technology

Reverse Engineering for Beginners

Free

Read Now

Description

Contents

Reviews

There are several popular meanings of the term “reverse engineering”:

The reverse engineering of software; researching compiled programs

The scanning of 3D structures and the subsequent digital manipulation required in order to duplicate them

Recreating DBMS7 structure

This book is about the first meaning.

Topics discussed: x86/x64, ARM/ARM64, MIPS, Java/JVM.

Topics touched: Oracle RDBMS, Itanium, copy-protection dongles, LD_PRELOAD, stack overflow, ELF, win32 PE file format, x86-64, critical sections, syscalls, TLS, position-independent code (PIC), profile-guided optimization, C++ STL, OpenMP, win32 SEH.

Language

English

ISBN

Unknown

Code Patterns

The method

Some basics

A short introduction to the CPU

Numeral Systems

Converting From One Radix To Another

An Empty Function

x86

ARM

MIPS

Empty Functions in Practice

Returning Values

x86

ARM

MIPS

Hello, world!

x86

x86-64

GCC—one more thing

ARM

MIPS

Conclusion

Exercises

Function prologue and epilogue

Recursion

Stack

Why does the stack grow backwards?

What is the stack used for?

A typical stack layout

Noise in stack

Exercises

printf() with several arguments

x86

ARM

MIPS

Conclusion

By the way

scanf()

Simple example

Popular mistake

Global variables

scanf()

Exercise

Accessing passed arguments

x86

x64

ARM

MIPS

More about results returning

Attempt to use the result of a function returning void

What if we do not use the function result?

Returning a structure

Pointers

Swap input values

Returning values

GOTO operator

Dead code

Exercise

Conditional jumps

Simple example

Calculating absolute value

Ternary conditional operator

Getting minimal and maximal values

Conclusion

Exercise

switch()/case/default

Small number of cases

A lot of cases

When there are several case statements in one block

Fall-through

Exercises

Loops

Simple example

Memory blocks copying routine

Condition check

Conclusion

Exercises

More about strings

strlen()

Boundaries of strings

Replacing arithmetic instructions to other ones

Multiplication

Division

Exercise

Floating-point unit

IEEE 754

x86

ARM, MIPS, x86/x64 SIMD

C/C++

Simple example

Passing floating point numbers via arguments

Comparison example

Some constants

Copying

Stack, calculators and reverse Polish notation

80 bits?

x64

Exercises

Arrays

Simple example

Buffer overflow

Buffer overflow protection methods

One more word about arrays

Array of pointers to strings

Multidimensional arrays

Pack of strings as a two-dimensional array

Conclusion

By the way

Exercises

Manipulating specific bit(s)

Specific bit checking

Setting and clearing specific bits

Shifts

Setting and clearing specific bits: FPU example

Counting bits set to 1

Conclusion

Exercises

Linear congruential generator

x86

x64

32-bit ARM

MIPS

Thread-safe version of the example

Structures

MSVC: SYSTEMTIME example

Let's allocate space for a structure using malloc()

UNIX: struct tm

Fields packing in structure

Nested structures

Bit fields in a structure

Exercises

Unions

Pseudo-random number generator example

Calculating machine epsilon

FSCALE replacement

Fast square root calculation

Pointers to functions

MSVC

GCC

Danger of pointers to functions

64-bit values in 32-bit environment

Returning of 64-bit value

Arguments passing, addition, subtraction

Multiplication, division

Shifting right

Converting 32-bit value into 64-bit one

SIMD

Vectorization

SIMD strlen() implementation

64 bits

x86-64

ARM

Float point numbers

64-bit architecture criticism

Working with floating point numbers using SIMD

Simple example

Passing floating point number via arguments

Comparison example

Calculating machine epsilon: x64 and SIMD

Pseudo-random number generator example revisited

Summary

ARM-specific details

Number sign (#) before number

Addressing modes

Loading a constant into a register

Relocs in ARM64

MIPS-specific details

Loading a 32-bit constant into register

Further reading about MIPS

Important fundamentals

Integral datatypes

Bit

Nibble AKA nybble

Byte

Wide char

Signed integer vs unsigned

Word

Address register

Numbers

Signed number representations

Using IMUL over MUL

Couple of additions about two's complement form

Integer overflow

AND

Checking if a value is on 2n boundary

KOI-8R Cyrillic encoding

AND and OR as subtraction and addition

ZX Spectrum ROM text strings

XOR (exclusive OR)

Everyday speech

Encryption

RAID4

XOR swap algorithm

XOR linked list

Zobrist hashing / tabulation hashing

By the way

AND/OR/XOR as MOV

Population count

Endianness

Big-endian

Little-endian

Example

Bi-endian

Converting data

Memory

CPU

Branch predictors

Data dependencies

Hash functions

How do one-way functions work?

Slightly more advanced examples

Double negation

strstr() example

Temperature converting

Integer values

Floating-point values

Fibonacci numbers

Example #1

Example #2

Summary

CRC32 calculation example

Network address calculation example

calc_network_address()

form_IP()

print_as_IP()

form_netmask() and set_bit()

Summary

Loops: several iterators

Three iterators

Two iterators

Intel C++ 2011 case

Duff's device

Should one use unrolled loops?

Division using multiplication

x86

How it works

ARM

MIPS

Exercise

String to number conversion (atoi())

Simple example

A slightly advanced example

Exercise

Inline functions

Strings and memory functions

C99 restrict

Branchless abs() function

Optimizing GCC 4.9.1 x64

Optimizing GCC 4.9 ARM64

Variadic functions

Computing arithmetic mean

vprintf() function case

Pin case

Format string exploit

Strings trimming

x64: Optimizing MSVC 2013

x64: Non-optimizing GCC 4.9.1

x64: Optimizing GCC 4.9.1

ARM64: Non-optimizing GCC (Linaro) 4.9

ARM64: Optimizing GCC (Linaro) 4.9

ARM: Optimizing Keil 6/2013 (ARM mode)

ARM: Optimizing Keil 6/2013 (Thumb mode)

MIPS

toupper() function

x64

ARM

Using bit operations

Summary

Obfuscation

Text strings

Executable code

Virtual machine / pseudo-code

Other things to mention

Exercise

C++

Classes

ostream

References

STL

Memory

Negative array indices

Addressing string from the end

Addressing some kind of block from the end

Arrays started at 1

Packing 12-bit values into array

Introduction

Data structure

The algorithm

The C/C++ code

How it works

Optimizing GCC 4.8.2 for x86-64

Optimizing Keil 5.05 (Thumb mode)

Optimizing Keil 5.05 (ARM mode)

(32-bit ARM) Comparison of code density in Thumb and ARM modes

Optimizing GCC 4.9.3 for ARM64

Optimizing GCC 4.4.5 for MIPS

Difference from the real FAT12

Exercise

Summary

Conclusion

More about pointers

Working with addresses instead of pointers

Passing values as pointers; tagged unions

Pointers abuse in Windows kernel

Null pointers

Array as function argument

Pointer to function

Pointer as object identificator

Loop optimizations

Weird loop optimization

Another loop optimization

More about structures

Sometimes a C structure can be used instead of array

Unsized array in C structure

Version of C structure

High-score file in "Block out" game and primitive serialization

memmove() and memcpy()

Anti-debugging trick

setjmp/longjmp

Other weird stack hacks

Accessing arguments/local variables of caller

Returning string

OpenMP

MSVC

GCC

Another heisenbug

Windows 16-bit

Example#1

Example #2

Example #3

Example #4

Example #5

Example #6

Java

Introduction

Returning a value

Simple calculating functions

JVM memory model

Simple function calling

Calling beep()

Linear congruential PRNG

Conditional jumps

Passing arguments

Bitfields

Loops

switch()

Arrays

Strings

Exceptions

Classes

Simple patching

Summary

Finding important/interesting stuff in the code

Identification of executable files

Microsoft Visual C++

GCC

Intel Fortran

Watcom, OpenWatcom

Borland

Other known DLLs

Communication with outer world (function level)

Communication with the outer world (win32)

Often used functions in the Windows API

Extending trial period

Removing nag dialog box

tracer: Intercepting all functions in specific module

Strings

Text strings

Finding strings in binary

Error/debug messages

Suspicious magic strings

Calls to assert()

Constants

Magic numbers

Specific constants

Searching for constants

Finding the right instructions

Suspicious code patterns

XOR instructions

Hand-written assembly code

Using magic numbers while tracing

Loops

Some binary file patterns

Memory "snapshots" comparing

ISA detection

Incorrectly disassembled code

Correctly disassembled code

Text strings right in the middle of compressed data

Other things

General idea

Order of functions in binary code

Tiny functions

C++

OS-specific

Arguments passing methods (calling conventions)

cdecl

stdcall

fastcall

thiscall

x86-64

Return values of float and double type

Modifying arguments

Taking a pointer to function argument

Thread Local Storage

Linear congruential generator revisited

System calls (syscall-s)

Linux

Windows

Linux

Position-independent code

LD_PRELOAD hack in Linux

Windows NT

CRT (win32)

Win32 PE

Windows SEH

Windows NT: Critical section

Tools

Binary analysis

Disassemblers

Decompilers

Patch comparison/diffing

Live analysis

Debuggers

Library calls tracing

System calls tracing

Network sniffing

Sysinternals

Valgrind

Emulators

Other tools

Calculators

Do You Think Something Is Missing Here?

Case studies

Task manager practical joke (Windows Vista)

Using LEA to load values

Color Lines game practical joke

Minesweeper (Windows XP)

Finding grid automatically

Exercises

Hacking Windows clock

Dongles

Example #1: MacOS Classic and PowerPC

Example #2: SCO OpenServer

Example #3: MS-DOS

"QR9": Rubik's cube inspired amateur crypto-algorithm

Encrypted database case #1

Base64 and entropy

Is data compressed?

Is data encrypted?

CryptoPP

Cipher Feedback mode

Initializing Vector

Structure of the buffer

Noise at the end

Conclusion

Post Scriptum: brute-forcing IV

Overclocking Cointerra Bitcoin miner

Breaking simple executable cryptor

Other ideas to consider

SAP

About SAP client network traffic compression

SAP 6.0 password checking functions

Oracle RDBMS

V$VERSION table in the Oracle RDBMS

X$KSMLRU table in Oracle RDBMS

V$TIMER table in Oracle RDBMS

Handwritten assembly code

EICAR test file

Demos

10 PRINT CHR$(205.5+RND(1)); : GOTO 10

Mandelbrot set

Other examples

Examples of reversing proprietary file formats

Primitive XOR-encryption

Simplest ever XOR encryption

Norton Guide: simplest possible 1-byte XOR encryption

Simplest possible 4-byte XOR encryption

Simple encryption using XOR mask

Simple encryption using XOR mask, case II

Information entropy

Analyzing entropy in Mathematica

Conclusion

Tools

A word about primitive encryption like XORing

More about entropy of executable code

PRNG

More examples

Entropy of various files

Making lower level of entropy

Millenium game save file

fortune program indexing file

Hacking

The files

Oracle RDBMS: .SYM-files

Oracle RDBMS: .MSB-files

Summary

Exercises