Reverse Engineering for Beginners
Free

Reverse Engineering for Beginners

By Dennis Yurichev
Free
Book Description

There are several popular meanings of the term “reverse engineering”:

The reverse engineering of software; researching compiled programs
The scanning of 3D structures and the subsequent digital manipulation required in order to duplicate them
Recreating DBMS7 structure

This book is about the first meaning.

Topics discussed: x86/x64, ARM/ARM64, MIPS, Java/JVM.

Topics touched: Oracle RDBMS, Itanium, copy-protection dongles, LD_PRELOAD, stack overflow, ELF, win32 PE file format, x86-64, critical sections, syscalls, TLS, position-independent code (PIC), profile-guided optimization, C++ STL, OpenMP, win32 SEH.

Table of Contents
  • Code Patterns
    • The method
    • Some basics
      • A short introduction to the CPU
      • Numeral Systems
      • Converting From One Radix To Another
    • An Empty Function
      • x86
      • ARM
      • MIPS
      • Empty Functions in Practice
    • Returning Values
      • x86
      • ARM
      • MIPS
    • Hello, world!
      • x86
      • x86-64
      • GCC—one more thing
      • ARM
      • MIPS
      • Conclusion
      • Exercises
    • Function prologue and epilogue
      • Recursion
    • Stack
      • Why does the stack grow backwards?
      • What is the stack used for?
      • A typical stack layout
      • Noise in stack
      • Exercises
    • printf() with several arguments
      • x86
      • ARM
      • MIPS
      • Conclusion
      • By the way
    • scanf()
      • Simple example
      • Popular mistake
      • Global variables
      • scanf()
      • Exercise
    • Accessing passed arguments
      • x86
      • x64
      • ARM
      • MIPS
    • More about results returning
      • Attempt to use the result of a function returning void
      • What if we do not use the function result?
      • Returning a structure
    • Pointers
      • Swap input values
      • Returning values
    • GOTO operator
      • Dead code
      • Exercise
    • Conditional jumps
      • Simple example
      • Calculating absolute value
      • Ternary conditional operator
      • Getting minimal and maximal values
      • Conclusion
      • Exercise
    • switch()/case/default
      • Small number of cases
      • A lot of cases
      • When there are several case statements in one block
      • Fall-through
      • Exercises
    • Loops
      • Simple example
      • Memory blocks copying routine
      • Condition check
      • Conclusion
      • Exercises
    • More about strings
      • strlen()
      • Boundaries of strings
    • Replacing arithmetic instructions to other ones
      • Multiplication
      • Division
      • Exercise
    • Floating-point unit
      • IEEE 754
      • x86
      • ARM, MIPS, x86/x64 SIMD
      • C/C++
      • Simple example
      • Passing floating point numbers via arguments
      • Comparison example
      • Some constants
      • Copying
      • Stack, calculators and reverse Polish notation
      • 80 bits?
      • x64
      • Exercises
    • Arrays
      • Simple example
      • Buffer overflow
      • Buffer overflow protection methods
      • One more word about arrays
      • Array of pointers to strings
      • Multidimensional arrays
      • Pack of strings as a two-dimensional array
      • Conclusion
    • By the way
      • Exercises
    • Manipulating specific bit(s)
      • Specific bit checking
      • Setting and clearing specific bits
      • Shifts
      • Setting and clearing specific bits: FPU example
      • Counting bits set to 1
      • Conclusion
      • Exercises
    • Linear congruential generator
      • x86
      • x64
      • 32-bit ARM
      • MIPS
      • Thread-safe version of the example
    • Structures
      • MSVC: SYSTEMTIME example
      • Let's allocate space for a structure using malloc()
      • UNIX: struct tm
      • Fields packing in structure
      • Nested structures
      • Bit fields in a structure
      • Exercises
    • Unions
      • Pseudo-random number generator example
      • Calculating machine epsilon
    • FSCALE replacement
      • Fast square root calculation
    • Pointers to functions
      • MSVC
      • GCC
      • Danger of pointers to functions
    • 64-bit values in 32-bit environment
      • Returning of 64-bit value
      • Arguments passing, addition, subtraction
      • Multiplication, division
      • Shifting right
      • Converting 32-bit value into 64-bit one
    • SIMD
      • Vectorization
      • SIMD strlen() implementation
    • 64 bits
      • x86-64
      • ARM
      • Float point numbers
      • 64-bit architecture criticism
    • Working with floating point numbers using SIMD
      • Simple example
      • Passing floating point number via arguments
      • Comparison example
      • Calculating machine epsilon: x64 and SIMD
      • Pseudo-random number generator example revisited
      • Summary
    • ARM-specific details
      • Number sign (#) before number
      • Addressing modes
      • Loading a constant into a register
      • Relocs in ARM64
    • MIPS-specific details
      • Loading a 32-bit constant into register
      • Further reading about MIPS
  • Important fundamentals
    • Integral datatypes
      • Bit
      • Nibble AKA nybble
      • Byte
      • Wide char
      • Signed integer vs unsigned
      • Word
      • Address register
      • Numbers
    • Signed number representations
      • Using IMUL over MUL
      • Couple of additions about two's complement form
    • Integer overflow
    • AND
      • Checking if a value is on 2n boundary
      • KOI-8R Cyrillic encoding
    • AND and OR as subtraction and addition
      • ZX Spectrum ROM text strings
    • XOR (exclusive OR)
      • Everyday speech
      • Encryption
      • RAID4
      • XOR swap algorithm
      • XOR linked list
      • Zobrist hashing / tabulation hashing
      • By the way
      • AND/OR/XOR as MOV
    • Population count
    • Endianness
      • Big-endian
      • Little-endian
      • Example
      • Bi-endian
      • Converting data
    • Memory
    • CPU
      • Branch predictors
      • Data dependencies
    • Hash functions
      • How do one-way functions work?
  • Slightly more advanced examples
    • Double negation
    • strstr() example
    • Temperature converting
      • Integer values
      • Floating-point values
    • Fibonacci numbers
      • Example #1
      • Example #2
      • Summary
    • CRC32 calculation example
    • Network address calculation example
      • calc_network_address()
      • form_IP()
      • print_as_IP()
      • form_netmask() and set_bit()
      • Summary
    • Loops: several iterators
      • Three iterators
      • Two iterators
      • Intel C++ 2011 case
    • Duff's device
      • Should one use unrolled loops?
    • Division using multiplication
      • x86
      • How it works
      • ARM
      • MIPS
      • Exercise
    • String to number conversion (atoi())
      • Simple example
      • A slightly advanced example
      • Exercise
    • Inline functions
      • Strings and memory functions
    • C99 restrict
    • Branchless abs() function
      • Optimizing GCC 4.9.1 x64
      • Optimizing GCC 4.9 ARM64
    • Variadic functions
      • Computing arithmetic mean
      • vprintf() function case
      • Pin case
      • Format string exploit
    • Strings trimming
      • x64: Optimizing MSVC 2013
      • x64: Non-optimizing GCC 4.9.1
      • x64: Optimizing GCC 4.9.1
      • ARM64: Non-optimizing GCC (Linaro) 4.9
      • ARM64: Optimizing GCC (Linaro) 4.9
      • ARM: Optimizing Keil 6/2013 (ARM mode)
      • ARM: Optimizing Keil 6/2013 (Thumb mode)
      • MIPS
    • toupper() function
      • x64
      • ARM
      • Using bit operations
      • Summary
    • Obfuscation
      • Text strings
      • Executable code
      • Virtual machine / pseudo-code
      • Other things to mention
      • Exercise
    • C++
      • Classes
      • ostream
      • References
      • STL
      • Memory
    • Negative array indices
      • Addressing string from the end
      • Addressing some kind of block from the end
      • Arrays started at 1
    • Packing 12-bit values into array
      • Introduction
      • Data structure
      • The algorithm
      • The C/C++ code
      • How it works
      • Optimizing GCC 4.8.2 for x86-64
      • Optimizing Keil 5.05 (Thumb mode)
      • Optimizing Keil 5.05 (ARM mode)
      • (32-bit ARM) Comparison of code density in Thumb and ARM modes
      • Optimizing GCC 4.9.3 for ARM64
      • Optimizing GCC 4.4.5 for MIPS
      • Difference from the real FAT12
      • Exercise
      • Summary
      • Conclusion
    • More about pointers
      • Working with addresses instead of pointers
      • Passing values as pointers; tagged unions
      • Pointers abuse in Windows kernel
      • Null pointers
      • Array as function argument
      • Pointer to function
      • Pointer as object identificator
    • Loop optimizations
      • Weird loop optimization
      • Another loop optimization
    • More about structures
      • Sometimes a C structure can be used instead of array
      • Unsized array in C structure
      • Version of C structure
      • High-score file in "Block out" game and primitive serialization
    • memmove() and memcpy()
      • Anti-debugging trick
    • setjmp/longjmp
    • Other weird stack hacks
      • Accessing arguments/local variables of caller
      • Returning string
    • OpenMP
      • MSVC
      • GCC
    • Another heisenbug
    • Windows 16-bit
      • Example#1
      • Example #2
      • Example #3
      • Example #4
      • Example #5
      • Example #6
  • Java
    • Java
      • Introduction
      • Returning a value
      • Simple calculating functions
      • JVM memory model
      • Simple function calling
      • Calling beep()
      • Linear congruential PRNG
      • Conditional jumps
      • Passing arguments
      • Bitfields
      • Loops
      • switch()
      • Arrays
      • Strings
      • Exceptions
      • Classes
      • Simple patching
      • Summary
  • Finding important/interesting stuff in the code
    • Identification of executable files
      • Microsoft Visual C++
      • GCC
      • Intel Fortran
      • Watcom, OpenWatcom
      • Borland
      • Other known DLLs
    • Communication with outer world (function level)
    • Communication with the outer world (win32)
      • Often used functions in the Windows API
      • Extending trial period
      • Removing nag dialog box
      • tracer: Intercepting all functions in specific module
    • Strings
      • Text strings
      • Finding strings in binary
      • Error/debug messages
      • Suspicious magic strings
    • Calls to assert()
    • Constants
      • Magic numbers
      • Specific constants
      • Searching for constants
    • Finding the right instructions
    • Suspicious code patterns
      • XOR instructions
      • Hand-written assembly code
    • Using magic numbers while tracing
    • Loops
      • Some binary file patterns
      • Memory "snapshots" comparing
    • ISA detection
      • Incorrectly disassembled code
      • Correctly disassembled code
    • Text strings right in the middle of compressed data
    • Other things
      • General idea
      • Order of functions in binary code
      • Tiny functions
      • C++
  • OS-specific
    • Arguments passing methods (calling conventions)
      • cdecl
      • stdcall
      • fastcall
      • thiscall
      • x86-64
      • Return values of float and double type
      • Modifying arguments
      • Taking a pointer to function argument
    • Thread Local Storage
      • Linear congruential generator revisited
    • System calls (syscall-s)
      • Linux
      • Windows
    • Linux
      • Position-independent code
      • LD_PRELOAD hack in Linux
    • Windows NT
      • CRT (win32)
      • Win32 PE
      • Windows SEH
      • Windows NT: Critical section
  • Tools
    • Binary analysis
      • Disassemblers
      • Decompilers
      • Patch comparison/diffing
    • Live analysis
      • Debuggers
      • Library calls tracing
      • System calls tracing
      • Network sniffing
      • Sysinternals
      • Valgrind
      • Emulators
    • Other tools
      • Calculators
    • Do You Think Something Is Missing Here?
  • Case studies
    • Task manager practical joke (Windows Vista)
      • Using LEA to load values
    • Color Lines game practical joke
    • Minesweeper (Windows XP)
      • Finding grid automatically
      • Exercises
    • Hacking Windows clock
    • Dongles
      • Example #1: MacOS Classic and PowerPC
      • Example #2: SCO OpenServer
      • Example #3: MS-DOS
    • "QR9": Rubik's cube inspired amateur crypto-algorithm
    • Encrypted database case #1
      • Base64 and entropy
      • Is data compressed?
      • Is data encrypted?
      • CryptoPP
      • Cipher Feedback mode
      • Initializing Vector
      • Structure of the buffer
      • Noise at the end
      • Conclusion
      • Post Scriptum: brute-forcing IV
    • Overclocking Cointerra Bitcoin miner
    • Breaking simple executable cryptor
      • Other ideas to consider
    • SAP
      • About SAP client network traffic compression
      • SAP 6.0 password checking functions
    • Oracle RDBMS
      • V$VERSION table in the Oracle RDBMS
      • X$KSMLRU table in Oracle RDBMS
      • V$TIMER table in Oracle RDBMS
    • Handwritten assembly code
      • EICAR test file
    • Demos
      • 10 PRINT CHR$(205.5+RND(1)); : GOTO 10
      • Mandelbrot set
    • Other examples
  • Examples of reversing proprietary file formats
    • Primitive XOR-encryption
      • Simplest ever XOR encryption
      • Norton Guide: simplest possible 1-byte XOR encryption
      • Simplest possible 4-byte XOR encryption
      • Simple encryption using XOR mask
      • Simple encryption using XOR mask, case II
    • Information entropy
      • Analyzing entropy in Mathematica
      • Conclusion
      • Tools
      • A word about primitive encryption like XORing
      • More about entropy of executable code
      • PRNG
      • More examples
      • Entropy of various files
      • Making lower level of entropy
    • Millenium game save file
    • fortune program indexing file
      • Hacking
      • The files
    • Oracle RDBMS: .SYM-files
    • Oracle RDBMS: .MSB-files
      • Summary
    • Exercises
    • Further reading
  • Dynamic binary instrumentation
    • Using PIN DBI for XOR interception
    • Cracking Minesweeper with PIN
      • Intercepting all rand() calls
      • Replacing rand() calls with our function
      • Peeking into placement of mines
      • Exercise
    • Why ``instrumentation''?
  • Other things
    • Executable files patching
      • Text strings
      • x86 code
    • Function arguments number statistics
    • Compiler intrinsic
    • Compiler's anomalies
      • Oracle RDBMS 11.2 and Intel C++ 10.1
      • MSVC 6.0
      • Summary
    • Itanium
    • 8086 memory model
    • Basic blocks reordering
      • Profile-guided optimization
    • My experience with Hex-Rays 2.2.0
      • Bugs
      • Odd peculiarities
      • Silence
      • Comma
      • Data types
      • Long and messed expressions
      • My plan
      • Summary
  • Books/blogs worth reading
    • Books and other materials
      • Reverse Engineering
      • Windows
      • C/C++
      • x86 / x86-64
      • ARM
      • Assembly language
      • Java
      • UNIX
      • Programming in general
      • Cryptography
  • Communities
  • Afterword
    • Questions?
  • Appendix
    • x86
      • Terminology
      • General purpose registers
      • FPU registers
      • SIMD registers
      • Debugging registers
      • Instructions
      • npad
    • ARM
      • Terminology
      • Versions
      • 32-bit ARM (AArch32)
      • 64-bit ARM (AArch64)
      • Instructions
    • MIPS
      • Registers
      • Instructions
    • Some GCC library functions
    • Some MSVC library functions
    • Cheatsheets
      • IDA
      • OllyDbg
      • MSVC
      • GCC
      • GDB
  • Acronyms used
  • Glossary
  • Index
    No review for this book yet, be the first to review.
      No comment for this book yet, be the first to comment
      You May Also Like
      Also Available On
      App store smallGoogle play small
      Categories
      Curated Lists
      • Pattern Recognition and Machine Learning (Information Science and Statistics)
        by Christopher M. Bishop
        Data mining
        by I. H. Witten
        The Elements of Statistical Learning: Data Mining, Inference, and Prediction
        by Various
        See more...
      • CK-12 Chemistry
        by Various
        Concept Development Studies in Chemistry
        by John Hutchinson
        An Introduction to Chemistry - Atoms First
        by Mark Bishop
        See more...
      • Microsoft Word - How to Use Advanced Algebra II.doc
        by Jonathan Emmons
        Advanced Algebra II: Activities and Homework
        by Kenny Felder
        de2de
        by
        See more...
      • The Sun Who Lost His Way
        by
        Tania is a Detective
        by Kanika G
        Firenze_s-Light
        by
        See more...
      • Java 3D Programming
        by Daniel Selman
        The Java EE 6 Tutorial
        by Oracle Corporation
        JavaKid811
        by
        See more...