Binary analysis is the process of examining compiled software (the machine-readable code your computer actually runs) to understand what a program does, find security flaws, or detect malicious behavior. Unlike reviewing human-readable source code, binary analysis works with the finished product: the executable files, firmware, and libraries that have already been translated into ones and zeros. It’s a core skill in cybersecurity, reverse engineering, and software quality assurance.
Why Analyze Binaries Instead of Source Code
Most software you interact with ships as a compiled binary. You don’t get the original source code for your router’s firmware, a suspicious email attachment, or a proprietary application you’re auditing for security. Binary analysis lets researchers and security professionals inspect these programs anyway. It’s also essential when source code technically exists but doesn’t tell the full story: compilers can introduce unexpected behavior, and some vulnerabilities only surface in the final compiled output.
In cybersecurity, binary analysis is used to identify vulnerabilities, detect malware, attribute malicious code to specific authors, and verify that software meets security compliance standards. In digital forensics, it helps investigators reconstruct what a piece of software did on a compromised system.
Static vs. Dynamic Analysis
The two fundamental approaches to binary analysis are static and dynamic, and most real-world work combines both.
Static analysis examines a binary without running it. Analysts feed the file into tools that scan for patterns: known vulnerability signatures, insecure coding practices, hardcoded credentials, or logical errors like infinite loops and unreachable code. Because the program never executes, static analysis is safe to perform on malware and can begin the moment you have a file to inspect. The tradeoff is that it can miss problems that only appear when the software is actually running.
Dynamic analysis runs the program in a controlled environment and watches what happens. This reveals runtime behavior like how the software handles user input, manages memory, communicates with external servers, and processes authentication. Performance bottlenecks and security issues tied to session management or data processing often only show up during execution. The downside is that dynamic analysis only covers the specific paths the program takes during testing, so it may miss code that wasn’t triggered.
Hybrid approaches combine both methods for broader coverage. A static pass might identify suspicious code regions, then dynamic testing confirms whether those regions are actually exploitable during execution.
Disassembly and Decompilation
Before anyone can meaningfully analyze a binary, it needs to be translated back into something a human can read. This happens through two related processes.
Disassembly converts the binary’s machine code into assembly language, a low-level representation that maps closely to what the processor actually executes. Tools approach this in two ways. Simpler disassemblers like objdump use “linear sweep,” starting at an entry point and decoding instructions sequentially. More sophisticated tools like Ghidra, IDA Pro, and Binary Ninja use “recursive traversal,” which follows the program’s actual control flow to decode all reachable instructions. Recursive traversal produces more accurate results, especially when the binary contains data mixed in with code.
Decompilation goes a step further, attempting to reconstruct higher-level source code from the binary. The output isn’t identical to what the original developer wrote, but it’s far more readable than assembly. Decompilation involves disassembly, data flow analysis, control flow analysis, and data type inspection to produce something resembling C or another high-level language.
Control Flow Graphs and Program Mapping
One of the most useful outputs of binary analysis is a control flow graph, a visual map of how a program executes. Code blocks are connected by edges showing the order in which they run, including branches, loops, and function calls. When the program hits a call instruction, the graph traces the jump to the called function. When it returns, the graph tracks that path back.
These graphs help analysts trace how data moves through a program. If you’re looking for a vulnerability, you can follow a path from the point where user input enters the program, through each internal function that transforms it, all the way to the function where a flaw might be exploitable. That chain of transformations shows up as a clear path in the graph, making it much easier to understand complex programs that might contain hundreds of thousands of instructions.
Common Vulnerabilities Found Through Binary Analysis
Binary analysis routinely uncovers several categories of security flaws. The most common include:
- Stack overflow: The program writes more data to a memory buffer than it can hold, overwriting adjacent memory. A real-world example is CVE-2018-7445, a vulnerability in MikroTik RouterOS where a function copied data into a buffer without checking whether it would fit. The first byte of input was used as the copy size, with no validation, letting an attacker overflow the stack.
- Heap overflow: Similar to a stack overflow but targeting a different region of memory, often harder to detect and exploit.
- Format string overflow: The program uses user-supplied input as a formatting template, allowing attackers to read or write arbitrary memory.
- Integer overflow: A mathematical operation produces a number too large for its data type, wrapping around to an unexpected value and causing downstream errors.
- Logic vulnerabilities: Flaws in the program’s decision-making that let an attacker bypass intended restrictions without corrupting memory at all.
Automated tools can scan binaries for these patterns at scale. In competitive security challenges, datasets of 20 to 25 binary files containing these vulnerability types are used to benchmark automated discovery systems.
Symbolic Execution
One advanced technique used in binary analysis is symbolic execution. Instead of running a program with concrete input values (like “test” or “12345”), symbolic execution uses placeholder variables that represent all possible inputs simultaneously. The system then tracks how these symbolic values flow through the program, building mathematical equations that describe the conditions needed to reach each code path.
This lets analysts systematically explore paths through a program that would be nearly impossible to reach through manual testing. If a vulnerability requires a very specific sequence of inputs to trigger, symbolic execution can generate those inputs automatically. The challenge is complexity: real-world programs have enormous numbers of possible paths, and the mathematical models can become too large to solve efficiently. Research from Stanford has focused on making these systems more reliable by cross-checking their internal components for consistency, since errors in the analysis stack can produce false paths that waste time and mislead analysts.
Obfuscation and Anti-Analysis Techniques
Not all binaries cooperate with analysis. Malware authors and some commercial software developers deliberately make their code harder to inspect. Research from the University of Wisconsin cataloged a broad set of these techniques, organized by what they prevent.
Some obfuscations make code hard to discover in the first place: hiding control transfers, using exception handlers to redirect execution, or unpacking code in small increments so the full program is never visible at once. Others target disassemblers specifically by mixing ambiguous code and data or inserting bytes designed to confuse decoding tools. Still others make the program structure unreadable by disguising function calls, tampering with the call stack, or creating overlapping functions. At the implementation level, techniques include obfuscating constant values, inserting “do-nothing” code that clutters the output, and self-checksumming routines that detect when a binary has been modified for analysis.
Firmware binaries add their own challenges. Packing schemes are often vendor-specific and proprietary. Testers frequently encounter encryption, non-standard compression, or monolithic images that bundle the kernel, operating system, and applications into a single file that has to be carefully separated before analysis can begin.
Firmware and IoT Security Auditing
Binary analysis plays a critical role in evaluating the security of embedded devices like routers, cameras, and industrial controllers. The process follows a consistent workflow. First, the firmware is acquired from a vendor’s update repository, extracted directly from a device’s storage chip, or captured during a wireless update. Next, the binary is unpacked, dealing with any compression or encryption. Then it’s decompiled into a more readable form.
Once the firmware is accessible, auditors apply static analysis to scan for known vulnerability patterns and dynamic analysis by running the firmware in an emulated environment to observe its behavior. This emulation step, called “re-hosting,” is often the hardest part, since embedded firmware expects specific hardware that has to be simulated convincingly enough for the code to run.
Binary Analysis in Software Development Pipelines
Binary analysis isn’t just for security researchers working after the fact. Modern development teams integrate it directly into their build and deployment pipelines. In a DevSecOps workflow, static analysis tools scan source code, bytecode, or compiled binaries automatically each time a developer commits new code. This “shift-left” approach catches vulnerabilities during development, when they’re cheapest and fastest to fix.
During later stages like staging and deployment, dynamic and interactive analysis tools run in parallel to verify runtime behavior and input handling. Some organizations use AI-enhanced systems to prioritize which discovered vulnerabilities pose the greatest risk, helping teams focus their effort where it matters most rather than drowning in low-severity findings.