UC Riverside

Malware infects thousands of systems globally each day causing millions of dollars in damages. Tools like anti-malware engines and disassemblers are essential front-line tools in malware defense. Anti-malware engines are used to detect malware while disassemblers are used to analyze the malware, understand its operations, and defuse it. Our overarching goal is to identify and improve our ability to detect and understand malware and consists of three major thrusts. First, we address the problem of identifying which available disassembler gives the most accurate disassembly for malware binaries of the ARM and MIPS architecture. Surprisingly, our comprehensive and systematic evaluation revealed that disassemblers have complementary capabilities. Furthermore, it also led to a bug discovery in Ghidra. Second, we leverage the results from our evaluation, identify weaknesses in disassemblers, and we develop methods to improve disassembly accuracy. As a key novelty, we develop the first approach to combine disassemblers efficiently using an ensemble approach to improve disassembly accuracy significantly. Third, we adopt a hacker-centric approach and we stress-test the effectiveness and robustness of anti-malware engines against IoT malware. Our goal is to develop evasion techniques that: (a) minimize the required effort to modify the source code and (b) preserve the functionality of the malware. Surprisingly, we find that anti-malware engines rely significantly on string matching for detection and labelling. Leveraging this, we show that some simple techniques achieve 100% evasion rate for IoT malware binaries by applying string manipulations in the source code. This thesis is a significant contribution towards (a) assessing existing and developing new capabilities in disassembling binaries and (b) understanding how anti-malware engines detect and label malware, especially in the space of IoT malware.