Understanding and protecting closed-source systems through dynamic analysis

Virtualization;Security;Reverse engineering

Dolan-Gavitt, Brendan ; Lee, Wenke Computer Science Traynor, Patrick Ahamad, Mustaque Giffin, Jon Cui, Weidong ; Lee, Wenke

University:Georgia Institute of Technology

Department:Computer Science

关键词: Virtualization;    Security;    Reverse engineering;   

Others  :  https://smartech.gatech.edu/bitstream/1853/53012/1/DOLAN-GAVITT-DISSERTATION-2014.pdf

美国|英语

来源: SMARTech Repository

【 摘 要 】

In this dissertation, we focus on dynamic analyses that examine the data handled by programs and operating systems in order to divine the undocumented constraints and implementation details that determine their behavior in the field. First, we introduce a novel technique for uncovering the constraints actually used in OS kernels to decide whether a given instance of a kernel data structure is valid. Next, we tackle the semantic gap problem in virtual machine security: we present a pair of systems that allow, on the one hand, automatic extraction of whole-system algorithms for collecting information about a running system, and, on the other, the rapid identification of “hook points” within a system or program where security tools can interpose to be notified of security-relevant events. Finally, we present and evaluate a new dynamic measure of code similarity that examines the content of the data handled by the code, rather than the syntactic structure of the codeitself. This problem has implications both for understanding the capabilities of novelmalware as well as understanding large binary code bases such as operating system kernels.