Making the Abstract Physical: Toward Improving Binary Reverse Engineering via Embodied Immersion

Abstract

Performing reverse engineering (RE) to determine precisely what a piece of software does is vital to tasks such as securing networks, mitigating malware, and maintaining legacy software. Much software is distributed as binary executable programs, which are particularly difficult for humans to comprehend because the compilation process is a one-way transformation from context-rich source code to a highly-optimized binary program. Our central problem is that binary RE is a highly-specialized skill that requires extensive training and experience. Additionally, the RE process requires a human-in-the-loop because the compound uncertainties introduced in disassembling and decompiling a binary program prevent a fully-automated solution. This work postulates and tests methods to improve the effectiveness of the human-machine joint cognitive system performing sensemaking in the context of binary RE, particularly, leveraging affordances of immersive virtual reality (VR) that exploit facets of cognition typically underutilized in the task of binary RE.

In tackling this problem, we followed a hybrid human-centered interaction design process combining Design Thinking (DT) with Cognitive Systems Engineering (CSE). Within our discovery phase, we performed a thorough interdisciplinary survey providing the theoretical basis for augmented the RE process with immersive VR. In our definition phase, we prioritized the identified affordances in VR into an initial set for the development phase, launching into multiple iterations of build and test of our VR system, Cognitive Binary Reverse Engineering (CogBRE), leveraging feedback from RE practitioners in each iteration.

Based on informal feedback from these iterations indicating that practitioners value the ability to place and organize code fragments and flow graphs in the expanse of VR as they form an understanding of a binary program, we designed and executed a formal user study. Using a between-subjects design, we compared CogBRE (in two VR configurations) to a traditional desktop interface across several metrics, including performance, cognitive load, usability, and user experience. While task accuracy was statistically equivalent across conditions, participants in VR reported significantly lower cognitive load and described meaningful advantages in spatial organization and contextual reasoning.

Recognizing the rise of large language models (LLMs) and spurred by early feedback that users sought novel graph visualizations, we conducted a pilot study using an LLM to generate 3D function call graphs based on self-directed interrogation of disassembled binaries. Evaluators found that the LLM-generated visualizations were often correct, interpretable, and helpful, suggesting future potential for LLMs as collaborative agents in immersive environments.

This work contributes a novel VR system to augment the binary RE process, empirical evidence that embodied spatial interaction can reduce subjective cognitive load, early evidence that LLMs can serve as visualization and reasoning partners, and a generalizable research framework for applying embodied interaction to other cognitively demanding technical domains.