Evasion and Countermeasures Techniques to Detect Dynamic Binary Instrumentation Frameworks

Dynamic Binary Instrumentation (DBI) is a dynamic analysis technique that allows arbitrary code to be executed when a program is running. DBI frameworks have started to be used to analyze malicious applications. As a result, different approaches have merged to detect and avoid them. Commonly referred to as split personality malware or evasive malware are pieces of malicious software that incorporate snippets of code to detect when they are under DBI framework analysis and thus mimic benign behavior. Recent studies have questioned the use of DBI in malware analysis, arguing that it increases the attack surface. In this paper, we examine the anti-instrumentation techniques that abuse desktop-based DBI frameworks and existing countermeasures to determine if it is possible to reduce the exploitable attack surface introduced by these DBI frameworks. In particular, we review the related literature to identify (i) the existing set of DBI framework evasion techniques and (ii) the existing set of countermeasures to avoid them. We also analyze and compare the taxonomies introduced in the literature, and propose a new taxonomy that expands and completes previous taxonomies. We also note some relevant issues and outline ways of future research in the use of DBI frameworks for security purposes

SHFuzz: Selective Hybrid Fuzzing with Branch Scheduling Based on Binary Instrumentation

Hybrid fuzzing is a popular software testing technique that combines random fuzzing with concolic execution. It is widely used in the security domain known for its ability to find deeply hidden vulnerabilities and reach high code coverage. Hybrid fuzzing is based on negating branches in the execution path of a specific input to generate new test cases. However, due to numerous inputs and related branches, it does not show the best of its effectiveness without input and branch selection methods. In this paper, we systematically analyze the branch scheduling problem in the internal attributes of hybrid fuzzing, focusing on the synchronization mechanism. To solve the problems, we propose the Selective Hybrid Fuzzing (SHF) approach with branch scheduling based on binary instrumentation. There are two major parts to the SHF approach: (1) we propose a critical branch selection algorithm to select critical branches by three metrics: hit accuracy, solvability, and complexity; (2) we propose a priority score calculation algorithm to select inputs by the number of critical branches. With the SHF approach, we choose only the branches that can be negated to generate new coverage, instead of repeatedly executing the same branches and generating duplicates of inputs. We implement a hybrid fuzzer called SHFuzz with our SHF approach and compare it with the state-of-the-art hybrid fuzzer QSYM. In the evaluation, SHFuzz outperforms QSYM in 20 real-world applications from the Google Fuzzer Test Suite and other program suites in a 12 h test. On average, SHFuzz achieves 8.40% more code coverage and 100 more unique crashes in each application. Our work also finds existing vulnerabilities 7.85× faster than QSYM. We also find new bugs by SHFuzz, which QSYM fails to find. Our evaluation shows that the selective hybrid fuzzing approach can reduce the number of branches executed in concolic execution, enhancing hybrid fuzzing on code coverage and bug finding capabilities.

BAHK: Flexible Automated Binary Analysis Method with the Assistance of Hardware and System Kernel

To protect core functions, applications often utilize the countermeasure techniques such as antidebugging to avoid analysis by outsiders, especially the malware. Dynamic binary instrumentation is commonly used in the analysis of binary programs. However, it can be easily detected and has stability and applicability problems as it involves program rewriting and just-in-time compilation. This paper proposes a new lightweight analysis method for binary programs with the assistance of hardware features and the operating system kernel, named BAHK, which can automatically analyze the target program by stealth and has wide applicability. With the support of underlying infrastructures, this paper designs several optimization strategies and specific analysis approaches at instruction level to reduce the impact of fine-grained analysis on the performance of target program so that it can be well applied in practice. The experimental results show that the proposed method has good stealthiness, low memory consumption, and positive user experience. In some cases, it shows better analysis performance than the traditional dynamic binary instrumentation method. Finally, the real case studies further show its feasibility and effectiveness.