Program security evaluation using dynamic disassembly of machine instructions in virtualized environments

Abstract

Having strong built-in security features has become a paramount requirement in any system. There is a clear difference between bolted vs. built-in security, where in bolted security, the security of the system will depend on the security strength of its bolted parts, where as in built-in security, it is embedded to the system by design. Therefore in order to ensure security, it is required to build security features in to the system by design so that the ultimate security of the system will be ensured by default; ensuring security by design and by default. The execution of a computer program is not stand alone, but instead is a collaborative execution of several programs. Generally at run time, a given program will call functions from other programs and also transfer its control to other program segments, introducing a change to its control flow. In most cases caller (the main program) is not fully aware about its callee (the called program), in the context of its vulnerabilities and security risks. In addition to that, this control transfer will potentially change the trust boundary of the system, while increasing the attack surface of the program in terms of Control Flow Integrity (CFI). On the contrary, completely eliminating this execution behavior is impractical since it is required to build applications having such a modular design due to various reasons, such as performance. Complexity is treated as the enemy of computer security. The more complex a system gets, harder to make it secure. This principle has been studied in detail in the context of program complexity and its relation with security. This research explicitly addresses the question “what is the risk that a microprocessor undergoes due to the execution of user programs?” This opens up a new dimension in security by imposing the importance of runtime program analysis. The research introduces RECSRF; a novel framework to quantitatively evaluate the security of an execution in line with the impact it makes over the microprocessor. RECSRF consists of two components; a novel concept called The Runtime Execution Complexity (REC) of a program execution, which evaluates the tradeoff between performance vs. security, while adhering the Control Flow Integrity (CFI) of programs, and an information theoretic technique to approximate the Security Risk Factor (SRF), which approximates the risk of a particular execution by analyzing dynamically disassembled machine instructions. The RECSRF value allows software designers to select the most secure resource combination among given set of resources, and software implementers to decide whether to proceed or not with a software change. The method can also be used to detect control flow hijacks at runtime by using it as an intrusion detection mechanism which allows transforming the same to an intrusion preventer upon successful implementation. The most notable feature of RECSRF is that it can be applied on highly volatile microprocessors such as on microprocessors hosting virtualized environments.