Method to detect application execution hijacking using memory protection

Description

FIELD

Embodiments of the disclosure relate to the field of cyber-security. More specifically, one embodiment of the disclosure relates to a system, apparatus and method for identifying potential application-execution hijacking attacks using memory protection techniques.

GENERAL BACKGROUND

Over the last decade, malicious software has become a pervasive problem for Internet users as many networked resources include vulnerabilities that are subject to attack. For instance, over the past few years, more and more vulnerabilities are being discovered in software that is loaded onto network devices, such as vulnerabilities within operating systems for example. While some vulnerabilities continue to be addressed through software patches, prior to the release of such software patches, network devices will continue to be targeted for attack by malware, namely information such as computer code that attempts during execution to take advantage of a vulnerability in computer software by acquiring sensitive information or adversely influencing or attacking normal operations of the network device or the entire enterprise network.

For example, one commonly exploited vulnerability is known as a buffer overflow. In general, programs write data to a buffer. However, during a buffer overflow, the written data overruns the buffer's allocated boundary and overwrites adjacent memory locations. As a result, buffer overflows are the basis of many software vulnerabilities and can be maliciously exploited to cause erratic program behavior, memory access errors, incorrect results, and/or the misappropriation of sensitive data such as intellectual property.

Various techniques have been attempted to detect and prevent software exploits, however each technique has various tradeoffs. One of the most generic techniques include Data Execution Prevention (DEP), which is generally provided for by a processor. Using DEP, memory spaces are automatically marked as non-executable unless they are explicitly told they are being allocated for executable code. Specifically, a flag is set on a per-page basis and is set via a bit in the page table entry (PTE) for that page. If an attempt is made to execute code from a memory region that is marked as non-executable, the hardware feature passes an exception to DEP within the operating system and provides a corresponding indication. Consequently, DEP causes an exception within the code stack that is executing, thereby causing a failure coupled with an access violation. DEP may be made stronger by CPU support with the No-Execute (NX) bit, also known as the XD bit, EVP bit, or XN bit, which allows the CPU to enforce execution rights at the hardware level.

Unfortunately, in short order, bypasses were developed by hackers to overcome DEP schemes. Specifically, a technique known as Return-Oriented Programming (ROP) was developed to circumvent DEP schemes. ROP techniques search for portions of code known as ROP gadgets in legitimate modules within a particular process. ROP gadgets generally comprise of one or more instructions, followed by a return. Combining a plurality of ROP gadgets along with appropriate values in the stack allows for the malicious shell code to be executed. Typically, the hacker's goal is to locate the address of a memory protection API, such as VirtualProtect, and mark the relevant memory region as executable (as compared to non-executable). Thereafter, the hacker may introduce a final ROP gadget to transfer the execution to the relevant memory region to execute the shellcode. As a result, the DEP scheme may be bypassed.

In an effort to make potential DEP bypasses more difficult, Address Space Layout Randomization (ASLR) was developed. ASLR involves randomly offsetting memory structures and module base addresses such that merely “guessing” the location of ROP gadgets and APIs becomes exceedingly difficult. On certain operating systems, such as the Microsoft® Windows® operating system, ASLR may be configured to randomize the location of executables and Dynamic Link Libraries (DLLs) in memory, stacks and heaps. For example, when an executable is loaded into memory, the operating system may receive a processor's timestamp counter (TSC), shift the TSC by a nominal amount, perform a division (e.g., a modulo operation), and then add a constant. The result of this operation may then be multiplied by yet another constant, at which point an executable image is loaded at the calculated offset.

However, some DLLs (including for example, ntdll, kernel32, etc.) are shared in memory across processes, their offsets are determined by a system-wide bias value that is computed at boot. Notably, the offset value is computed only once per boot. When DLLs are loaded, they are disposed into a shared memory region. The order in which modules are loaded is randomized too. Furthermore, when threads are created, their stack base address is randomized. Once the base address has been calculated, another value is derived from the TSC to compute the final stack base address. By using this method, ASLR was intended to provide a high theoretical degree of randomness.

When all of these ALSR mechanisms were combined with DEP, it was understood that shellcode would be prevented from executing because the memory region could not be executed. Moreover, it was expected that potential hackers also would not know the location of any ROP instructions in memory because the ROP gadget's address would be unreliable due to the randomization.

Nonetheless, bypasses were developed by hackers to overcome ASLR mechanisms so that DEP schemes could be easily exploited. For example, NOP sleds may be utilized to create a probabilistic exploit. Furthermore, using a pointer leak, a hacker can make an educated guess regarding a value on the stack at a reliable location to locate a usable function pointer or ROP gadget.

In other words, using these and other techniques, it may be possible to create a payload that reliably bypasses both DEP and ASLR. Moreover, once ASLR and DEP are compromised, it is a straight forward matter to control the execution of shellcode in the context of the application. Therefore, there exists a need for a system, apparatus and method for identifying potential application-execution hijacking attacks using memory protection techniques so as to prevent the execution of malicious shellcode.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:

FIG. 1A is an exemplary logical representation of a communication system deploying an application-execution hijacking detection system communicatively coupled to a network.

FIG. 1B an exemplary embodiment of a representation of the application-execution hijacking detection system of FIG. 1A.

FIG. 2 is an exemplary representation of a plurality of mobile devices deploying an application-execution hijacking detection application communicatively coupled to a network.

FIG. 3 is an exemplary embodiment of a mobile device deploying an application-execution hijacking detection application according to the present disclosure.

FIG. 4 is an exemplary block diagram of an operational flow of an application-execution hijacking detection system.

FIG. 5 is an exemplary block diagram of an alternative operational flow of an application-execution hijacking detection system.

FIG. 6 is a flowchart of an exemplary method for detecting application-execution hijacking malware by applying a guard page to a base address of a loaded module.

FIG. 7 is a flowchart of an exemplary method for detecting application-execution hijacking malware by applying a guard page to an import table of a loaded module.

FIG. 8 is a flowchart of an exemplary method for detecting application-execution hijacking malware by applying a hardware breakpoint to a base address of a loaded module.

FIG. 9 is a flowchart of an exemplary method for detecting application-execution hijacking malware by applying a guard page to a process environment block of a loaded module.

DETAILED DESCRIPTION

Various embodiments of the disclosure relate to a network appliance, such as an application-execution hijacking detection system (AEH) system for example, where the network appliance comprises a dynamic analysis server. According to one embodiment of the disclosure, information from received network traffic is analyzed to determine whether at least a portion of the received network traffic is likely to be associated with malware. A portion of the received network traffic, (hereinafter “object(s)”), that is determined to likely be associated with malware is deemed “suspicious.” The dynamic analysis server comprises virtual execution logic to automatically analyze one or more objects while the object(s) executes within a virtual machine (VM). In particular, the dynamic analysis server comprises event logic to analyze whether attempts have been made to access protected (guarded) pages of loaded modules so as to detect a potential AEH attack. In one embodiment, instead of implementing virtual execution logic, an AEH detection application may be configured so as to communicate the output of DLL/kernel logic directly into an application running on a mobile device, for example. It is envisioned that deployments other than VM-based deployments may also be used, including but not limited to runtime system deployments and the like.

Herein, according to one embodiment of the disclosure, the dynamic analysis server is configured to monitor and store access events of guarded page areas of any of various loaded modules. Any time a read, write or execute operation (“access event”) is performed on such guarded page areas, the access event is analyzed to determine whether it is malicious (and therefore associated with a malware attack), or non-malicious. For example, if the access source is from the heap, then there is a high likelihood that the access event is malicious. As used herein, the heap refers to any portion of memory where dynamically allocated memory resides. More specifically, in the context of loaded modules, the memory type is a “MEM_IMAGE” type. Conversely, if the memory type at any address is not the “MEM_IMAGE” type, then the address is from the heap, and therefore malicious. It is envisioned that upon a finding of maliciousness, an alert may be generated to communicate details of the access event, including for example the access source and corresponding memory type. In some embodiments, determining that an access event is malicious may result in the termination of the respective application, process and/or operation.

I. Terminology

In the following description, certain terminology is used to describe features of the invention. For example, in certain situations, term “logic” is representative of hardware, firmware and/or software that is configured to perform one or more functions. As hardware, logic may include circuitry having data processing or storage functionality. Examples of such circuitry may include, but are not limited or restricted to a microprocessor, one or more processor cores, a programmable gate array, a microcontroller, an application specific integrated circuit, wireless receiver, transmitter and/or transceiver circuitry, semiconductor memory, or combinatorial logic.

Logic may be software in the form of one or more software modules, such as executable code in the form of an executable application, an application programming interface (API), a subroutine, a function, a procedure, an applet, a servlet, a routine, source code, object code, a shared library/dynamic load library, or one or more instructions. These software modules may be stored in any type of a suitable non-transitory storage medium, or transitory storage medium (e.g., electrical, optical, acoustical or other form of propagated signals such as carrier waves, infrared signals, or digital signals). Examples of non-transitory storage medium may include, but are not limited or restricted to a programmable circuit; a semiconductor memory; non-persistent storage such as volatile memory (e.g., any type of random access memory “RAM”); persistent storage such as non-volatile memory (e.g., read-only memory “ROM”, power-backed RAM, flash memory, phase-change memory, etc.), a solid-state drive, hard disk drive, an optical disc drive, or a portable memory device. As firmware, the executable code is stored in persistent storage.

The term “object” generally refers to a collection of data, whether in transit (e.g., over a network) or at rest (e.g., stored), often having a logical structure or organization that enables it to be classified for purposes of analysis. During analysis, for example, the object may exhibit a set of expected characteristics and, during processing, a set of expected behaviors. The object may also exhibit a set of unexpected characteristics and a set of unexpected behaviors that may evidence an exploit and potentially allow the object to be classified as an exploit.

Examples of objects may include one or more flows or a self-contained element within a flow itself. A “flow” generally refers to related packets that are received, transmitted, or exchanged within a communication session. For convenience, a packet is broadly referred to as a series of bits or bytes having a prescribed format, which may include packets, frames, or cells.

As an illustrative example, an object may include a set of flows such as (1) a sequence of transmissions in accordance with a particular communication protocol (e.g., User Datagram Protocol (UDP); Transmission Control Protocol (TCP); or Hypertext Transfer Protocol (HTTP); etc.), or (2) inter-process communications (e.g., Remote Procedure Call “RPC” or analogous processes, etc.). Similar, as another illustrative example, the object may be a self-contained element, where different types of such objects may include an executable file, non-executable file (such as a document or a dynamically link library), a Portable Document Format (PDF) file, a JavaScript file, Zip file, a Flash file, a document (for example, a Microsoft Office® document), an electronic mail (email), downloaded web page, an instant messaging element in accordance with Session Initiation Protocol (SIP) or another messaging protocol, or the like.

According to one embodiment, the term “malware” may be construed broadly as any code or activity that initiates a malicious attack and/or operations associated with anomalous or unwanted behavior. For instance, malware may correspond to a type of malicious computer code that executes an exploit to take advantage of a vulnerability, for example, to harm or co-opt operation of a network device or misappropriate, modify or delete data. Malware may also correspond to an exploit, namely information (e.g., executable code, data, command(s), etc.) that attempts to take advantage of a vulnerability in software and/or an action by a person gaining unauthorized access to one or more areas of a network device to cause the network device to experience undesirable or anomalous behaviors. The undesirable or anomalous behaviors may include a communication-based anomaly or an execution-based anomaly, which, for example, could (1) alter the functionality of an network device executing application software in an atypical manner (a file is opened by a first process where the file is configured to be opened by a second process and not the first process); (2) alter the functionality of the network device executing that application software without any malicious intent; and/or (3) provide unwanted functionality which may be generally acceptable in another context. Additionally, malware may be code that initiates unwanted behavior which may be, as one example, uploading a contact list from an endpoint device to cloud storage without receiving permission from the user.

The term “shellcode” refers to a small piece of executable code that resides in data (e.g., is injected into data), is used as a payload of malware, or, in some cases, contains a shell command to execute an exploit.

The term “transmission medium” is a physical or logical communication path between two or more network devices (e.g., any devices with data processing and network connectivity such as, for example, a security appliance, a server, a mainframe, a computer such as a desktop or laptop, netbook, tablet, firewall, smart phone, router, switch, bridge, etc.). For instance, the communication path may include wired and/or wireless segments. Examples of wired and/or wireless segments include electrical wiring, optical fiber, cable, bus trace, or a wireless channel using infrared, radio frequency (RF), or any other wired/wireless signaling mechanism.

In certain instances, the term “detected” is used herein to represent that there is a prescribed level of confidence (or probability) on the presence of an exploit within an object under analysis.

The term “pattern” should be construed as a predefined grouping of instructions. A pattern may appear in memory, such as memory allocated in a virtual execution environment for use by an application being executed by a virtual machine. In some embodiments, the length of the pattern may correspond to the operating system of the network device which is undergoing analysis. For example, a pattern may consist of four bytes when the network device is running a 32-bit operating system (this may be referred to as a double word, or “DWORD”). Therefore, the DWORD may contain up to four (4) instructions, which may be four (4) NOP instructions, for example. Alternatively, a pattern may consist of eight bytes when the network device is running a 64-bit operating system (this may be referred to as a quad word, or “QWORD”). Therefore, the QWORD may contain up to eight (8) instructions, which may be eight (8) NOP instructions, for example.

The terms “network device” or “network appliance” should be construed as any electronic device with the capability of connecting to a network. Such a network may be a public network such as the Internet or a private network such as a wireless data telecommunication network, wide area network, a type of local area network (LAN), or a combination of networks. Examples of a network device may also include, but are not limited or restricted to mobile devices, such as a laptop, a mobile phone, a tablet, a computer, etc. or any other relatively portable device.

The term “computerized” generally represents that any corresponding operations are conducted by hardware in combination with software and/or firmware. Also, the terms “compare” or “comparison” generally mean determining if a match (e.g., a certain level of correlation) is achieved between two items where one of the items may include a particular signature pattern.

Lastly, the terms “or” and “and/or” as used herein are to be interpreted as inclusive or meaning any one or any combination. Therefore, “A, B or C” or “A, B and/or C” mean “any of the following: A; B; C; A and B; A and C; B and C; A, B and C.” An exception to this definition will occur only when a combination of elements, functions, steps or acts are in some way inherently mutually exclusive.

As this invention is susceptible to embodiments of many different forms, it is intended that the present disclosure is to be considered as an example of the principles of the invention and not intended to limit the invention to the specific embodiments shown and described.

II. Application-Execution Hijacking Detection System

A. General Architecture of Network Appliance Deploying Application-Execution Hijacking Detection Logic

Referring to FIG. 1A, an exemplary logical representation of an AEH detection environment 102 communicatively coupled to a network 110 via a network interface 120 is shown. The AEH detection environment 102 comprises a server device 105, an optional firewall 115, a client device 125 and an AEH detection system 100 communicatively coupled to the network 110 via a network interface 120. The AEH system 100 is further communicatively coupled to an analysis network 195. It is envisioned that the analysis network 195 may be configured to store and access data regarding malware attacks across a number of objects, including for example, web-based, email-based, and file-based threats. Moreover, it is envisioned that the analysis network 195 may be configured to store historical information regarding previously analyzed and/or known malware attacks. The analysis network 195 may also be periodically or aperiodically updated so as to store information regarding new malware attacks, reports, alerts, and/or corresponding features, for example.

Herein, according to the embodiment illustrated in FIG. 1A, the AEH detection system 100 comprises a network appliance that is adapted to analyze information associated with network traffic routed over a communication network 110 between at least one server device 105 and at least one client device 125. The communication network 110 may include a public network such as the Internet, in which case an optional firewall 115 (represented by dashed lines) may be interposed in the communication path between the public network and the client device 125. Alternatively, the communication network 110 may be a private network such as a wireless data telecommunication network, wide area network, a type of local area network (LAN), or a combination of networks.

As shown, the AEH detection system 100 may be communicatively coupled with the communication network 110 via a network interface 120. In general, the network interface 120 operates as a data-capturing device (sometimes referred to as a “tap” or “network tap”) that is configured to receive data propagating to/from the client device 125 and provide at least some of this data to the AEH detection system 100. Alternatively, it should be understood that the AEH detection system 100 may be positioned behind the firewall 115 and in-line with client device 125.

According to one embodiment of the disclosure, the network interface 120 is capable of receiving and routing objects associated with network traffic to the AEH detection system 100. The network interface 120 may provide the entire object or certain content within the object, for example, one or more files or packets that are part of a set of flows, packet payloads, or the like. In some embodiments, although not shown, network interface 120 may be contained within the AEH detection system 100.

It is contemplated that, for any embodiments where the AEH detection system 100 is implemented as a dedicated appliance or a dedicated computer system, the network interface 120 may include an assembly integrated into the appliance or computer system that includes a network interface card and related logic (not shown) for connecting to the communication network 110 to non-disruptively “tap” network traffic propagating through firewall 115 and provide either a duplicate copy of at least a portion of the network traffic or at least a portion the network traffic itself to the dynamic analysis server 130 and an optional static analysis server, if included within the AEH detection system 100. In other embodiments, the network interface 120 can be integrated into an intermediary device in the communication path (e.g., firewall 115, router, switch or other networked network device, which in some embodiments may be equipped with Switched Port Analyzer “SPAN” ports) or can be a standalone component, such as an appropriate commercially available network tap. In virtual environments, a virtual tap (vTAP) can be used to duplicate files from virtual networks.

As further shown in FIG. 1A, the AEH detection system 100 comprises the dynamic analysis server 130, rule-matching logic 150, and reporting logic 160. In some embodiments, an optional static analysis server may be provided within the AEH detection system 100 so as to perform static scanning on a particular object, namely heuristics, exploit signature checks and/or vulnerability signature checks for example. The optional static analysis server and the dynamic analysis server 130 may be one or more software modules executed by the same processor or different processors, where these different processors may be located within the same processor package (e.g., different processor cores) and/or located at remote or even geographically remote locations that are communicatively coupled (e.g., by a dedicated communication link) or a network.

Herein, the dynamic analysis server 130 comprises a virtual machine manager 140, a database 145 and one or more virtual machines (VMs) VM₁-VM_N (N≥1) that may be configured to perform in-depth dynamic analysis with respect to one or more suspicious objects. For instance, the dynamic analysis server 130 may simulate transmission and/or receipt of packets or other objects by a destination device comprising the virtual machines to determine whether certain guarded areas of loaded modules are being accessed in an effort to hijack a particular application.

According to one embodiment, each of the VMs (e.g., VM₁-VM_N) within the dynamic analysis server 130 may be configured with a software profile corresponding to a software image stored within the database 145 that is communicatively coupled with the virtual machine manager 140. Alternatively, the VMs (e.g., VM₁-VM_N) may be configured according to a prevalent software configuration, software configuration used by a network device within a particular enterprise network (e.g., client device 125), or an environment that is associated with the object to be processed, including software such as a web browser application, PDF™ reader application, or the like.

However, for a known vulnerability, the VMs (e.g., VM₁-VM_N) may be more narrowly configured to profiles associated with vulnerable modules. For example, if the access source comprises a certain memory type, VM₁-VM_N may be configured for faster processing and corresponding log file generation. Similarly, when relevant, if the access source is attempting to access a guarded page of its own module, then VM₁-VM_N may be configured accordingly.

In general, the dynamic analysis server 130 is adapted to execute one or more VMs (e.g., VM₁-VM_N) to detect an application-execution hijacking attempt by simulating the execution of an object under analysis within a run-time environment as expected by the type of object. For example, the dynamic analysis server 130 analyzes the received network traffic and determines which application is suitable for executing an object of the received network traffic within one or more VMs, namely VM₁, and/or VM_N.

Each of the VMs (VM₁-VM_N) comprise event logic 135 that is configured so as to detect and/or store all access events, however the event logic 135 may be more narrowly tailored to only focus on access events with respect to guarded areas. The event logic 135 comprises Dynamic Link Library (DLL)/kernel logic 138 that is configured to respond to the occurrence, during computation for example, of exceptions. As used herein, “exceptions” generally refer to anomalous or exceptional conditions requiring special processing that may change the normal flow of a program's execution. It is envisioned that when an attempt is made to access a guarded area, an exception is generated and appropriately handled by the DLL/kernel logic 138. Preferably, the DLL/kernel logic 138 is configured to handle all exceptions that may occur in the event that a guarded area is accessed. In general, the DLL/kernel logic 138 may be considered a library that contains code and data that may be used by more than one programs simultaneously to promote code reuse and efficient memory usage. By using the DLL/kernel logic 138, a program may be modularized into separate components, known as modules. Each module may be loaded into a main program at run time, if that module is loaded. Consequently, as used herein, the DLL/kernel logic 138 utilizes this modular nature to detect access events in conjunction with the AEH detection system 100 to ultimately determine whether the access events are malicious or not.

In one embodiment, the event logic 135 may be configured to generate a log file corresponding to the access events, with special focus being paid to guarded page areas. It is envisioned that the log file may comprise any suitable file format. Once generated, the log file may be communicated to the rule-matching logic 150 so that certain detection logic 155 may be applied thereon.

In one embodiment, the detection logic 155 is configured to apply certain rules on the generated log file. Once the rules are applied, the result is communicated to the reporting logic 160. If maliciousness is found, an alert 190 is generated. The alert 190 may comprise details with respect to the object, such as, by way of non-limiting example, the source of a particular access event, and/or its memory type. In one embodiment, the DLL/kernel logic 138 may be configured to directly block one or more malicious objects by terminating the application and/or process that has been attacked, rather than generating a log file, using the optional process-handling logic 180, for example. It is envisioned that the rules may also be configured so as to perform a probabilistic analysis with respect to some or all of the data associated with the generated log file. For example, the occurrence of an access event may indicate to some level of probability, often well less than 100%, that the access event comprises a certain exploit or exhibits certain elements associated with malware. In one embodiment, the rule-matching logic 150 may be configured to take certain action, including for example, generating an alert 190 if the probability exceeds a prescribed value, for example.

Referring now to FIG. 1B, an exemplary embodiment of a representation of the AEH detection system of FIG. 1A is shown. In one embodiment, a network appliance 106 comprises a housing 103, which is made entirely or partially of a rigid material (e.g., hardened plastic, metal, glass, composite or any combination thereof) that protect circuitry within the housing 103, namely one or more processors 109 that are coupled to communication interface logic 112 via a first transmission medium 116. Communication interface logic 112 enables communications with other AEH detection systems 100 and/or the analysis network 195 of FIG. 1A, for example. According to one embodiment of the disclosure, communication interface logic 112 may be implemented as a physical interface including one or more ports for wired connectors. Additionally, or in the alternative, communication interface logic 112 may be implemented with one or more radio units for supporting wireless communications with other electronic devices.

Processor(s) 109 may further be coupled to persistent storage 122 via a second transmission medium 119. According to one embodiment of the disclosure, persistent storage 122 may include the AEH detection system 100, which in one embodiment comprises (a) dynamic analysis server 130; (b) rule-matching logic 150; (c) optional process handling logic 180; and reporting logic 160. It is envisioned that one or more of these systems (or logic units) could be implemented externally from the AEH detection system 100 without extending beyond the spirit and scope of the present disclosure.

B. General Architecture of a Mobile Device Deploying an Application-Execution Hijacking Detection Application

Referring to FIG. 2, a plurality of mobile devices 230A-230C deploying an AEH detection application communicatively coupled to an analysis network 205 is shown. In general, a network environment 200 is shown, wherein a router 215 is communicatively coupled to analysis network 205 and Internet 210. The router is also communicatively coupled to an optional firewall 220, which itself may be communicatively coupled to a network switch 225. As shown, the plurality of mobile devices 230A-230C may also be communicatively coupled to the analysis network 205 and Internet 210 using any transmission medium, including without limitation, wireless and hardwired connection schemes. It is envisioned that an exemplary AEH detection application 231A-231C corresponding to each of the mobile devices may be installed to detect application-execution hijacking malware. Of course, although only three mobile devices 230A-230C are shown in FIG. 2, any number of devices may have the exemplary AEH detection application 231A-231C loaded thereon. In one embodiment, the AEH detection application may be a downloaded from an app store, website, etc., and thus stored and made locally available with respect to a mobile device.

Referring to FIG. 3, a mobile device may be configured to deploy the AEH detection application 231 of FIG. 2. As shown in FIG. 3, for illustrative purposes, the network device 300 is represented as a mobile device (e.g., smartphone, tablet, laptop computer, netbook, etc.). The mobile device 300 includes a display screen 302, one or more processors 310, a receiver and/or transmitter (e.g. transceiver) such as an antenna 355, and communication interface logic 305. In one embodiment, the AEH detection application 231 comprises exploit detection logic 315, rule-matching logic 330, reporting logic 345, and user interface logic 350.

As shown, the AEH detection application 231 is substantially similar to the AEH detection environment 102 of FIG. 1, except that instead of implementing VMs, the AEH detection application 231 communicates the output of the DLL/kernel logic 315 directly into the AEH detection application 231 running on the mobile device 300. After the output of the DLL/kernel logic 315 is loaded, a log file is generated comprising access events, again with special attention being paid to access events corresponding to guarded page areas. It is envisioned that the log file may be customized so as to only focus on certain criteria. For example, the log file may be filtered according to a particular access source, the memory type being accessed, and in some instances, whether the access source is from the heap. Once the log file is generated, it is communicated to the reporting logic 340, comprising the alert generation logic 345, which may generate an alert and/or route the alert to the analysis network 205 via the communication interface logic 305, for further analysis. In addition, the alert may be routed to the Internet 210 using communication interface logic 305 for further analysis by a network administrator, for example. The reporting logic 340 may issue an alert or report (e.g., an email message, text message, display screen image, etc.) to security administrators or the user, for example, communicating the urgency in handling one or more predicted attacks using the user interface logic 350. The AEH detection application 231 may trigger a further analysis of the object to verify the behavior of the object as an exploit. It is envisioned that the generated alert or report may also comprise instructions so as to prevent one or more predicted malware attacks.

II. Application-Execution Hijacking Detection Methods

FIG. 4 is a flowchart of an exemplary method for detecting AEH malware by applying any of various protection mechanisms, including by way of non-limiting example, applying a page guard to an element of a loaded module so as to establish a protected region. Specifically, as shown at block 405, a page guard is applied to an element of a loaded module, such as by way of non-limiting example, a base address, import table (IT), and/or a process environment block. In general, page guards (also referred to as a “guard page” or “PAGE_GUARD”) provide an alarm for memory page access. Page guards may be used to monitor the growth of large dynamic data structures. For example, certain operating systems use page guards to implement automatic stack checking.

It is to be understood that to create a page guard, the PAGE_GUARD page protection modifier may be set with respect to a particular page. This value may be specified, along with other page protection modifiers, by way of non-limiting example, in various functions such as VirtualAlloc. The PAGE_GUARD modifier may also be used with any other page protection modifiers. In one embodiment, if a code attempts to access an address within a guard page, the system may raise an exception or any other type of violation. Although the principles described herein are especially relevant to the Microsoft® Windows® operating system, many other operating systems and platforms, including without limitation, Apple OS®, iOS®, Android®, Linux®, etc. may also be utilized. At block 410, access events comprising read, write and/or execute operations are monitored and stored, and particular attention is paid with respect to the source of such access attempts. In the event that a guarded area is accessed, an exception is generated at block 415 by the event logic as discussed herein. At block 420, the dynamic analysis server as discussed herein generate a log file comprising access events. At block 440, detection logic is applied to the log file to determine whether an access event is malicious or not. At block 460, after applying the detection logic as discussed in more detail below, a report/alert may be generated. The report/alert may be communicated to the user or a network administrator, for example, and/or stored for further analysis by the analysis network.

FIG. 5 is an exemplary block diagram of an alternative operational flow of an AEH detection system that is substantially similar to the flow described in FIG. 4, except that instead of generating an output file, the potentially malicious object and/or corresponding operation is terminated. For example, at block 505, a page guard is applied to an element of a loaded module, such as by way of non-limiting example, a base address, import table (IT), and/or a process environment block. At block 510, access events are monitored and stored, and particular attention is paid to the source of such access attempts. At block 520, if the access source is determined to be non-malicious, no alert is generated. At block 530, if the access source is indeed malicious, an alert may be generated. Furthermore, if the access source is malicious, the corresponding object and/or process may be terminated, and therefore prevented from executing, as shown at block 535. As stated above, it is envisioned that a probabilistic analysis may also be used in determining whether or not an alert should be generated. For example, the occurrence of an access event may indicate to some level of probability, often well less than 100%, that the access event comprises a certain exploit or exhibits certain elements associated with malware. In one embodiment, the rule-matching logic 150 may be configured to take certain action, including for example, generating an alert if the probability exceeds a prescribed value, for example.

FIGS. 6-9 are flowcharts of exemplary methods for detecting AEH malware, that may be implemented using the rule-matching logic as discussed herein so as to establish a protected region.

Specifically, FIG. 6 shows an exemplary method for detecting AEH malware by applying any of various protection mechanisms such as, by way of non-limiting example, applying a page guard to the base address of a loaded module so as to establish a protected region. At block 605, a loaded module is identified. At block 610, a page guard is applied to the base address of the loaded module as discussed herein, although any number of methods may be used. At block 615, rule-matching logic is configured to determine whether a guarded page is being accessed. If the result of this query is “no”, then there is a high likelihood that the access attempt is not malicious. If the result at block 615 is “yes”, then a second query is presented at block 625.

Specifically, at block 625, the rule-matching logic determines whether the access source is from the heap. In one embodiment, exceptions may comprise context information including for example, values in certain registers at the time the exception was generated that correspond to the address of a source instruction that attempted to access a page guarded area. Once the address of the source instruction is known, a plurality of methods may be used to determine if the address is from the heap. For example, the operating system may provide an API that is configured so as to retrieve information corresponding to the memory at any address. It is envisioned that the API may return a structure that includes information such as memory type. If the memory type is “MEM_IMAGE” then the address is from a loaded module. If the memory type is not “MEM_IMAGE” type, then it is from the heap, and likely malicious. If the result at block 625 is “no”, then there is a high likelihood that the access attempt is not malicious. On the other hand, if the result at block 625 is “yes”, then there is a high likelihood that the access attempt is indeed malicious. In some embodiments, if it is determined that an access attempt is malicious, then an alert may be generated and communicated as discussed herein.

FIG. 7 is a flowchart of an exemplary method for detecting application-execution hijacking malware by applying a guard page to an import table (IT) of a loaded module. In general, the IT is used as a lookup table when an application is calling a function in a different module. Accordingly, at block 705, a loaded module is identified, and at block 710, a protection mechanism such as a page guard is applied to the IAT of the loaded module as discussed herein, although any number of methods may be used so as to establish a protected region. As indicated at block 710, however, the instant method is only applicable with respect to scriptable DLLs that provide a scripting environment to applications, such as, by way of non-limiting example “jscript”, “vbscript”, and “flash.ocx”. At block 715, the rule-matching logic determines whether a guarded page is being accessed. If the result of this query is “no”, then there is a high likelihood that the access attempt is not malicious. If the result at block 715 is “yes”, then a second query is analyzed at block 725. Specifically, the rule-matching logic determines whether the access source is from the heap. If the result of this query is “yes”, then there is a high likelihood that the access attempt is malicious. If the result of this query is “no”, then a third query is presented at block 740. Specifically, at block 740, the rule-matching logic is configured to determine whether the access source is from the same loaded module, and accessing its own import table. If the result of this query is “no”, then there is a high likelihood that the access attempt is not malicious. On the other hand, if the result of this query is “yes”, then there is a high likelihood than the access attempt is malicious, as indicated at block 735.

FIG. 8 is a flowchart of an exemplary method for detecting AEH malware by applying a hardware breakpoint to a base address of a loaded module. At block 805, a loaded module is identified. At block 810, a hardware breakpoint is applied to the base address of the loaded module. Unlike software breakpoints, hardware breakpoints may be configured to establish breakpoints that are initiated when any instruction attempts to read, write, and/or execute a specific memory address. It should be appreciated, however, that hardware breakpoints have certain limitations, including for example the limited number of hardware breakpoints that may be active simultaneously. In some embodiments, for example, only four hardware may be active at the same time.

At block 815, the rule-matching logic determines whether a hardware breakpoint is being accessed. If the result of this query is “no”, then there is a high likelihood that the access attempt is not malicious. If the result at block 815 is “yes”, then a second query is analyzed at block 825. Specifically, the rule-matching logic determines whether the access source is from the heap. If the result at block 825 is “no”, then there is a high likelihood that the access attempt is not malicious. On the other hand, if the result at block 825 is “yes”, then there is a high likelihood that the access attempt is indeed malicious. In some embodiments, if it is determined that an access attempt is malicious, then an alert may be generated as discussed herein.

FIG. 9 is a flowchart of an exemplary method for detecting application-execution hijacking malware by applying a protection mechanism such as a page guard to a process environment block of a loaded module. It should be understood that the process environment block (PEB) is a data structure that is internally utilized by an operating system, most of whose fields are not intended for use by anything other than the operating system. Conventionally, the PEB comprises data structures that apply across a whole process, including global context, startup parameters, data structures for the program image loader, the program image base address, and synchronization objects used to provide mutual exclusion for process-wide data structures, for example.

Consequently, at block 905, a loaded module is identified. At block 910, a protection mechanism, such as a page guard is applied to the process environment block of the loaded module as discussed herein, although any number of methods may be used so as to establish a protected region. At block 915, the rule-matching logic determines whether a guarded page is being accessed. If the result of this query is “no”, then there is a high likelihood that the access attempt is not malicious. If the result at block 915 is “yes”, then a second query is analyzed at block 925. Specifically, the rule-matching logic determines whether the access source is from the heap. In one embodiment, the operating system may provide an API that is configured so as to retrieve information corresponding to the memory at any address. For example, the API may return an output structure that includes information such as the memory address. Similarly, the PEB may be configured so as to provide a list of all of the loaded modules, including, for example, DLLs. The list of all of the loaded modules from the PEB may be analyzed and compared with the output structure to determine if a particular address is from the heap, and therefore malicious. If the result at block 925 is “no”, then there is a high likelihood that the access attempt is not malicious. On the other hand, if the result at block 925 is “yes”, then there is a high likelihood that the access attempt is malicious. In some embodiments, if it is determined that an AEH attack is present, then an alert may be generated as discussed herein.

It should be understood that unless otherwise indicated, the principles described herein are not exclusive to any particular operating system, and thus, systems and methods may be implemented in and/or executed using many operating systems and platforms, including without limitation, Windows®, Apple OS®, iOS®, Android®, etc.

In the foregoing description, the invention is described with reference to specific exemplary embodiments thereof. However, it will be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.