Analysis of VMProtect

Abstract

The malware dropper “PrivateLoader” that I analyzed last year used a packer that appeared to be VMProtect. The creators of VMProtect claim that it is a feature intended to protect software. However, it is frequently used to obfuscate malware. Moreover, the way this packer protects its code involves a lot of wasteful processing and inefficient encryption, making it unsuitable for use in legitimate software. This suggests that this packer is useful only for malware protection. This document introduces several features of VMProtect found in “PrivateLoader” and provides the knowledge necessary to analyze the malware.

Restrictions:

• This is an article about the analysis of a packer, believed to be VMProtect, found in “PrivateLoader”. There remains the possibility that this malware disguised the packer it is using.
• The version of the packer is unknown. Latest or older versions may have different functionality.

Information:

The analysis tool for IDA traces against VMProtect and the analysis of “PrivateLoader” will be the subject of a separate article.

Malware hash:

MD5: F6570495946923AA4D1467FDBAFBC2F6
SHA1: A0390712FE78C98DB97DC7CCAEA6E0929F548C95

Overview of analyzed features

Distinctive program sections

This section of the executable file is very distinctive. The section names .vmp0, .vmp1, and .vmp2 are the basis for determining that it is a VMProtect.

Figure 1 Sections of “PrivateLoader”

Decrypt payload

• Sections of malware such as .text, .data, and .rdata are empty. These sections store data that is decrypted from other section at the beginning of processing.

Obfuscation

• The code contains a lot of unnecessary instructions.
• Windows API calls are made via code in obfuscated .vmp0 sections.

Encryption of resource data

• The resource data is believed to be encrypted and stored. The reason is that the LdrFindResource_U, LdrAccessResource, user32_LoadStringA, user32_LoadStringW, kernelbase_LoadStringA, and kernelbase_LoadStringW APIs are patched.

Decrypt payload

Regular Windows software stores executable code and data in sections such as .text, .data, .rdata, etc. However, the executable files packed by this packer had these sections filled with 0x00. The data in these sections is encrypted and stored in separate sections. This data is decrypted during the initial processing and stored in these sections.(See: Figure 2–4)

Figure 2 .text section filled with 0x00

Figure 3 Decryption and storing the data

Figure 4 Result of storing decrypted data in .text section

Decryption mechanism

The following describes the decryption process for this data. This is a very simple Caesar cipher — it might be better described as encoding rather than encryption.

Figure 5 Decryption mechanism for data to be stored in segments

Figure 5 shows the data decryption mechanism. Each step is briefly described below.

1. Clears the register (R10) in which decode data is stored and sets the flag. By setting 0x00000001, the register is cleared and the flag is set.
2. Reads 1 byte of data stored in .vmp2 into RAX.
3. Determine data. This involves some complex calculations for obfuscation, rather than a simple comparison of values. However, the result of the calculation is a decision between two options: greater than or less than the threshold.
4. After shifting the value of the decoded data register (R10) one bit to the left, a value of 0 or 1 is set to the least significant one bit, depending on the result of the judgment. Figure 5, 4–1, achieves a 1-bit shift to the left and setting the final bit to 0 by adding the value of R10 to R10. Figure 5, 4–2, achieves a 1-bit shift to the left and setting the final bit to 1 by adding the value of R10 to R10 and then adding another 1.
5. Checks whether decoding of one byte has been completed; checks whether the flag set by 1 in Figure 5 is set to 1 in the lower 9th bit. If it is less than 1 byte, that bit should be 0.
6. The decoded 1-byte data is stored in the section area. If the malware continues to decrypt the following data, repeat the process from 1.

There are two major problems with this encryption method. The first problem is that this method lacks mathematical cryptographic strength. It simply determines if the obfuscated value is greater or less than a certain threshold. This means that plaintext can be easily obtained simply by analyzing the data, formulas, and thresholds. This is vulnerable compared to modern cryptography. The second problem is the inefficiency of the data. At least 1 byte (8 bits) is needed to obtain a 1-bit value. In addition, extra parameters referenced for use in the calculation must also be stored. Requiring at least 8 times the data to decrypt is not an efficient method.

Code Obfuscation

The obfuscation of this packer’s code is very childish. By mixing in so much meaningless code, it only makes the code harder to read. This method is used as a rudimentary obfuscation method in many binary code malware. When these codes are executed, it means that many instructions are executed that are not needed. This is detrimental to the user executing the program because it wastes energy and time.

Figure 6 Obfuscation by adding useless instructions

Windows API Call Obfuscation

The analyzed malware calls Windows APIs via functions in the .vmp0 section, and the code in this .vmp0 section is obfuscated. In the analyzed specimen, the Windows API was executed using the “retn” instruction, meaning that the jump address is controlled by the “retn” instruction.

The following 4 steps are used to make Windows API calls:

1. Read obfuscated address (from .vmp0 section).
2. Calculate address to recover API address.
3. Set calculated address to stack.
4. Calculate the stack address (rsp) to refer to the API address.

An example is explained using figures.

Figure 7 Read obfuscated address from .vmp0 section

Figure 7 shows the “Read obfuscated address”. This mov instruction refers to the data stored at the address of rdi (0x00007FF60AF1537B). This rdi address indicates within the .vmp0 section. The parameter “0x00007FFEF0221DBA” is stored at address “0x00007FFF60AF1537B”. (Note the little endian.) The result of executing the mov instruction is the parameter “0x00007FFEF0221DBA”, which is an obfuscated API address.

Figure 8 Calculate address to recover API address

Figure 8 shows the “Calculate address to recover API address”. Rdi contains the parameters obtained in the previous step, “0x00007FFEF0221DBA”. This lea instruction outputs the result of adding rdi to a fixed value (0x71BF42D6). The resulting value “0x00007FFF61E16090” was the address of the “RtlLeaveCriticalSection” API.

Figure 9 Set calculated address to stack

Figure 9 shows the “Set calculated address to stack”. In this case, the xchg instruction is used to exchange values between the rdi, which contains the API address “0x00007FFF61E16090”, and the stack address. Setting parameters to the stack can be done by using the mov instruction or other means. This is implementation-dependent.

Figure 10 Calculate the stack address (rsp) to refer to the API address

Figure 9 shows the “Calculate the stack address (rsp) to refer to the API address”. In this case, the stack address (rsp) is added just before the “retn” instruction. As a result of the addition, the address of rsp is 0x000000C7214EEB08. This address contains the address of “RtlLeaveCriticalSection” API. The next instruction is “retn”. The “retn” instruction normally returns to the next instruction of the caller. However, this instruction effectively jumps to the rsp address. In this case, the rsp address contains the address of “RtlLeaveCriticalSection”, so the “retn” instruction acts as a jump to this API. These are ways to obfuscate API calls.

Encryption of resource data

I expect this packer to obfuscate resource data in Windows executables. The reason is that they are patching the code of the API that accesses the resource. I have confirmed that the following API code has been patched by the packer.

• LdrFindResource_U
• LdrAccessResource
• LoadStringA (user32.dll)
• LoadStringW (user32.dll)
• LoadStringA (kernelbase.dll)
• LoadStringW (kernelbase.dll)

The “PrivateLoader” malware did not use these APIs. Therefore, I did not analyze the patched code, and obfuscation of the resource data is only speculative.

Figure 11 Patching the code of the API

The left figure in Figure 11 shows the original API code. The first code is overwritten by the packer using the WriteProcessMemory API. The code is eventually modified to jump to the code in the .vmp0 section.

Consideration

This packer was very cheap. This is said to be VMProtect. However, this is not suitable for protecting software as a legitimate service. I have never seen a packed application by a legitimate VMProtect. Therefore, I cannot determine whether the packer used in this malware is VMProtect or an inferior packer disguised as VMProtect.

It is unlikely that this packer will be used with legitimate software. The reasons are as follows:

• Methods of code obfuscation rely on the insertion of a lot of redundant instructions. This degrades CPU performance. (See Figure 6)
• The code is unnecessarily redundant to hide API calls, which degrades CPU performance. (See Figure 7–10)
• The efficiency of data obfuscation is poor. At minimum, encrypting the code requires more than eight times the storage capacity. (See Figure 5)
• At the very least, the code obfuscation does not use sufficiently strong encryption. It is expected that the data protection capabilities will be weak.

The decrease in CPU performance and the increase in the size of the executable program are detrimental to the software’s users. A typical legitimate software vendor would prefer not to impose such costs on its users. Additionally, the inclusion of many redundant instructions or insufficient encryption is expected to weaken the software’s protection, reducing its value for software vendors. Furthermore, it has been confirmed that this packer is used at least in the “PrivateLoader” malware. Therefore, even legitimate software might be at risk of being blocked by security products if they detect this packer.

Based on the above, even if this packer claims to protect legitimate applications, it is considered to have value only to the extent of helping malware evade detection by security products.