Virtualization Based Security - Part 2: kernel communications

This blog post is a second article covering Virtualization Based Security and Device Guard features. In the first part, we covered the system boot process, from the Windows bootloader to the VTL0 startup. In this part, we explain how kernel communications between VTL0 and VTL1 actually work. As they use hypercalls to communicate, we will first describe the Hyper-V hypercalls implementation, then how the kernels use them to communicate. To finish with, we list all the different hypercalls and secure service calls we have identified during this work.

Hyper-V hypercalls

Kernel communications between VTL0 and VTL1 use Hyper-V hypercalls. These hypercalls are performed using the VMCALL instruction, with the hypercall number in the RCX register, and RDX pointing to a Guest Physical Page (GPA) which contains the parameters. If the 0x10000 RCX bit is set, the hypercall is a “FAST” hypercall, and the parameters (and the return values) are stored in R8, RDX and XMM registers (size lays on the R9 register). In order to perform the call, Windows uses a “hypercall trampoline”, which is a small fastcall routine which basically executes a VMCALL and RET.

This routine is stored in the “hypercall page”. This page contains at this day five trampolines, and is given by Hyper-V at its startup to winload.efi, which will copy it in the VTL0 and VTL1 address spaces. The main difference between these five trampolines is that the first one just does a VMCALL/RET, but the four next ones (they are defined contiguously) store RCX into RAX, and enforce a fixed value into RCX. The second and the third enforce RCX to 0x11, and the next ones to 0x12.

These four trampolines are actually used by the different VTLs. Each kernel may “ask” Hyper-V for the 0xD0002 Virtual Processor Register value (internal Hyper-V values which can be queried or set using their identifier) using a dedicated hypercall, which will return two offsets. These offsets are hypercall page related, and will be used by the kernel to call the correct trampolines. Actually, VTL1 and VTL0 use the 0x11 trampoline to communicate with each other, and the VTL1 uses the 0x12trampoline to finalize its initialization.

The hypercall page’s content can be represented as:

VMCALL               <-- first trampoline
RET
MOV RAX, RCX         <-- second one (enforces 0x11)
MOV RCX, 0x11 
VMCALL
RET 
MOV RAX, RCX         <-- third (0x11)
MOV RCX, 0x11 
VMCALL 
RET 
MOV RAX, RCX         <-- fourth (0x12)
MOV RCX, 0x12 
VMCALL 
RET 
MOV RAX, RCX         <-- fifth (0x12)
MOV RCX, 0x12 
VMCALL 
RET 
db 0x00 
db 0x00 
db 0x00 
db 0x00 
db 0x00 
db 0x00 
... 
... 
db 0x00 

We therefore have five trampolines, at offsets 0x00, 0x04, 0x0F, 0x1D and 0x28. Note that their contents can be easily retrieved in Windows crashsdumps using WinDbg or within the Hyper-V binary (hvix64.exe/hvax64.exe for intel/amd) inner code.

Remark: several hypercalls may specify a data size in the 12 least significant bits of the RCX most significant DWORD. These sizes are not the data length in bytes, but are related to the current call, and may indicate entries count, etc.

For a hypercall example, the VTL1’s ShvlProtectContiguousPages hypercall (12) parameter is a structure which follows this scheme:

typedef struct _param {
    ULONGLONG infinite; // always 0xFFFFFFFFFFFFFFFF
    ULONG protection_asked; // 0xD EXECUTE, 0xF WRITE
    ULONG zero_value; // always 0 
    ULONGLONG pfns[]; // entries count is set in the hypercall number
} param;

In order to tell Hyper-V the pfns parameter size, RCX high DWORD must contain its elements count. For only one entry and a FAST hypercall, RCX value will therefore have to be 0x10010000C.

Secure kernel hypercalls

The two VTL are able to perform multiple hypercalls in order to communicate with Hyper-V. They may perform the same hypercalls, but Hyper-V will refuse several hypercall if they are called from VTL0. The two VTL also use one dedicated hypercall to communicate with each other. This is summarized in Figure 1 below.

Figure1
Figure 1: Hypercalls categories

Let's first describe the “VTL1 to Hyper-V” hypercalls (in green). We will then cover the 0x11 hypercall later.

The VTL1 can use three hypercall trampolines:

  • ShvlpHypercallCodePage, which is equivalent of the NTOS HvlpHypercallCodePage one (offset 0), and points over the first trampoline;
  • ShvlpVtlReturn, which enforces RCX to 0x11 and allows for VTL0 and VTL1 communications;
  • ShvlpVtlCall, which enforces RCX to 0x12 and is only used during the VTL1 initialization.

The last two are retrieved using the 0xD0002 virtual register (through the 24 least significant bits of the ShvlpGetVpRegister return value, each offset is 12 bits length). These two offsets point towards the 0x11 and the 0x12 trampolines.

By the way, the VTL0 NTOS kernel gets its HvlpVsmVtlCallCodeVa value (the hypercall trampoline used for VTL0 to VTL1 communications) using the same process, but gets inverted results. This is why we believe the reason for using these trampolines is that any VM gets the same hypercall page from Hyper-V, and may ask for the virtual register’s value. Hyper-V will return different offsets depending on which VTL the VM is.

The following tables give the possible VTL1 hypercalls, for each trampoline:

ShvlpVtlCall

Table 1: ShvlpVtlCall hypercalls
Hypercall number Caller - usage
0x12 ShvlInitSystem – end of VTL1 initialization?

ShvlpVtlReturn (VTL0 returns/calls)

Table 2: ShvlpVtlReturn hypercalls
Hypercall number Caller - usage
0x11 SkCallNormalMode/SkpPrepareForReturnToNormalMode – returns to NTOS / calls NTOS

ShvlpHypercallCodePage (HyperV)

Remark: We wrote an (H) when realizing an hypothesis.

Table 3: ShvlpHypercallCodePage hypercalls
Hypercall number Caller - usage
0x2 ShvlFlushEntireTb, etc. - Translation buffer flushs
0x3 Translation buffer flushs
0xB SkpgPatchGuardCallbackRoutine – (H) HyperGuard delayed routines registering
0x15
0xC ShvlProtectContiguousPages – Memory protection modification
0xD ShvlInitSystem – Called just before ShvlEnableVpVtl, seems to send several settings to HV
0xF ShvlEnableVpVtl – Sends settings to HV, and notably the ShvlpVtl1Entry function pointer
0x50 ShvlGetVpRegister – Gets a virtual processor (VP) register
0x51 ShvlSetVpRegister – Sets a virtual processor register
0x52 SkpgTranslateVa – (H) VTL0 memory access by HyperGuard
0x86 ShvlPrepareForHibernate
0x87 ShvlNotifyRootCrashDump
0x94 BugCheck
0x8E LiveDumpCollect
0x97 SkpGetPageList
0xAE ShvlSetGpaPageAttributes – 1607 build: changes a GPA attributes, seems to only been used on VTL1 memory

VTL0 to VTL1 transition

Almost all of NTOS “Vsl” prefixed functions end up in VslpEnterIumSecureMode, with a Secure Service Call Number (SSCN). This functions calls HvlSwitchToVsmVtl1, which uses the HvlpVsmVtlCallVa hypercall trampoline (regular hyper-V hypercalls use HvcallCodeVa trampoline). The SSCN is then copied into RAX, and RCX value is set to 0x11.

Hyper-V dispatches the 0x11 hypercall into the securekernel.exe function SkpReturnFromNormalMode, which then calls IumInvokeSecureService (actually we are not sure IumInvokeSecureService is called directly or not, we think that SkpReturnFromNormalMode must be called first in order to make IumInvokeSecureService returning to VTL0 after secure service call completion). IumInvokeSecureService is mostly a big switch/case block, which handles all possible SSCNs.

Finally, the SkCallNormalMode is called, which ends up on SkpPrepareForReturnToNormalMode. Actually, the secure kernel’s NTOS calls can be considered as “fake returns” to VTL0, since they consist in 0x11 hypercalls too.

We have identified all the possible SSCNs from VTL0 in the array below. For each one, we only specified the called functions as their name is generally self-explanatory. The corresponding parameters must be however retrieved manually by reverse engineering the VTL0 callers or the VTL1 callees.

Table 4: VTL0 to VTL1 SSCNs
SSCN Called function
1 SkmmInitializeUserSharedData / SkInitSystem
2 SkeStartProcessor
3 SkpsRegisterSystemDll
4 InterlockedCompareExchange(IumSystemProcessRegistered) / PsIumSystemProcess manipulation
5 SkmmCreateProcessAddressSpace/SkobCreateHandle
6 SkeInitializeProcess
7 IumCreateThread
8 SkiTerminateAllThreads
9 IumTerminateThread
10 SkeRundownProcess
11 SkpsIsProcessDebuggingEnabled / SkmmDisableProcessMemoryProtection
12 Unknown
13 & 14 SkmmMapMdl / IumpGetSetContext
15 SkeReferenceProcessByHandle / SkmmMapDataTransfer / SkpEncryptWithTrustletKey
16 SkeReferenceProcessByHandle / Unknown
17 SkRetrieveMailbox
18 SkIstTrustletRunning
19 SkmmCreateSecureAllocation
20 SkmmMapDataTransfer / SkmiFillSecureAllocation
21 SkmmConvertSecureAllocationToCatalog
22 SkmmCreateSecureImageSection
23 SkmmFinializeSecureImageHash
24 SkmmFinishSecureImageValidation
25 SkmmPrepareImageRelocations
26 SkmmRelocateImage
27 SkobCloseHandleEx
28 SkmmValidateDynamicCodePages
29 SkmmTransferImageVersionResource
30 EntropyProvideData / BCryptGenRandom
31 SkpEncryptHiberData / SkpSetHiberCrashState
32 SkpSetHiberCrashState / SkpgHibernateActive = 0 / SkFinalizePageEncryption
33 SkmmConfigureDynamicMemory
34 IumConnectSwInterrupt
35 Unknown = 0x3000
36 SkLiveDumpStart
37 SkpLiveDumpContext / Unknown
38 SkLiveDumpSetupBuffer
39 SkLiveDumpFinalize
40 SkpLiveDumpFreeContext / SkpReleaseLiveDumpLock
41 SkNotifyPowerState
42 IumDispatchQueryProfileInformation
192 SkeReferenceProcessByHandle
193 SkmmValidateSecureImagePages
208 SkmmInitSystem / IumpInitializeSystem
209 SkpWorkItemList / Unknown
210 SkmiReleaseUnprivilegedPagesInRange / SmiReserveNtAddressRange
211 SkmmApplyDynamicRelocations
212 SkEtwEnableCallback
224 SkiAttachProcess / SkmiFlushAddressRange
225 SkmmFastFlushRangeList
226 SkmmSlowFlushRangeList
227 SkmmRemoveProtectedPage
228 SkmmCopyProtectedPage
229 SkmmMakeProtectedPageWritable
230 SkmmMakeProtectedPageExecutable
231 (H) Gets Skmi flags
232 SkhalEfiInvokeRuntimeService
233 SkLiveDumpCOllect
234 SkmmRegisterFailureLog
235 SkPrepareForHibernate
236 SkPrepareForCrashDump
237 SkhalpEfiRuntimeInitialize / SkhalpReportBugCheckInProgress
238 & 240 Returns an error code
241 SkKsrCall
Otherwise SkeBugCheckEx(0x121, 0xFFFFFFFFC000001C, , 0, 0)

As you can see, several called functions are unknown. This is because they do not perform obvious calls, and we did not spent a lot of time on analyzing what they manipulate.

Conclusion

This article has quickly described how the Virtualization Based Security VTL0-VTL1 kernels communications actually work.

If you want more Hyper-V related information, you can also read these two articles:

In a near future, we plan to publish a third article on VBS which will focus on the HVCI internals, and especially on the W^X VTL0 kernel protection.

Disclaimer

These findings have almost all been retrieved by static analysis using IDA Pro. We apologize if they contain mistakes (actually they probably do), and ask the readers to take this “as it is”. Any helpful remark or criticism is welcome, just email us at etudes{at}amossys.fr!