Reversing a macOS Kernel Extension

In my last post I covered the basics of kernel debugging in macOS. In this post we will put some of that to use and work through the process of reversing a macOS kernel module.

As I said in my last post, in macOS there is a kernel module named “Don’t Steal Mac OS X” (DSMOS) which registers a function with the Mach-O loader to unpack binaries that have the SG_PROTECTED_VERSION_1 flag set on their __TEXT segment. Finder, Dock, and loginwindow are a few examples of binaries that have this flag set. My goal for this post is to simply work through the kernel module with the intent of discovering its functionality and use it as an opportunity to learn a bit about kernel debugging.

If you’d like to follow along I pulled the DSMOS module off of a laptop running macOS Sierra Beta (16A286a). Based on cursory looks at a couple copies from different versions of macOS it hasn’t changed much recently so you should be able to follow along with a copy from Mac OS X 10.11 or macOS Sierra. As you’ll see in the screenshots, I used IDA Pro for this reversing however using a program like Hopper would be fine as well.

First Look

At a glance, the DSMOS kernel module is fairly simple in terms of number of functions. It has 25 functions of which we only really care about 6. Most of the functions we don’t care about are constructors or destructors. Admittedly I haven’t taken the time to understand constructors and destructors used by a kernel module sp will be skipping them in this post.

Typically when I first look at a binary I start by looking at the strings in the binary.

As seen in Figure 1, there really aren’t that many strings and they aren’t all that exciting. The most interesting is probably the string “AppleSMC” which is an indicator that this module interacts with the System Management Controller.

Given that there are so few functions in this binary my approach was to simply go through each of them, have a quick look at the control flow graph (CFG) for a rough estimate of complexity, and put the function either on the “care” or “don’t care” list. Doing this I ended up with 9 functions of interest (see Table 1).

Table 1: Potentially interesting function addresses and associated names.

With these functions as starting points, the next step is to start working through them. At this point our goal is identify what each functionality each provides.

Registering an IOService Notification Handler

The relevant block of code from sub_A9E is shown in Figure 2. In words, this function first retrieves a matching dictionary for the AppleSMC service then installs a notification handler that is called when IOKit detects a service with the class name AppleSMC has been registered. In the call to IOService::addNotification() shown in Figure 2 the first argument is the address of the hanlder to be called. This handler is labelled as notificationHanlder in Figure 2 and not listed in Table 1 (it was a false negative); its located at address 00000B1A with a default name in IDA of sub_B1A. sub_B1A isn’t all that interesting, all it does is wrap sub_B2A dropping some arguments in the process.

The Notification Handler

When an IOService registers the AppleSMC class the code in sub_B2A will be notified. This function begins by calling OSMetaClassBase::safeMetaCast() to cast the incoming service into an AppleSMC service. Note that Apple’s documentation states that developers should not call OSMetaClassBase methods directly and should instead use provided macros. In this case, the call safeMetaCast() was likely generated by using the OSDynamicCast macro which Apple lists as a valid macro to be used by developers.

The next block in sub_B2A, shown in Figure 3, is where things actually start.

Since C++ is horribly annoying to reverse due to all the indirect calls, rather than figuring out what method is represented by rax+850h I turned to Google. Searching for OSK0 and OSK1 turns up an article posted by Amit Singh. In it he talks briefly about an older version of the DSMOS kernel extenion and also provides code that uses the OSK0 and OSK1 strings to query the SMC for two keys. Once these keys have been acquired the kernel extension then computes a SHA-256 hash and compares to a value stored in memory. If this comparison fails, an error is printed (not shown).

If the hashes match then we skip to the block shown in Figure 4.

The first part of this basic block takes the address of byte_3AA4 and our keys returned from the SMC then calls sub_1616. If you look at sub_1616 you’ll see it contains a couple loops and a bunch of byte manipulation I didn’t want to reverse. Looking at where byte_3AA4 is used you’ll see it is used in two places: here in sub_B2A and in sub_D30. Let’s wait a bit to see how it is used before figuring out how it is generated.

After the call to sub_1616 we have two AES decryption keys set. The first key is the value returned from the SMC when queried with OSK0 and the second key is the value returned when OSK1 is used to query the SMC. Finally, we see a global variable named initialized set to 1 and a call to dsmos_page_transform_hook with the address of sub_D30 as a parameter.

void
dsmos_page_transform_hook(dsmos_page_transform_hook_t hook)
{

    printf("DSMOS has arrived\n");
    /* set the hook now - new callers will run with it */
    dsmos_hook = hook;
}

Listing 1: Source code for dsmos_page_transform_hook from XNU source

Searching for dsmos_page_transform_hook in the XNU source we find the code in Listing 1. This is a pretty simple funciton that simply sets the value of dsmos_hook to the provided function address.

Usage of dsmos_hook in XNU

At this point we will take step briefly away from IDA and kernel extension turning our attention to the XNU source. For this work I used the source of XNU 3248.60.10 which is the version used by Mac OS X 10.11.6. If you haven’t done so already, you can download the source from http://opensource.apple.com/release/os-x-10116/.

As we saw, dsmos_page_transform_hook simply set the value of dsmos_hook. Continuing from here we find that dsmos_hook is only used in dsmos_page_transform_hook as we saw and in dsmos_page_transform (Listing 2).

int
dsmos_page_transform(const void* from, void *to, unsigned long long src_offset, void *ops)
{
    static boolean_t first_wait = TRUE;

    if (dsmos_hook == NULL) {
        if (first_wait) {
            first_wait = FALSE;
            printf("Waiting for DSMOS...\n");
        }   
        return KERN_ABORTED;
    }
    return (*dsmos_hook) (from, to, src_offset, ops);
}

Listing 2: Usage of dsmos_hook by XNU

After ensuring dsmos_hook the code in LIsting 2 just calls the hook with the parameters passed to dsmos_page_transform. This approach allows Apple some flexibility and opens up the opportunity to have multiple hooks in the future. Once again searching the XNU source, we see that the only use of dsmos_page_transform is in a function called unprotect_dsmos_segment. I have not included the source of unprotect_dsmos_segment since it is a bit longer and also not very exciting. The most interesting part about it is that it checks to see that the segment is long enough before attempting to call dsmos_page_transform on it.

Continuing along, unprotect_dsmos_segment is only called by load_segment. load_segment is a much larger function and is not shown in its entirety but the relevant portion is shown in Listing 3.

if (scp->flags & SG_PROTECTED_VERSION_1) {
    ret = unprotect_dsmos_segment(file_start,
                          file_end - file_start,
                          vp,
                          pager_offset,
                          map,
                          vm_start,
                          vm_end - vm_start);
    if (ret != LOAD_SUCCESS) {
        return ret;
    }
}

Listing 3: Call to unprotect_dsmos_segment from load_segment

The interesting part of the code in Listing 3 is that unprotect_dsmos_segment is only called on segments with the SG_PROTECTED_VERSION_1 flag set. As mentioned earlier, macOS only includes a few binaries with this flag set such as Finder, Dock, and loginwindow.

The Hook Implementation

At this point we know that the DSMOS kernel extension queries the SMC for a pair of keys, initializes some AES decryption contexts and global variables, then installs a hook by calling dsmos_page_transform_hook. We also know that the Mach-O loader in the kernel will call this hook when it finds a segment with the SG_PROTECTED_VERSION_1 flag set. The next question then is: what does the hook installed by the DSMOS kernel extension actually do?

Prior to the code shown in Figure 5 is the function prologue and setting of a stack cookie; after the code is the checking of the stack cookie and function epilogue. The code shown starts by checking to see if the initialization flag is set. This is the same initialization flag we saw being set in sub_B2A (see Figure 4). If this flag is not set the function exits, otherwise it enters a series of checks to identify which kernel is calling the hook. Searching the XNU source you can find the constant 0x2e69cf40 in the implementation of unprotect_dsmos_segment as shown in Listing 4.

                struct pager_crypt_info crypt_info;
                crypt_info.page_decrypt = dsmos_page_transform;
                crypt_info.crypt_ops = NULL;
                crypt_info.crypt_end = NULL;
#pragma unused(vp, macho_offset)
                crypt_info.crypt_ops = (void *)0x2e69cf40;
                vm_map_offset_t crypto_backing_offset;
                crypto_backing_offset = -1; /* i.e. use map entry's offset */

Listing 4: XNU setting value of crypt_info.crypt_ops

As Figure 5 shows, there are three basic cases implemented in the hook: no protection, an old kernel, and a new kernel. The basic block responsible for each case is labelled accordingly. I did not try to figure out which kernels mapped to which version however if you read the article by Amit Singh you’ll notice that he talks about the method where each half of the page is encrypted with one of the SMC keys. In our kernel extension this corresponds to the old_kernel basic block.

The method currently in use by Apple starts at the basic block labelled as new_kernel in Figure 5. In it we see an 8 byte buffer is zeroed then a call is made to a function I’ve called unprotect (named sub_1734 by IDA originally). Looking at the parameters to unprotect we see it takes the global buffer byte_3AA4 we saw earlier, the source buffer containing the page to be transformed, and the destination buffer to store the transformed page in among other parameters.

This is the point in our reversing where things become very tedious since Apple has moved away from using AES to encrypt the pages to a custom method composed of many byte operations (e.g. shift left/right, logical/exclusive or).

Unprotecting a Protected Page

To properly set expectations, due to the tedious nature of this protection mechanism and me being somewhat satisfied with what I’ve learned so far I did not go through the full exercise of reversing Apple’s “unprotect” method. Originally I had intended to write a program that would be able to apply the transform to a given binary but that program is only partially completed and does not work. So, with expectations set sufficiently low lets get a feel for the implementation and a couple ways of approaching it.

First lets step back briefly. Remember we saw the global variable byte_3AA4 being initialized in sub_B2A and that I had said the code for that was also incredibly tediuous? Well, thanks to the ability to dump memory from the kernel through the debugger we don’t need to reverse it at all. We just need to connect to a running kernel and ask it politely.

Dumping byte_3AA4 From a Running Kernel

If you are unclear about how to use the kernel debugger then check out my previous post. To get started, on your remote machine start the debugger by hitting the NMI keys (left command, right command, and power together) then connect to the debugger from your local machine. The following lldb sessions shows the steps all put together.

(lldb) kdp-remote 192.168.42.101
Version: Darwin Kernel Version 16.0.0: Fri Aug  5 19:25:15 PDT 2016; root:xnu-3789.1.24~6/DEVELOPMENT_X86_64; UUID=4F6F13D1-366B-3A79-AE9C-4
4484E7FAB18; stext=0xffffff802b000000
Kernel UUID: 4F6F13D1-366B-3A79-AE9C-44484E7FAB18
Load Address: 0xffffff802b000000

...

Process 1 stopped
* thread #2: tid = 0x00b8, 0xffffff802b39a3de kernel.development`Debugger [inlined] hw_atomic_sub(delt=1) at locks.c:1513, name = '0xffffff8
037046ee0', queue = '0x0', stop reason = signal SIGSTOP
    frame #0: 0xffffff802b39a3de kernel.development`Debugger [inlined] hw_atomic_sub(delt=1) at locks.c:1513 [opt]
(lldb) showallkexts
OverflowError: long too big to convert
UUID                                 kmod_info            address              size                  id  refs TEXT exec            size     
                         version name                          

...

B97F871A-44FD-3EA4-BC46-8FD682118C79 0xffffff7fadf449a0   0xffffff7fadf41000   0x5000               130     0 0xffffff7fadf41000   0x5000   
                           7.0.0 com.apple.Dont_Steal_Mac_OS_X 

...

(lldb) memory read --force --binary --outfile byte_3AA4.bin 0xffffff7fadf44aa4 0xffffff7fadf44aa4+4172 
4172 bytes written to '/Users/dean/Sites/lightbulbone.github.io/byte_3AA4.bin'

We start out by connecting to the remote host using the kdp-remote command. Once everything has loaded we can get the address of the DSMOS kernel extension in memory using the showallkexts command. In my case the base address is 0xffffff7fadf41000. We then read the memory at address 0xffffff7fadf44aa4 which is the extension base address plus the offset of 0x3aa4; we read 4172 bytes since that is the size of the buffer.

If you were writing a program to unprotect binaries you could use this extracted binary blob rather than trying to reverse the initialization algorithm.

Emulating the Unprotect Algorithm

Due to the tedious nature of the algorithm used to “unprotect” a page I decided to try using the Unicorn Engine to emulate it. This effort largely failed because it meant I would have to set up memory in Unicorn the same way as it is in the kernel extension and, as I said, the motivation wasn’t quite there. As far as I know this is possible however it to can be rather tedious; especially in cases where the algorithm isn’t as self-contained as in this case. Using an IDA plugin such as sk3wldbg may help however I was not aware of it at the time.

Reversing the Unprotect Algorithm

In the end I just sat down and started working through the algorithm in IDA. I did begin to write a program to unprotect before my motivation to work through the tedious code fell through the floor. For me, looking at DSMOS was an opportunity to learn what a kernel module I’ve known about for many years and become more familiar with the macOS kernel. That being said a few things a worth pointing out.

In Figure 6 a portion of sub_1734 is shown. In it we see the first eight bytes of the from pointer (stored in r14) being used to build a value to pass to sub_125A.

And, in Figure 7 we see part of sub_125A. In this part we see the first two iterations of an unrolled loop. The point of Figures 6 and 7 is to show some common constructs that come up when reversing code. If you’re not familiar with these constructs it may help to write some code yourself and then analyze the binary after compilation.

Summary

The intent of this post was to reverse engineer the DSMOS kernel extension in macOS. The goal was to understand what functionality the DSMOS extension provided to the kernel and to become more familiar with the XNU kernel. We also touched on IOKit briefly as well as a possible application of the Unicorn engine.

If you have any questions or comments, please feel to reach out to me on Twitter @lightbulbone.