2017-4-14 Tick (FPU) Tock (IRQ)

random About

Tick (FPU) Tock (IRQ)

Apr 13, 2017

Introduced with iOS9, the kernel integrity protection aka watchtower aka KPP, posed a new
problem to arm64 jailbreaks. Effectively, it was observed that after patching the kernel code
- customary in older jailbreaks - the device would silently panic after a while. It soon
became obvious that some thing was checking the kernel code. That thing was outside the
kernel, and it became evident pretty quickly that there was a kind of hypervisor at work.

To this day, I’m still surprised nobody did a write-up on the KPP – at least none that I know
of. So, I will try to explain how the hypervision works. Specific implementation details will
maybe come at a later time, provided I have the time for it. Now, without further ado, here
we go…

Part 1 (the setup)

The KPP lies in a Mach-O executable file appended right after the compressed kernel
chunk, inside the kernelcache img4. The iBoot carves out the KPP image, loads it at
0x4100000000 and runs it in EL3.

EL3
_start(monitor_boot_args *mba)

The structure received upon entry looks like this:

struct monitor_boot_args {
uint64_t version;
uint64_t virtBase;
uint64_t physBase;
uint64_t memSize;
struct kernel_boot_args *kernArgs;
uint64_t kernEntry;
uint64_t kernPhysBase;
uint64_t kernPhysSlide;
uint64_t kernVirtSlide;
};

https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html 1/7
2017-4-14 Tick (FPU) Tock (IRQ)

The KPP overwrites its own Mach-O header at 0x4100000000 with a trampoline which
calls _start(NULL) and installs two exception handlers, sync_handler and
irq_handler . Recall the AArch64 exception table:

VBAR_ELn Exception Type Description

+0x000 Synchronous Current EL with SP0
+0x080 IRQ/vIRQ  
+0x100 FIQ/vFIQ  
+0x180 SError/vSError  
+0x200 Synchronous Current EL with SPx
+0x280 IRQ/vIRQ  
+0x300 FIQ/vFIQ  
+0x380 SError/vSError  
+0x400 Synchronous Lower EL using AArch64
+0x480 IRQ/vIRQ  
+0x500 FIQ/vFIQ  
+0x580 SError/vSError  
+0x600 Synchronous Lower EL using AArch32
+0x680 IRQ/vIRQ  
+0x700 FIQ/vFIQ  
+0x780 SError/vSError  
ExceptionVector with location of the two handlers.

Next, it parses the kernel and its kexts (from __PRELINK_INFO):

save __TEXT, __DATA segments to map list

save __TEXT, __DATA::__const zones to hash list

Finally, if enabled – which always is – it sets (among others) the following registers:

CPACR_EL1 = 0x100000; // CPACR_EL1.FPEN=1, causes instructions in EL0 that use

CPTR_EL3 = 0x80000000; // CPTR_EL3.TCPAC=1, accesses to CPACR_EL1 will trap from
SCR_EL3 = 0x631; // SCR_EL3.IRQ=0, When executing at any Exception level,
// SCR_EL3.SMD=0, SMC instructions are ENABLED at EL1 and
// SCR_EL3.SIF=1, Secure state instruction fetches from N

EL1
Kernel starts executing in EL1:

https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html 2/7
2017-4-14 Tick (FPU) Tock (IRQ)

_start() => start_first_cpu() => arm_init():

=> cpu_machine_idle_init() => monitor_call(0x800)
=> machine_startup() => kernel_bootstrap() => kernel_bootstrap_thread
_start_cpu() => arm_init_cpu() => cpu_machine_idle_init() => monitor_call

monitor_call() will escalate to EL3 into hypervisor’s sync_handler :

EL3
sync_handler:

if (ESR_EL3 == 0x5E000011) { // ESR_EL3.EC==0x17 && ESR_EL3.IL==1 && ESR_EL3.ISS

switch (arg0) {
case 0x800: // called by cpu_machine_idle_init()
/* save kernel entrypoint */
return ok;
case 0x801: // called by kernel_bootstrap_thread()
if (enabled) {
if (locked) {
FAIL(4);
}
/* do lockdown:
* hash all regions from hash list
* initialize some vars
* save SCTLR_EL1, TCR_EL1, TTBR1_EL1, VBAR_EL1
*/
...
SCR_EL3.SMD=1;
locked = 1;
}
return OK;
case 0x802: // wtf is this shit?
FAIL(5);
}
}
(to be continued)

When something goes wrong, FAIL(code) sets a global variable violated and signals
the kernel by:

ESR_EL1 = 0xBF575400 | code // ESR_EL1.EC=0x2F, ESR_EL1.ISV=1, ESR_EL1.IS=0x5754

Code Meaning

https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html 3/7
2017-4-14 Tick (FPU) Tock (IRQ)

1 violation in frame
2 bad syscall
3 not locked
4 already locked
5 software request
6 invalid TTE/PTE
7 violation in mapping
8 violation in system register
Execution is then transferred down as SError back into kernel’s ExceptionTable:

SError => fleh_serror() => sleh_serror() => kernel_integrity_error_handler

Otherwise, if everything is fine and dandy, execution resumes in the kernel, right after
monitor_call() .

This was the setup phase, required in order to let the kernel set-up the write-once memory
locations. Next, onto the heartbeat phase…

Part 2 (the ticking)

Meanwhile, userland runs code. When a FPU instruction is executed,
CPACR_EL1.FPEN==1 causes a trap in kernel.

EL1
In the running kernel, fleh_synchronous() fleh_irq() fleh_fiq() and
fleh_serror() all eventually tail to

exception_return_dispatch() => check_user_asts() => MSR CPACR_EL1, X0

When CPACR_EL1 is hit, it is trapped by EL3, as per CPTR_EL3.TCPAC==1 and execution

is transferred to EL3 sync_handler .

EL3
This time, it means business.

sync_handler:

(continued)
else if (ESR_EL3 == 0x62340400) { // ESR_EL3.EC==0x18 && ESR_EL3.IL==1 && ESR_EL
if (violated) {
https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html 4/7
2017-4-14 Tick (FPU) Tock (IRQ)

FAIL(1);
}
if (!locked) {
FAIL(3);
}
if (!(++number_of_hits & watchtower_throttle)) {
if (!(++flip_flop & 1)) {
if (hash_is_ready) {
blake2b_final(&hash, digest);
if (memcmp(cur->digest, digest, 32)) {
FAIL(1);
}
cur = get_next_region();
if (!cur) {
cur = get_first_region();
}
cur_data_ptr = cur->base;
cur_data_left = cur->size;
blake2b_init(&hash);
hash_is_ready = 0;
} else {
chunk = min(cur_data_left, 128);
blake2b_update(&hash, cur_data_ptr, chunk);
cur_data_ptr += chunk;
cur_data_left -= chunk;
if (!cur_data_left) {
hash_is_ready = 1;
}
}
} else {
/* walk and check TTE/PTE
* verify map list
* check system registers SCTLR_EL1, TCR_EL1, TTBR1_EL1, VBAR_EL1
*/
...
}
}
ELR_EL3 += 4; // skip insn
CPTR_EL3 = 0; // CPTR_EL3.TCPAC=0, accesses to CPACR_EL1 will not tr
CPACR_EL1 = 0x300000; // CPACR_EL1.FPEN=3, does not cause any FPU instructio
SCR_EL3 = 0x6B3; // SCR_EL3.IRQ=1, When executing at any Exception leve
// SCR_EL3.SMD=1, SMC instructions are UNDEFINED at EL
// SCR_EL3.SIF=1, Secure state instruction fetches fro
return OK;
}

https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html 5/7
2017-4-14 Tick (FPU) Tock (IRQ)

As you can see, it skips a few beats once in a while ( watchtower_throttle ==4) to go
easy on CPU usage and/or battery. It also slowly but surely crawl over all protected areas in
a sequential manner.

If all checks pass, the hypervisor disables FPU trapping (allowing the FPU to finally
execute), enables IRQ to EL3 (to make sure it is hit again), resumes kernel right after
CPACR_EL1 hit and waits.

EL1
Kernel/userland runs happily. When the next IRQ fires, it is taken to hypervisor’s EL3 IRQ
handler.

EL3
irq_handler:

CPACR_EL1 = 0x100000; // CPACR_EL1.FPEN=1, causes instructions in EL0 that use

CPTR_EL3 = 0x80000000; // CPTR_EL3.TCPAC=1, accesses to CPACR_EL1 will trap from
SCR_EL3 = 0x431; // SCR_EL3.IRQ=0, When executing at any Exception level,
// SCR_EL3.SMD=0, SMC instructions are ENABLED at EL1 and
// SCR_EL3.SIF=0, Secure state instruction fetches from N

That is: reset IRQs back to EL1, re-enable FPU trapping, re-enable trap for CPACR_EL1
accesses.

These last 4 steps are then repeated forever.

***
In summary: KPP makes sure the FPU always trap and the trap cannot be disabled. When
FPU hits (tick) the kernel tries to disable the trapping but is immediately taken to KPP. KPP
then runs its checks, frees the FPU, but routes the IRQs to itself. As soon as any IRQ fires
(tock) it makes the FPU trap again and de-routes the IRQs.

This is the engine that keeps the hypervisor beating. If you patch away the trigger, that is,
CPACR_EL1 access, the FPU can’t execute. However, there is one catch. We can “steal”
away the CPACR_EL1 access to a separate trampoline:

1. undo patches
2. hit CPACR_EL1, hypervisor runs and restores execution right after our CPACR_EL1
3. redo patches
4. profit

This bypass was demo-ed by @qwertyoruiop in yalu102.

https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html 6/7
2017-4-14 Tick (FPU) Tock (IRQ)

random

xerub xerub An awesome description.

xerub

https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html 7/7