Why System Call Filtering Fails Against Modern Exploits

Production container runtimes deploy secure computing mode filters as primary defense perimeters. Docker, containerd, and systemd load Berkeley Packet Filter programs via prctl(PR_SET_SECCOMP) to restrict raw kernel surface. Standard profiles block roughly 44 system calls out of more than 450 available on Linux x86_64. Security teams treat this barrier as absolute boundary isolation.

Engineers assume system call filtering reliably blocks malicious interactions between user space and kernel core. Reality breaks this assumption. Seccomp evaluates register state at kernel entry points. It cannot safely inspect pointers referencing user-space memory without generating race conditions. Attackers bypass filter logic through memory mutation, system call multiplexing, cross-architecture ABI trickery, and post-filter kernel memory corruption.

Understanding these failure modes exposes why seccomp cannot serve as single-point isolation mechanism. Systems defense requires evaluating register validation physics, kernel entry bottlenecks, and memory race windows.

How seccomp-BPF Enforces System Call Filtering at Kernel Entry

Linux implements seccomp-BPF inside kernel entry pipelines. When user process invokes SYSCALL instruction on x86_64, CPU transitions privilege rings and executes entry_SYSCALL_64. Control transfers through do_syscall_64() in arch/x86/entry/common.c. If thread flags set _TIF_SECCOMP, kernel invokes __secure_computing().

struct seccomp_data {
    int   nr;
    __u32 arch;
    __u64 instruction_pointer;
    __u64 args[6];
};

This structure feeds directly into classic BPF virtual machine. Filter program executes sequential evaluation steps:

1. Validate architecture token against host target architecture (AUDIT_ARCH_X86_64).
2. Read system call number from offset zero (offsetof(struct seccomp_data, nr)).
3. Evaluate argument registers held inside args[0] through args[5].
4. Return 32-bit execution verdict: SECCOMP_RET_ALLOW, SECCOMP_RET_ERRNO, SECCOMP_RET_TRACE, or SECCOMP_RET_KILL_PROCESS.

Context switches, privilege boundaries, and filter validation carry quantifiable performance costs. Reading registers and switching state slows execution paths compared with unintercepted operations. Measuring these entry transitions reveals why high-throughput engines isolate filter checks from hot paths; detailed benchmarks appear in System Call vs vDSO: What Entering the Kernel Costs.

The architectural design constraint: struct seccomp_data contains only direct register values. x86_64 ABI passes first six system call arguments inside rdi, rsi, rdx, r10, r8, and r9. Scalar values like file descriptors, bitwise flags, and error numbers reside completely within these registers. Pointers, however, transmit only 64-bit virtual memory addresses. Filter code sees memory addresses, not target memory contents.

The Pointer Dereference Blindspot and TOCTOU Memory Races

System calls accepting string buffers or nested structures pass user-space memory pointers. Examples include openat(int dfd, const char *filename, int flags, mode_t mode) and connect(int sockfd, const struct sockaddr *addr, socklen_t addrlen). Seccomp inspects pointer value in register rsi, not string stored at that address.

Filter cannot dereference user memory directly inside execution loop. Allowing cBPF interpreter to read arbitrary virtual addresses causes kernel page faults, deadlocks, and cache line invalidations inside security checks. Therefore, seccomp filters can only evaluate memory address itself, not payload.

This architecture creates exploitable Time-of-Check to Time-of-Use (TOCTOU) race condition when developers attempt user-space filtering with user notification (SECCOMP_RET_USER_NOTIF) or external monitoring threads:

Thread 1 (Worker)                 Thread 2 (Attacker)              Kernel Seccomp
-----------------                 -------------------              --------------
Calls openat(dfd, ptr, O_RDONLY)                                   Reads ptr
ptr points to "/etc/hosts"                                         Filter allows
                                  Overwrites ptr with "/etc/shadow"
                                                                   copy_from_user() runs
                                                                   Kernel opens /etc/shadow

Because memory is shared across process threads, Thread 2 writes new string bytes across buffer between seccomp evaluation and kernel copy_from_user() invocation in do_sys_openat2().

Exploitation reliability scales with memory bus contention and deliberate thread stalling. Attackers induce microsecond delays during kernel resolution loops by scheduling low-priority worker threads across CPU cores or generating deliberate page fault cascades. Using memory synchronization primitives, attackers widen TOCTOU window indefinitely; mechanics of thread suspension and memory timing are documented in How Page Fault Handling Wins userfaultfd Race Conditions.

// Demonstrating the pointer race structure
#define _GNU_SOURCE
#include 
#include 
#include 
#include 
#include 

static char target_path[64] = "/tmp/safe_file";
static volatile int stop_threads = 0;

void *racer_thread(void *arg) {
    while (!stop_threads) {
        strcpy(target_path, "/etc/shadow");
        strcpy(target_path, "/tmp/safe_file");
    }
    return NULL;
}

int main(void) {
    pthread_t tid;
    pthread_create(&tid, NULL, racer_thread, NULL);

    for (int i = 0; i < 50000; i++) {
        int fd = openat(AT_FDCWD, target_path, O_RDONLY);
        if (fd >= 0) {
            close(fd);
        }
    }

    stop_threads = 1;
    pthread_join(tid, NULL);
    return 0;
}

This structural blindspot prevents system call filtering from enforcing path-based access controls reliably.

Multiplexed System Calls and Argument Smuggling

System calls acting as multiplexers present massive evasion surface. Monolithic kernels combine complex subsystems behind single system call dispatch routines.

Primary multiplexers include:

• ioctl(int fd, unsigned long request, ...): Accepts arbitrary command integers and pointer payloads. Single allowed file descriptor permits driver-level sub-commands.
• prctl(int option, unsigned long arg2, ...): Handles process names, memory tags, capability sets, speculative execution mitigations.
• bpf(int cmd, union bpf_attr *uattr, unsigned int size): Loads programs, creates maps, links probes. Argument payload resides entirely inside pointer to union bpf_attr.
• recvmsg / sendmsg: Passes ancillary data buffers containing control messages (SCM_RIGHTS), passing live file descriptors between processes without triggering openat checks.

Consider ioctl. Filter designer intends to permit terminal operations:

BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_ioctl, 0, 5),
BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, args[1])),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, TCGETS, 0, 1),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),

Filter allows TCGETS (0x5401). But device driver handling target file descriptor decodes args[1] through loose integer masks or custom switch blocks. On Linux TTY layer, TIOCSTI allows pushing characters into input queue of terminal, enabling unprivileged process to inject commands into parent shell process.

Another critical evasion occurs through modern asynchronous execution frameworks. io_uring_enter(unsigned int fd, unsigned int to_submit, unsigned int min_complete, unsigned int flags, sigset_t *sig) bypasses standard per-operation system call checks. An application calls io_uring_setup() once. Subsequent read, write, open, and socket operations dispatch via submission queue rings located inside shared user-kernel memory.

Worker kernel threads execute requested operations without passing through standard user-to-kernel SYSCALL vector for each operation. Unless seccomp policy explicitly denies io_uring_setup, all system call filtering assumptions collapse.

Architecture Mismatches and Syscall Number Collisions

x86_64 platforms maintain backwards compatibility through multiple application binary interfaces (ABIs). Kernel supports standard 64-bit ABI, 32-bit x86 ABI (IA32 emulation), and x32 ABI (64-bit pointers with 32-bit data types).

Each ABI uses separate system call number dispatch table:

• __NR_open on x86_64: 2
• __NR_open on i386 (x86_32): 5
• __NR_openat on x86_64: 257
• __NR_openat on ARM64: 56

Flawed seccomp filter configuration fails to validate incoming architecture token before evaluating call index:

// Vulnerable BPF: checks call number without checking architecture
BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, 2, 0, 1), // Block __NR_open (x86_64)
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),

Target process switches execution mode using 32-bit code segment selector via int 0x80 or far call transition. System call number 5 passes check, executes 32-bit open implementation, and evades filter entirely.

Second architectural flaw exists inside x32 ABI implementation. Linux kernel reserves bit 30 (__X32_SYSCALL_BIT, 0x40000000) to distinguish x32 requests from standard x86_64 system calls. If filter inspects data->nr without masking this bit:

// User invokes: syscall(2 | 0x40000000, ...)

Filter compares call value against numeric 2. Evaluation returns false. Filter allows call. Kernel clears bit 30 in dispatch table, invoking intended system call handler despite security filter block. Hardened configurations must assert explicit equality against target architecture:

// Correct validation sequence
BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, arch)),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, AUDIT_ARCH_X86_64, 1, 0),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),

Post-Filter Escapes and Kernel Privilege Escalation

Seccomp successfully blocking dangerous system calls does not prevent container escapes if permitted system calls expose vulnerable kernel execution paths. Production container profiles keep broad system call sets open to maintain standard runtime capability:

• epoll_create1, epoll_ctl, epoll_wait
• mmap, mprotect, madvise
• clone, clone3, unshare
• keyctl
• futex

Kernel vulnerabilities exist precisely inside memory management, synchronization, and object allocation code paths backing permitted calls. A memory corruption vulnerability inside madvise() or pipe() handler executes entirely within bounds of allowed seccomp filter.

Once attacker achieves arbitrary kernel read/write primitives via bug inside permitted call, attacker overwrites thread credentials (struct cred). Overwriting UID, GID, and capability bitmasks neutralizes Linux security constraints instantly. Memory structure corruption methods overwrite kernel credential objects without triggering system call violations; exploitation mechanics are analyzed in How Linux Dirty Cred Exploits Bypass Kernel Mitigations.

After corrupting memory, attacker locates struct task_struct of running thread in kernel memory. Field task_struct->thread_info.flags holds _TIF_SECCOMP bit. Field task_struct->seccomp.filter holds pointer to root BPF program. Attacker writes zero to bitmask and clears filter pointer. Seccomp terminates silently. All subsequent system call invocations execute without interception.

Designing Hardened seccomp Profiles That Resist Evasion

Building resilient system call filtering requires defensive engineering that assumes active adversarial evasion. Sandboxes cannot rely on default permissive masks.

Enforce Strict Default-Deny Policies

Default profiles maintain allow-lists with extensive exclusions. Robust environments invert this logic. Block everything by default. Use SECCOMP_RET_KILL_PROCESS as fallback, terminating entire application instance when violation occurs:

#include 
#include 
#include 
#include 
#include 

int apply_strict_seccomp(void) {
    struct sock_filter filter[] = {
        // Step 1: Validate system architecture
        BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, arch)),
        BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, AUDIT_ARCH_X86_64, 1, 0),
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),

        // Step 2: Load system call number
        BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)),

        // Step 3: Reject x32 bit manipulation attempts
        BPF_JUMP(BPF_JMP | BPF_JGE | BPF_K, 0x40000000, 0, 1),
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),

        // Step 4: Allow specific required primitives (e.g., write, exit_group)
        BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_write, 0, 1),
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),

        BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_exit_group, 0, 1),
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),

        // Step 5: Default action kills process
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),
    };

    struct sock_fprog prog = {
        .len = (unsigned short)(sizeof(filter) / sizeof(filter[0])),
        .filter = filter,
    };

    if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0)) {
        return -1;
    }
    return prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &prog);
}

Eliminate Multiplexers and Dangerous ABIs

Eliminate multiplexer exposure completely inside untrusted execution profiles:

1. Block io_uring_setup, io_uring_enter, and io_uring_register. Force operations to utilize conventional system calls subject to seccomp evaluation.
2. Block 32-bit execution entirely by disabling CONFIG_IA32_EMULATION and CONFIG_X86_X32 inside custom production kernels. When multi-arch support cannot be compiled out, seccomp policy must terminate processes requesting foreign architectures.
3. Deny bpf() and keyctl() system calls. Unprivileged access to kernel tracing, eBPF maps, or key storage exposes wide kernel surface.

Combine seccomp with Landlock LSM

Because seccomp cannot safely inspect paths, delegate filesystem and networking access controls to Landlock. Landlock enforces path restrictions within kernel inode resolution pipelines (vfs_open), executing past user memory dereference phases.

Combining seccomp for scalar system call reduction and Landlock for path resolution stops TOCTOU attacks. Attackers winning race conditions against path pointers collide with kernel-enforced inode restrictions inside VFS layer, preventing unauthorized descriptor creation.

// Example: Pair seccomp with Landlock ruleset
// Seccomp handles execution surface:
//   - Disallow clone3, io_uring, unshare, ptrace
// Landlock handles path surface:
//   - Restrict write operations exclusively to /tmp/app_scratch

Conclusion

System call filtering via seccomp-BPF operates as execution gatekeeper, not comprehensive application sandbox. Its operational model inspects CPU registers at kernel boundary transitions. The moment security logic attempts to validate user-space memory pointers, TOCTOU race conditions and multithreaded mutation invalidate inspection guarantees.

Relying on seccomp alone creates fragile systems. Complex system calls multiplex wide driver logic behind scalar identifiers. Asynchronous interfaces like io_uring bypass per-call interception vectors completely. Permitted call sets provide sufficient foothold for heap corruption exploits targeting underlying kernel structures.

Resilient system hardening requires multi-layer architectures:

• Enforce default-kill seccomp profiles rejecting unverified ABIs.
• Disable multiplexed interfaces and asynchronous submission queues inside untrusted domains.
• Combine seccomp register validation with kernel LSMs like Landlock to isolate filesystem and network paths.
• Remove obsolete execution modes at kernel configuration level.

Enforcing security boundaries requires placing controls at memory layers that match the data being protected. Register filters protect system call entry points; kernel object layers must protect memory.

Measured on our own hardware: What Does Entering the Kernel Cost? A System Call vs the vDSO

On this virtual server, how much slower is reading the monotonic clock through a real system call than through the vDSO, and what does the cheapest possible system call cost?

We ran it. The numbers below come from a program executed on the server hosting this site on 2026-10-25 — an AMD EPYC 9354P 32-Core Processor with 8 cores visible, 31.3 GB of memory, Linux 6.8.0-147-generic.

Metric	Value
vdso speedup factor	21.96
iterations	2000000
repeats	5
syscall clock gettime ns	600.9
syscall getppid ns	490.7
vdso clock gettime ns	27.4

This is a shared virtual server, not an isolated test rig, so treat the absolute figures as indicative and the ratio between the two cases as the finding. The full method, the machine specification, and the complete source code are on the What Does Entering the Kernel Cost? A System Call vs the vDSO benchmark page, so you can check the method or run it yourself.

References

1. Linux Seccomp Documentation: https://docs.kernel.org/userspace-api/seccomp_filter.html
2. The Linux Programmer's Manual: seccomp(2): https://man7.org/linux/man-pages/man2/seccomp.2.html
3. Linux Kernel Source Tree (v6.10): kernel/seccomp.c: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/seccomp.c