Why Transparent Huge Pages on a VPS Degrade Memory Latency

Every memory read on x86-64 hardware requires translating a virtual address into a physical address. When an application traverses a large working set, translation lookaside buffer (TLB) misses force the hardware memory management unit (MMU) to execute multi-level page table walks. On bare metal, switching from standard 4 KiB pages to 2 MiB transparent huge pages increases TLB reach by a factor of 512, eliminating translation overhead for sequential and clustered memory workloads.

However, enabling Transparent Huge Pages on a VPS often produces the opposite result. Instead of accelerating memory-bound workloads, virtualized instances frequently suffer from erratic tail latency and unpredicted execution pauses when cycling through large in-memory buffers.

The breakdown stems from how hardware virtualization handles memory mapping. A virtual machine does not own physical silicon. It operates inside an abstracted address space managed by a hypervisor. When guest-level page sizing does not align with host-level allocation strategies, the translation machinery encounters compounded lookup penalties, compaction stalls, and hypervisor traps that neutralize the advantages of 2 MiB mappings.

The Mechanics of Two-Dimensional Page Walks in KVM

On bare-metal Linux, a virtual address translation for a standard 4 KiB page requires walking four levels of page tables: the Page Map Level 4 (PML4), Page Directory Pointer Table (PDPT), Page Directory (PD), and Page Table (PT). A TLB miss requires four sequential memory dereferences before the hardware fetches the requested data.

Hardware-assisted virtualization changes this equation by introducing a second layer of translation. Under Intel VT-x with Extended Page Tables (EPT) or AMD-V with Nested Page Tables (NPT), the processor must translate two distinct address domains:

1. Guest Virtual Address (gVA) to Guest Physical Address (gPA), controlled by the guest kernel through the guest CR3 register.
2. Guest Physical Address (gPA) to Host Physical Address (hPA), controlled by the host hypervisor through the EPT/NPT base pointer (EPTP).

The hardware page walker cannot simply look up a gPA directly in physical memory. The guest page table pointers themselves are stored as guest physical addresses. To read the base of the guest PDPT, the CPU must first translate the gPA of the PML4 entry into an hPA using the host EPT.

Guest Level 4 (gPML4) -> Requires 4 EPT lookups to find hPA
Guest Level 3 (gPDPT) -> Requires 4 EPT lookups to find hPA
Guest Level 2 (gPD)   -> Requires 4 EPT lookups to find hPA
Guest Level 1 (gPT)   -> Requires 4 EPT lookups to find hPA
Target Data Frame     -> Requires 4 EPT lookups to find hPA

For each step of the 4-level guest translation, the hardware walker must complete up to 4 host-level translations. The formula for the maximum number of memory dereferences during a full two-dimensional TLB miss is:

$$\text{Dereferences} = (N_{\text{guest}} + 1) \times (N_{\text{host}} + 1) - 1$$

With 4 levels in the guest ($N_{\text{guest}} = 4$) and 4 levels in the host ($N_{\text{host}} = 4$), resolving a single memory reference requires up to:

$$(4 + 1) \times (4 + 1) - 1 = 24 \text{ memory accesses}$$

Modern processors employ an internal paging-structure cache to hold intermediate translation steps, mitigating this theoretical worst-case penalty. But when an application performs dependent random reads across a wide memory region, cache capacity evictions force the hardware back into the memory bus. If a server experiences 24 serialized memory accesses to resolve a single pointer dereference, application execution stalls completely.

How Transparent Huge Pages Collapse the Guest Translation Tree

Transparent Huge Pages (THP) attempt to solve this translation tax by collapsing the bottom layer of the paging tree. In standard x86-64 paging, a 2 MiB page is created by setting the Page Size (PS) bit in the Page Directory entry (the Page Middle Directory in Linux kernel terminology).

When the PS bit is active:

• The Page Middle Directory entry points directly to a contiguous 2 MiB physical frame.
• The Page Table (PT) level is bypassed entirely.
• The offset field within the virtual address expands from 12 bits (covering 4 KiB) to 21 bits (covering 2 MiB).

On bare metal, this reduces the guest walk from four levels to three. More importantly, it reconfigures how translation entries populate the CPU hardware caches. A typical modern server core features a 64-entry L1 data TLB (dTLB) for 4 KiB pages and a dedicated 32-entry L1 dTLB for 2 MiB pages. Behind that sits a unified L2 TLB (STLB) containing roughly 1,536 to 2,048 entries.

With standard 4 KiB pages, an entire 2,048-entry STLB covers only:

$$2048 \times 4\text{ KiB} = 8\text{ MiB}$$

If your active working set is a 1 GiB buffer, 8 MiB represents less than 1% of your data. A pointer chasing algorithm that accesses memory at random offsets will miss the TLB on almost every hop.

By contrast, if the same 1 GiB buffer is backed by 2 MiB huge pages, the entire range requires only 512 entries:

$$512 \times 2\text{ MiB} = 1\text{ GiB}$$

Because 512 entries fit within the 2,048-entry STLB, the address translation for every single byte of the 1 GiB buffer stays pinned in the CPU TLB. Hardware translation latency vanishes, leaving only the raw latency of the DRAM row access. Understanding cache layouts and alignment is critical for avoiding false sharing at the line level, as explained in our analysis of How Cache Line False Sharing Degrades App Performance. But in virtual machines, this 2 MiB mapping often fails to materialize as expected.

Why Transparent Huge Pages on a VPS Suffer From Asymmetric Mappings

The primary failure mode of Transparent Huge Pages on a VPS is architectural asymmetry between the guest kernel and the host hypervisor.

Inside your VPS, Linux sees a contiguous virtual address space and a virtual physical address space (gPA). When transparent_hugepage=always is set, the guest page fault handler allocates a 2 MiB compound page and maps it through a huge PMD entry. The guest kernel marks the memory as a huge page and operates under the assumption that translation overhead has been eliminated.

However, the hypervisor running on the physical host does not automatically mirror this structure. In most public cloud environments and multi-tenant VPS clusters, the hypervisor process (such as QEMU/KVM) allocates guest memory using standard anonymous mmap calls without dedicated hugetlbfs backings on the host.

Guest View:  [gVA] ---> [Guest PMD: 2 MiB Leaf] ---> [gPA (Contiguous 2 MiB)]
                              |
Host View:   [gPA] ---> [Host PML4] -> [Host PDPT] -> [Host PD] -> [Host PT] -> [hPA (Fragmented 4 KiB)]

When the host uses 4 KiB pages to back a guest's 2 MiB page:

1. The guest page walk collapses from 4 levels to 3 levels.
2. The host EPT page walk remains at 4 full levels.
3. Every 4 KiB segment inside the guest's "contiguous" 2 MiB page is scattered across arbitrary physical memory addresses on the host host machine.

Instead of reducing translation lookups to a minimal baseline, the hardware walker must execute a complete 4-level EPT walk for every 4 KiB boundary crossed within the guest's 2 MiB page. The formula for the two-dimensional walk drops from 24 accesses down to:

$$(3 + 1) \times (4 + 1) - 1 = 19 \text{ memory accesses}$$

Saving 5 accesses out of 24 is negligible when 19 serialized round trips to memory still occur.

Cloud providers deliberately avoid backing VPS instances with host-level 2 MiB huge pages because host huge pages inhibit core hypervisor capabilities:

• Memory Overcommit: Splitting host memory into rigid 2 MiB blocks prevents dynamic ballooning and zero-page deduplication (KSM).
• Live Migration: Streaming 4 KiB pages over the network during VM live migration allows fine-grained dirty page tracking using write-protection bits. With 2 MiB host pages, a single byte write dirties the entire 2 MiB range, vastly increasing migration time and network bandwidth.
• Host Fragmentation: Multi-tenant host nodes continuously spawning and destroying virtual machines develop fragmented physical RAM. Finding contiguous 2 MiB host frames for thousands of guest instances causes physical memory starvation on the host.

Because the guest is unaware of host-side fragmentation, it expends significant CPU overhead organizing its internal memory into 2 MiB blocks that yield minimal microarchitectural benefit.

khugepaged, Direct Compaction, and Tail Latency Spikes

The memory management subsystem in Linux allocates huge pages through two distinct mechanisms: the synchronous fault path and the asynchronous khugepaged kernel thread. Both paths introduce latency spikes inside a virtual machine.

When an application triggers a page fault on an anonymous memory region, the kernel checks /sys/kernel/mm/transparent_hugepage/defrag. If this setting is configured to always, the faulting thread synchronously attempts memory compaction if a contiguous 2 MiB block of guest physical frames is unavailable.

Direct compaction initiates an intensive scanning loop:

1. The kernel allocates migration target pages.
2. It walks page tables, isolates active 4 KiB pages, and updates page table references.
3. It copies the payload of 512 discrete 4 KiB pages into a single 2 MiB page.
4. It unlinks the original pages and releases them to the zone buddy allocator.

Inside a VPS, this compaction loop is exceptionally costly. As the guest kernel rearranges page tables and issues inter-processor interrupts (IPIs) to invalidate TLB entries across vCPUs, the hypervisor must intercept these events.

vCPU preemption during a spinlock hold—known as the Lock Holder Preemption (LHP) problem—causes execution to stall. If vCPU 0 holds a page table lock while performing memory compaction and the host hypervisor deschedules vCPU 0 to run another tenant, vCPU 1 will spin fruitlessly on that lock, wasting its entire host CPU timeslice.

Storage and network subsystems that rely on predictable, bounded memory latency suffer immediate degradations. When disk I/O requires rapid ring-buffer submissions, such as workloads analyzed in io_uring vs pread: How Batched NVMe I/O Actually Scales, or when host memory acts as a DMA target as described in How NVMe Host Memory Buffer Works in Client DRAM-Less SSDs, compaction stalls block the processing loop, causing queue backpressure and request timeouts.

If defrag is configured to defer, the kernel offloads this workload to khugepaged. While this prevents the initial allocating thread from stalling, khugepaged creates background CPU spikes. It periodically wakes up, sweeps memory, acquires the mmap_lock on running processes, and converts candidate 4 KiB clusters into huge pages. While khugepaged holds the read lock or upgrades to a write lock on mmap_lock, application threads attempting to allocate memory or modify memory maps are frozen.

Measuring THP Allocation Efficiency During Random Memory Access

To understand why random reads defeat the THP mechanism on virtualized servers, consider a dependent pointer-chase benchmark across a 1 GiB buffer.

In a pointer-chase test, every memory address accessed is contained within the value read from the previous address:

$$A_{i+1} = \text{Buffer}[A_i]$$

Because each address depends on the prior read, CPU hardware prefetchers (such as the L2 stream prefetcher and L1 next-page prefetcher) cannot predict which cache line to load. Every read must complete a full trip through the cache hierarchy.

The following C program allocates a 1 GiB buffer, initializes it with a pseudo-random permutation cycle to guarantee dependent pointer chasing without revisiting entries, and explicitly requests huge pages via madvise():

#define _GNU_SOURCE
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#define BUFFER_SIZE (1ULL * 1024 * 1024 * 1024) // 1 GiB
#define NUM_ELEMENTS (BUFFER_SIZE / sizeof(uint64_t))

int main(void) {
    // Allocate 1 GiB page-aligned buffer
    uint64_t *buffer = mmap(NULL, BUFFER_SIZE, 
                            PROT_READ | PROT_WRITE, 
                            MAP_ANONYMOUS | MAP_PRIVATE, 
                            -1, 0);
    if (buffer == MAP_FAILED) {
        perror("mmap");
        return 1;
    }

    // Advise kernel to back this range with 2 MiB huge pages
    if (madvise(buffer, BUFFER_SIZE, MADV_HUGEPAGE) != 0) {
        perror("madvise");
    }

    // Generate random permutation via Fisher-Yates shuffle
    printf("Initializing 1 GiB permutation table...\n");
    for (uint64_t i = 0; i < NUM_ELEMENTS; i++) {
        buffer[i] = i;
    }

    srand(1337);
    for (uint64_t i = NUM_ELEMENTS - 1; i > 0; i--) {
        uint64_t j = ((uint64_t)rand() << 15 ^ rand()) % (i + 1);
        uint64_t temp = buffer[i];
        buffer[i] = buffer[j];
        buffer[j] = temp;
    }

    // Verify how much memory actually received 2 MiB backing
    FILE *smaps = fopen("/proc/self/smaps", "r");
    if (smaps) {
        char line[256];
        while (fgets(line, sizeof(line), smaps)) {
            if (strstr(line, "AnonHugePages:")) {
                printf("%s", line);
            }
        }
        fclose(smaps);
    }

    // Execute pointer chasing
    printf("Starting dependent pointer traversal...\n");
    uint64_t current = 0;
    struct timespec start, end;
    clock_gettime(CLOCK_MONOTONIC, &start);

    for (uint64_t i = 0; i < 50000000; i++) {
        current = buffer[current];
    }

    clock_gettime(CLOCK_MONOTONIC, &end);
    double elapsed = (end.tv_sec - start.tv_sec) + 
                     (end.tv_nsec - start.tv_nsec) / 1e9;

    printf("Sink: %lu, Elapsed: %.4f seconds\n", current, elapsed);
    munmap(buffer, BUFFER_SIZE);
    return 0;
}

When this program runs on a bare-metal machine, AnonHugePages in /proc/self/smaps immediately reports 1048576 kB (the entire 1 GiB buffer). The CPU's STLB caches all 512 entries, keeping address translation overhead at near zero.

When executed on a typical multi-tenant cloud VPS, inspect /proc/self/smaps and you will observe that AnonHugePages rarely reaches the full 1 GiB allocation upon startup. Instead, it reports a fragmented distribution, often hovering between 200 MiB and 600 MiB.

The consequences for random access patterns are severe:

1. Partial TLB Coverage: The buffer is split between 4 KiB and 2 MiB pages. The 4 KiB segments saturate the small STLB, causing cache thrashing.
2. Translation Inconsistency: The CPU pipeline must continuously switch between 3-level and 4-level guest walks while resolving address translations.
3. Compound Page Splitting: If the VPS kernel experiences low-memory conditions during the allocation, it splits huge pages back into 512 distinct 4 KiB pages via split_huge_page(), invalidating TLB structures and initiating hypervisor page faults.

To observe this breakdown at the hardware counter level, monitor your process using Linux perf:

perf stat -e dTLB-loads,dTLB-load-misses,dtlb_load_misses.walk_duration ./pointer_chase

On a VPS, dTLB-load-misses remains elevated regardless of whether THP is enabled in the guest, while dtlb_load_misses.walk_duration spikes due to the underlying EPT walk latency.

Kernel Configuration and Diagnostic Strategies for VPS Deployments

To ensure predictable performance across cloud instances, systems engineers must configure transparent huge page settings based on the virtualization profile of the host.

First, check the current THP state:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

Three options exist for enabled:

• always: The kernel attempts to back every anonymous allocation with a 2 MiB page.
• madvise: The kernel only attempts huge page allocation on memory explicitly registered with madvise(addr, len, MADV_HUGEPAGE).
• never: Disables transparent huge pages entirely.

On a multi-tenant VPS, setting enabled to always is a mistake. Database engines (such as PostgreSQL and Redis), JVM runtimes, and low-latency network daemons exhibit severe tail-latency degradation under global THP due to memory bloat and background compaction stalls.

When an application allocates a small struct and writes 1 byte, the kernel allocates a 2 MiB page instead of a 4 KiB page. On a virtual machine with limited RAM (e.g., 4 GiB to 8 GiB), this memory amplification triggers the Linux out-of-memory (OOM) killer prematurely.

Apply the following production adjustments inside your VPS:

# Restrict THP to explicit application requests
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

# Prevent synchronous compaction stalls during allocation
echo defer > /sys/kernel/mm/transparent_hugepage/defrag

# Disable background khugepaged compaction sweeps
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag

To persist these values across system reboots, place them in /etc/tmpfiles.d/thp.conf or pass them as kernel boot parameters via GRUB:

transparent_hugepage=madvise

Next, monitor kernel diagnostic counters in /proc/vmstat to assess whether THP is stalling your system:

grep -E "thp|compact" /proc/vmstat

Pay close attention to these metrics:

• compact_stall: Increments every time an application thread is forced to freeze and execute synchronous memory compaction. If this counter is climbing rapidly, applications are incurring double-digit millisecond latency penalties.
• thp_fault_fallback: Increments when the kernel attempts to allocate a 2 MiB page but fails due to memory fragmentation, falling back to 4 KiB pages.
• thp_collapse_alloc_failed: Increments when khugepaged fails to find a contiguous block of physical memory to construct a huge page.
• thp_split_page: Increments when an existing 2 MiB page is broken back down into 512 individual 4 KiB pages.

If thp_fault_fallback and thp_split_page are high relative to thp_fault_alloc, the guest kernel is squandering CPU cycles building huge pages only to immediately fragment them. In this scenario, disabling THP entirely (echo never > /sys/kernel/mm/transparent_hugepage/enabled) flattens p99 latencies.

Conclusion

Transparent Huge Pages provide measurable performance benefits on bare-metal systems with uniform hardware access, reducing TLB pressure across expansive memory structures. However, on a virtual private server, the abstraction layers of hardware-assisted virtualization undermine these assumptions.

Without explicit host-level huge page backing, a 2 MiB page inside a guest VM provides only an illusion of efficiency. The underlying EPT page tables continue to require deep, serialized memory traversals across 4 KiB host boundaries. When coupled with the overhead of guest memory compaction, khugepaged lock contention, and the risk of hypervisor vCPU preemption, global THP transforms from an optimization into a primary source of latency spikes.

For virtualized systems, set THP to madvise or never. Reserve huge page mappings strictly for static, read-heavy buffers where you can profile memory behavior directly, and never assume that bare-metal performance characteristics translate cleanly through a hypervisor.

Measured on our own hardware: What 64 Bytes of Padding Are Worth

How much throughput does false sharing cost when several threads increment counters that share one cache line, compared with the same counters padded onto separate lines?

We ran it. The numbers below come from a program executed on the server hosting this site on 2026-09-20 — an AMD EPYC 9354P 32-Core Processor with 8 cores visible, 31.3 GB of memory, Linux 6.8.0-139-generic.

Metric	Value
slowdown factor	12.2
iterations per thread	10000000
padded best seconds	0.0318
padded mean seconds	0.0403
padded million ops per sec	2514.92
repeats	5
shared line best seconds	0.3881
shared line mean seconds	0.42
shared line million ops per sec	206.15
threads	8

This is a shared virtual server, not an isolated test rig, so treat the absolute figures as indicative and the ratio between the two cases as the finding. The full method, the machine specification, and the complete source code are on the What 64 Bytes of Padding Are Worth benchmark page, so you can check the method or run it yourself.

Measured on our own hardware: Do Transparent Huge Pages Help on a VPS? Random Access Over 1 GiB

On this virtual server, does asking the kernel for 2 MiB transparent huge pages make a dependent random read over a 1 GiB buffer faster than the same buffer on 4 KiB pages, and how much of the buffer actually gets huge pages?

We ran it. The numbers below come from a program executed on the server hosting this site on 2026-09-20 — an AMD EPYC 9354P 32-Core Processor with 8 cores visible, 31.3 GB of memory, Linux 6.8.0-139-generic.

Metric	Value
huge page speedup factor	1.66
accesses	50000000
buffer mib	1024
checksum	61078
huge case anon huge pages kb	1048576
huge case fraction backed by 2m pages	1
huge pages ns per access best	165.96
huge pages ns per access mean	168.09
mode	measure
pages 2m in buffer	512
pages 4k in buffer	262144
repeats	5
small case anon huge pages kb	0
small pages ns per access best	275.62
small pages ns per access mean	289.73

This is a shared virtual server, not an isolated test rig, so treat the absolute figures as indicative and the ratio between the two cases as the finding. The full method, the machine specification, and the complete source code are on the Do Transparent Huge Pages Help on a VPS? Random Access Over 1 GiB benchmark page, so you can check the method or run it yourself.

Measured on our own hardware: io_uring vs pread: Random Reads at Queue Depth 32

How many random 4 KiB reads per second can one thread sustain with serial pread() syscalls, compared with io_uring submitting 32 at a time, when the page cache is taken out of the picture?

We ran it. The numbers below come from a program executed on the server hosting this site on 2026-09-20 — an AMD EPYC 9354P 32-Core Processor with 8 cores visible, 31.3 GB of memory, Linux 6.8.0-139-generic.

Metric	Value
io uring speedup factor	7.85
block bytes	4096
blocks read	20000
io uring best seconds	0.4712
io uring iops	42449
o direct	true
pread best seconds	3.6977
pread iops	5409
pread mean latency us	184.88
queue depth	32
repeats	3

This is a shared virtual server, not an isolated test rig, so treat the absolute figures as indicative and the ratio between the two cases as the finding. The full method, the machine specification, and the complete source code are on the io_uring vs pread: Random Reads at Queue Depth 32 benchmark page, so you can check the method or run it yourself.

References

1. Intel Corporation. "Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3C: System Programming Guide, Part 3." Order Number 326019. Available: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
2. The Linux Kernel Organization. "Transparent Hugepage Support." Linux Kernel Admin-Guide Documentation. Available: https://docs.kernel.org/admin-guide/mm/transhuge.html
3. Linux KVM Project. "KVM: Memory Virtualization Architecture and EPT Operations." Available: https://www.linux-kvm.org/page/Memory