Hugepages¶

What Are HugePages?¶

Normally, Linux memory is managed in pages of 4 KB (on x86_64). Every page needs a mapping in the page table, which the CPU uses to translate virtual addresses into physical addresses via the MMU (Memory Management Unit).

For small processes, 4 KB pages are fine.
But for large memory-hungry applications (tens of GBs), the number of page table entries explodes.

That leads to performance problems.

HugePages are a memory management feature in Linux that allows applications to use larger memory pages than the default system page size (typically 4 KB). The two commonly used hugepage sizes are:

2 MB HugePages
1 GB HugePages

These larger pages reduce the overhead associated with managing smaller memory pages, which improves performance and efficiency for certain workloads.

The problem hugepages solve¶

TLB pressure:
- The CPU caches virtual→physical translations in the Translation Lookaside Buffer (TLB).
- With many 4 KB pages, you get lots of page table entries. The TLB can’t cache them all → more TLB misses → more memory stalls.
- Hugepages (2 MB or 1 GB on x86) reduce the number of entries needed dramatically. Example:
- Mapping 1 GB memory with 4 KB pages → 262,144 entries.
- Mapping 1 GB memory with 2 MB hugepages → 512 entries.
- Mapping 1 GB memory with 1 GB hugepage → 1 entry.
Page table size:
- Less overhead in page tables → less memory wasted.
Performance stability:
- Applications that do a lot of memory lookups (databases, NFV/DPDK apps, Java apps) benefit because memory access latency becomes more predictable.

Why hugepages are needed¶

Databases (e.g., Oracle, PostgreSQL, MySQL) → stable performance with large buffer pools.
Virtualization (KVM/QEMU) → VMs benefit from lower TLB misses when backing guest RAM with hugepages.
Networking (DPDK, VPP, SR-IOV) → packet buffers must be allocated from hugepage-backed memory for DMA and performance reasons.
Big-data & HPC apps → large memory regions accessed sequentially or randomly.

Without hugepages, performance suffers due to TLB thrashing and higher memory management overhead.

How to configure hugepages¶

Linux offers two mechanisms:

Static (reserved) hugepages
Dynamic hugepages

1. Static (reserved) hugepages¶

You tell the kernel upfront to reserve a certain number of hugepages at boot.

Check what sizes are supported:
1
grep Hugepagesize /proc/meminfo
→ usually 2048 kB (2 MB), sometimes also 1048576 kB (1 GB).
Reserve at boot (e.g., in GRUB config /etc/default/grub):
1
GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=4"
→ This reserves 4 × 1 GB hugepages = 4 GB RAM.
After updating GRUB:
1 2
sudo update-grub sudo reboot
Verify:
1
grep Huge /proc/meminfo

When reserving huge page on the host, you say: "Take this part of RAM and carve it out into big fixed-size blocks (2 MB or 1 GB). Do not use it for normal processes or the page cache. Only let apps that explicitly request hugepages allocate from it."

Why reserve them in advance?

Because hugepages must be physically contiguous memory (2 MB or 1 GB blocks).
If you try to allocate them on the fly, the kernel may fail (especially for 1 GB pages) if memory is fragmented.

So by reserving at boot, you guarantee:

Hugepages are always available.
They come from low-level physical memory before it gets fragmented.
Your application won’t fail later due to “out of hugepages.”

2. Dynamic hugepages (Transparent Huge Pages, THP)¶

Linux can automatically use hugepages when possible, without explicit reservation.

Controlled via:

1	`cat /sys/kernel/mm/transparent_hugepage/enabled`

Options: always, madvise, never.
THP is good for general-purpose workloads, but some apps (databases, DPDK) prefer explicit hugepages for more control.

How to use hugepages¶

VM with DPDK¶

Best-practice layered approach for running DPDK in a VM.

Step 1: Host, reserve and mount hugepages¶

# Reserve 4 GB of 2 MB hugepages: 4*1024 / 2 = 2048 pages
echo 2048 | sudo tee /proc/sys/vm/nr_hugepages

# Create and mount hugetlbfs
sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge

# Verify
grep -E 'HugePages_|Hugepagesize' /proc/meminfo
# You should see: HugePages_Total: 2048  HugePages_Free: (close to 2048)  Hugepagesize: 2048 kB

Tip (NUMA hosts): you can reserve per-NUMA node at /sys/devices/system/node/nodeX/hugepages/hugepages-2048kB/nr_hugepages.

Step 2: Host, launch the VM with hugepage-backed RAM¶

Simple QEMU CLI:

qemu-system-x86_64 \
  -enable-kvm \
  -cpu host \
  -smp 4 \
  -m 4G \
  -mem-prealloc \
  -mem-path /mnt/huge \
  -drive file=vm.img,if=virtio,cache=none,format=qcow2 \
  -net nic,model=virtio -net user

This makes the VM’s 4 GB come from host hugepages (reduces host-side TLB pressure). If you prefer the modern “memory-backend-file” form (handy for NUMA pinning and sharing):

qemu-system-x86_64 \
  -enable-kvm -cpu host -smp 4 \
  -object memory-backend-file,id=mem0,size=4G,mem-path=/mnt/huge,share=on,prealloc=on \
  -machine type=q35,accel=kvm \
  -m 4G -numa node,memdev=mem0 \
  -drive file=vm.img,if=virtio,cache=none,format=qcow2

Libvirt/XML users: set a <memoryBacking><hugepages/></memoryBacking> block or use 1G pages if supported.

Step 3: Inside the VM (guest), reserve and mount hugepages for DPDK¶

# Reserve 2 GB (2 MB pages → 2*1024 / 2 = 1024 pages)
echo 1024 | sudo tee /proc/sys/vm/nr_hugepages

# Mount hugetlbfs
sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge

# Verify
grep -E 'HugePages_|Hugepagesize' /proc/meminfo

Optional but common for performance-sensitive apps:

1 2	`# Transparent Huge Pages often disabled for determinism echo never \| sudo tee /sys/kernel/mm/transparent_hugepage/enabled`

Make it persistent (examples):

1 2	`/etc/sysctl.conf: vm.nr_hugepages = 1024 /etc/fstab: nodev /mnt/huge hugetlbfs defaults 0 0`

Step 4: Inside the VM (guest), run a DPDK app using the hugepages¶

Typical DPDK EAL flags to consume 2 GB from /mnt/huge:

# Example NIC binding is out of scope here; assume your NIC/virtio-pmd is ready.
# Launch a DPDK sample app (e.g., testpmd) and tell EAL where the hugepages are.

sudo ./build/app/dpdk-testpmd \
  -l 0-3 \                         # CPU cores (example)
  -n 4 \                           # memory channels
  --file-prefix=dpdkvm \           # unique mem file prefix
  --huge-dir=/mnt/huge \           # use the guest's hugepage mount
  --socket-mem=2048 \              # allocate ~2 GB on socket 0
  -- \                             # end of EAL, start of app args
  --forward-mode=io --port-topology=chained

Notes: