API reference

The reference below is generated directly from the header doc-comments with Doxygen and Breathe. For a narrative overview of what each header is for, see Libraries.

Primitives

bitfield.h

Bitfield utilities

Helpers for working with individual bits and bitfields in low-level code. Designed for clarity and performance in systems, protocol, and register-level programming.

Features

Intended usage includes manipulating hardware registers, encoding protocol fields, or packing/unpacking structured data into compact bit layouts.

Example usage:

uint64_t reg = 0; reg = bitfield_set(reg, 0, 4, 0x5); // Set bits 0–3 reg = bitfield_set(reg, 4, 4, 0xA); // Set bits 4–7 reg = bitfield_set(reg, 8, 8, 0xFF); // Set bits 8–15

uint64_t x = bitfield_get(reg, 4, 4); // Extract bits 4–7

This header is intended to grow with additional bit manipulation utilities over time.

Version

0.7.0

Functions

static inline uint64_t bitfield_mask(uint8_t offset, uint8_t width)

Returns a bitmask for a given offset and width.

static inline uint64_t bitfield_get(uint64_t val, uint8_t offset, uint8_t width)

Extracts a bitfield from a 64-bit integer.

static inline uint64_t bitfield_set(uint64_t val, uint8_t offset, uint8_t width, uint64_t field)

Sets a bitfield in a 64-bit integer.

barriers.h

Memory-barriers and other intrinsics

This header defines memory and instruction barriers for x86 (x86_64, i386) and ARM (aarch64, arm). These are used to control the ordering of memory operations. It also provides a CPU relaxation utility for use in spin-loops.

Will not compile on other architectures.

Version

0.7.0

Functions

static inline void barrier(void)

Compiler memory barrier.

Prevents the compiler from reordering memory accesses across this point. This does not emit any CPU instruction.

static inline void rmb(void)

Read memory barrier.

Ensures that all read operations before the barrier are globally visible before any subsequent reads. Enforces ordering of loads.

static inline void wmb(void)

Write memory barrier.

Ensures that all write operations before the barrier are globally visible before any subsequent writes. Enforces ordering of stores.

static inline void mb(void)

Full memory barrier.

Ensures that all memory operations (loads and stores) before the barrier are globally visible before any subsequent memory operations.

static inline void dma_rmb(void)

DMA read memory barrier.

Ensures the CPU observes DMA writes performed by a device before any subsequent reads by the CPU from the same memory region.

Use this before reading from memory that was written by a device (e.g., completion queues).

static inline void cpu_relax(void)

CPU relaxation hint for spin-wait loops.

On x86, emits the pause instruction (rep; nop), reducing power and bus contention. On ARM, emits yield. On other architectures, defaults to a no-op.

mmio.h

Helpers for 32-bit and 64-bit MMIO read/write access

This header provides simple functions for accessing memory-mapped I/O (MMIO) regions, typically used for interacting with device registers on PCIe devices. The accessors use volatile semantics to prevent the compiler from reordering or eliminating memory operations that may have side effects at the hardware level.

These functions assume that MMIO registers are little-endian, as is standard on most PCIe devices. No byte-swapping is performed. As such, this library is not suitable for use on big-endian systems without modification.

This implementation has been verified only on x86 systems and may require additional memory barriers or platform-specific instructions to work reliably on weakly ordered architectures (e.g., ARM). The user is responsible for ensuring correct ordering when porting to such platforms.

The region pointer typically refers to the base of a memory-mapped PCI BAR (e.g., from struct pci_func_bar.region), obtained via UIO, VFIO, or similar mechanisms.

Version

0.7.0

Functions

static inline uint32_t mmio_read32(void *region, uint32_t offset)

Read a 32-bit value from an MMIO region at the given offset.

Parameters:

  • region – Pointer to the base of the MMIO region

  • offset – Byte offset from the base

Returns:

32-bit value read from the region

static inline void mmio_write32(void *region, uint32_t offset, uint32_t value)

Write a 32-bit value to an MMIO region at the given offset.

Parameters:

  • region – Pointer to the base of the MMIO region

  • offset – Byte offset from the base

  • value – 32-bit value to write

static inline uint64_t mmio_read64(void *region, uint32_t offset)

Read a 64-bit value from an MMIO region at the given offset.

For portability, this performs two 32-bit accesses instead of a single 64-bit access.

Parameters:

  • region – Pointer to the base of the MMIO region

  • offset – Byte offset from the base

Returns:

64-bit value read from the region

static inline void mmio_write64(void *region, uint32_t offset, uint64_t val)

Write a 64-bit value to an MMIO region at the given offset.

For portability, this performs two 32-bit accesses instead of a single 64-bit access.

Parameters:

  • region – Pointer to the base of the MMIO region

  • offset – Byte offset from the base

  • value – 64-bit value to write

PCI and VFIO

pci.h

Helpers for Linux PCI interface via sysfs

  • Scan system for PCI devices / functions

    • Callback invocation on each discovered function

  • Retrieve “handles” to PCI devices via pci_func_{open,close} using PCI BDF

    • Handles provide PCI addresses, identifiers, and a container for BAR regions

  • Does BAR region mapping via /sys/bus/pci/devices/<PCI_ADDR>/resourceX

  • Provides MMIO accessor functions: pci_region_read32, pci_region_read64, pci_region_write32, and pci_region_write64

Version

0.7.0

Defines

PCI_BDF_LEN
PCI_NBARS

Typedefs

typedef int (*pci_func_callback)(struct pci_func *func, void *callback_arg)

Callback function definition for pci_scan(); must return a ‘pci_scan_action’.

Return PCI_SCAN_ACTION_CLAIM_FUNC to take ownership of the struct pci_func, or PCI_SCAN_ACTION_RELEASE_FUNC to have pci_scan() clean it up.

Enums

enum pci_scan_action

Values:

enumerator PCI_SCAN_ACTION_CLAIM_FUNC
enumerator PCI_SCAN_ACTION_RELEASE_FUNC

Functions

static inline uint8_t pci_addr_get_function(uint32_t bdf)
static inline uint8_t pci_addr_get_device(uint32_t bdf)
static inline uint8_t pci_addr_get_bus(uint32_t bdf)
static inline uint16_t pci_addr_get_domain(uint32_t bdf)
static inline int pci_bar_pr(struct pci_func_bar *bar)
static inline int pci_func_pr(struct pci_func *func)
static inline int pci_addr_from_text(const char *text, struct pci_addr *addr)

Fills the given ‘addr’ with parts found when scanning a text repr.

on the form ‘0000:05:00.0’

static inline int pci_addr_to_text(struct pci_addr *addr, char *text)

Populates the given char-array with a textual representation of the given ‘addr’.

static inline int pci_func_open(const char *bdf, struct pci_func *func)
static inline int pci_bar_unmap(struct pci_func_bar *bar)
static inline int pci_bar_size(const char *bdf, uint8_t id, size_t *size)

Read the size of PCI BAR id for the function at bdf from sysfs, without mapping it.

Useful for verifying BAR sizing before mapping or for callers that only need the size (e.g. to check BAR1 vs device memory size).

Returns:

0 on success, negative errno on failure.

static inline int pci_bar_largest_size(const char *bdf, size_t *size)

Read the size of the largest PCI memory BAR for the function at bdf.

Scans all BARs and returns the largest. Non-existent resourceN entries (unmapped BARs and the high half of a 64-bit BAR) are skipped.

Returns:

0 on success with *size set, negative errno on failure.

static inline int pci_bar_map(const char *bdf, uint8_t id, struct pci_func_bar *bar)
static inline void pci_func_close(struct pci_func *func)
static inline int pci_scan(pci_func_callback callback, void *callback_arg)

Scans /sys/bus/pci/devices for PCI functions and calls the provided callback for each one.

struct pci_addr
#include <pci.h>

Public Members

uint32_t value
struct pci_idents
#include <pci.h>

Representation of PCI identifiers.

Public Members

uint16_t vendor_id

Device vendor; e.g. Samsung, QEMU.

uint16_t device_id

Device identifier;.

uint32_t classcode

Base, sub, and programming-interface.

struct pci_func_bar
#include <pci.h>

Encapsulation of a PCI BAR region mapping.

Public Members

uint64_t size

The size of the BAR region.

void *region

Pointer to mmap’ed BAR region.

uint8_t id

One of the six BARs; [0-5].

int fd

Handle to file-representation.

struct pci_func
#include <pci.h>

Public Members

struct pci_addr addr

The address of the PCI device function.

char bdf[PCI_BDF_LEN + 1]

PCI address as a null-terminated full BDF string.

struct pci_idents ident

Describes who made it and what it is.

struct pci_func_bar bars[PCI_NBARS]

The six BARs associated with a PCI Function.

vfioctl.h

VFIO helper for user-space

The intent is for this library to be vendored by who-ever needs it. Possibly also its companion driver-script.

The documentation for using vfio-pci is to my knowledge split between:

Most of the functions here are simply wrappers of the IOCTLs of the same name. And a brief introduction to the data structures:

VFIO Container (/dev/vfio/vfio)

|

| | IOMMU Group 5 IOMMU Group 6 (/dev/vfio/5) (/dev/vfio/6) | |

| Dev A | | Dev C | | Dev B | | |

The above is the essential modeling of the isolation-level of memory among devices.

Version

0.7.0

Functions

static inline int vfio_container_close(struct vfio_container *container)

Close a container.

Parameters:

  • container – Pointer to a container handle (see vfio_container_open)

Returns:

On success, 0 is returned. On error, negative errno is set to indicate the error.

static inline int vfio_container_open(struct vfio_container *container)

Retrieve a container handle.

Parameters:

  • container – Assigns the file-descriptor to container.fd

Returns:

On success, 0 is returned and container.fd setup as a handle to a vfio container. On error, negative errno is set to indicate the error.

static inline int vfio_get_api_version(struct vfio_container *container, int *api_version)
static inline int vfio_check_extension(struct vfio_container *container, int extension)
static inline int vfio_set_iommu(struct vfio_container *container, int iommu_type)
static inline int vfio_group_close(struct vfio_group *group)
static inline int vfio_group_open(int id, struct vfio_group *group)

Open the group with the given ‘id’.

Parameters:

  • id – Numerical identifier of the given group

static inline int vfio_group_get_status(struct vfio_group *group)
static inline int vfio_group_set_container(struct vfio_group *group, struct vfio_container *container)
static inline int vfio_group_get_device_fd(struct vfio_group *group, const char *device_name)
static inline int vfio_device_get_info(int device_fd, struct vfio_device_info *info)
static inline int vfio_device_get_region_info(int device_fd, struct vfio_region_info *region)
static inline void *vfio_map_region(int device_fd, size_t size, off_t offset)
static inline int vfio_iommu_get_info(struct vfio_container *container, struct vfio_iommu_type1_info *info)
static inline int vfio_iommu_map_dma(struct vfio_container *container, struct vfio_iommu_type1_dma_map *map)
static inline int vfio_iommu_unmap_dma(struct vfio_container *container, struct vfio_iommu_type1_dma_unmap *unmap)
static inline int vfio_device_get_irq_info(struct vfio_device *dev, struct vfio_irq_info *irq)
static inline int vfio_device_set_irqs(struct vfio_device *dev, struct vfio_irq_set *irq_set)
static inline int vfio_device_reset(struct vfio_device *dev)
static inline int vfio_device_get_pci_hot_reset_info(struct vfio_device *dev, struct vfio_pci_hot_reset_info *info)
static inline int vfio_device_pci_hot_reset(struct vfio_device *dev, struct vfio_pci_hot_reset *reset)
static inline int vfio_device_bar_export_dmabuf(int device_fd, uint32_t region_index, uint64_t offset, uint64_t length)

Export a slice of a vfio-pci device region as a dma-buf fd.

Wraps VFIO_DEVICE_FEATURE | GET | DMA_BUF. The returned fd is a regular dma-buf that iommufd’s IOMMU_IOAS_MAP_FILE accepts on mainline 6.19+ because the exporter is vfio-pci (private interconnect via vfio_pci_dma_buf_iommufd_map).

Parameters:

  • device_fd – vfio-cdev device fd (see vfio_cdev_open).

  • region_index – BAR/region index, e.g. VFIO_PCI_BAR1_REGION_INDEX.

  • offset – Offset within the region.

  • length – Length of the slice.

Returns:

dma-buf fd (>= 0) on success, negative errno on failure.

struct vfio_group
#include <vfioctl.h>

Public Members

int fd
int id
struct vfio_group_status status
struct vfio_container
#include <vfioctl.h>

Public Members

int fd
struct vfio_iommu_type1_info info
struct vfio_iommu_type1_dma_map map
struct vfio_device
#include <vfioctl.h>

Public Members

int fd

Host memory

hostmem_hugepage.h

Memory allocator for hugepages

These wrappers provide an encapsulated use of hugepages. That is, they work with memfd_create() as well as hugetlbfs. The motivation for this library is to provide a minimal API encapsulating the differences in tlbfs and memfd, making it more convenient to build things on top.

The general motivation is to use hugepages as a way to obtain physically addressable memory which is pinned and contigous.

As these properties are required when working on user-space drivers. Additionally, then hugepages and mmap regions with SHARED are very useful as a basic IPC channel. Thus, providing this for the regions allocated here.

API: Hugepages

Caveat: system setup

The library makes use of memfd_create(MFD_HUGETLB), however, you still need to allocate them yourself. That is, have a system setup step than makes hugepages available, such as:

echo 128 | tee -a /proc/sys/vm/nr_hugepages ulimit -l unlimited

Thus, a utility for this similar to devbind.py is needed. This is what we have today with ‘xnvme-driver’, however, we want something simpler.

Caveat: CAP_SYS_ADMIN

Reading /proc/self/pagemap requires CAP_SYS_ADMIN, so hostmem_virt_to_phys() cannot be used by non-privileged users. Therefore, any process needing DMA via this allocator must run as root.

Possible Workaround: Since the allocator uses MAP_SHARED, a privileged “allocator-daemon” could handle virt_to_phys translations and share the results via shared memory with unprivileged clients. This allows integration into the heap with minimal complexity. Example:

After heap initialization, write the heap structure into hugepage memory. Because phys_lut[] resolves all physical addresses of the backing hugepages, any process that imports the hugepage also gains access to those physical addresses—without needing CAP_SYS_ADMIN.

Version

0.7.0

Functions

static inline int hostmem_hugepage_pp(struct hostmem_hugepage *hugepage)
static inline void hostmem_hugepage_free(struct hostmem_hugepage *hugepage)

Deallocate a hugepage allocation.

static inline int hostmem_hugepage_alloc(size_t size, struct hostmem_hugepage *hugepage, struct hostmem_config *config)

Allocate a hugepage of the given ‘size’.

Parameters:

  • size – Must be a multiple of 2M

  • hugepage – Pointer to a pre-allocated hugepage-descriptor

Returns:

On success, 0 is returned. On error, negative errno is returned to indicate the error.

static inline int hostmem_hugepage_import(const char *path, struct hostmem_hugepage *hugepage, struct hostmem_config *config)

Import (re-map) an existing hugepage shared by another process.

This function uses fstat() to determine the size of the shared memory region.

Parameters:

  • path – Path to the memfd or hugetlbfs file (e.g. /proc/<pid>/fd/<fd>)

  • hugepage – Pre-allocated pointer to the descriptor to fill in

Returns:

0 on success, negative errno on error

struct hostmem_hugepage
#include <hostmem_hugepage.h>

Public Members

int fd
void *virt
size_t size
uint64_t phys
char path[256]
struct hostmem_config *config
size_t nphys

Number of hugepages backing ‘virt’.

uint64_t *phys_lut

Per-hugepage physical base; NULL when pagemap read is unavailable.

hostmem_heap.h

Functions

static inline int hostmem_heap_pp(struct hostmem_heap *heap)
static inline void hostmem_heap_term(struct hostmem_heap *heap)
static inline int hostmem_heap_init(struct hostmem_heap *heap, size_t size, struct hostmem_config *config)

Initialize the given heap.

  • Pre-allocate a va-space of ‘size’ bytes backend by hugepage(s)

  • Setup the LUT / physical address for hugepage backing the va-space

TODO: use the hugepage memory for the heap-description! By doing so, then a helper process can do all the hugepage lookup-work, and then anybody who imports it, will be able to, as a non-privileged user to know of the virt-to-phys mapping

static inline void hostmem_heap_block_free(struct hostmem_heap *heap, void *ptr)
static inline void *hostmem_heap_block_alloc_array_aligned(struct hostmem_heap *heap, size_t elem_count, size_t elem_size, size_t alignment)
static inline void *hostmem_heap_block_alloc_array(struct hostmem_heap *heap, size_t elem_count, size_t elem_size)
static inline void *hostmem_heap_block_alloc_aligned(struct hostmem_heap *heap, size_t size, size_t alignment)
static inline void *hostmem_heap_block_alloc(struct hostmem_heap *heap, size_t size)
static inline int hostmem_heap_block_virt_to_phys(struct hostmem_heap *heap, void *virt, uint64_t *phys)
static inline uint64_t hostmem_heap_block_vtp(struct hostmem_heap *heap, void *virt)

Same as hostmem_buffer_virt_to_phys() but without any error-handling, thus return the phys address instead of error.

struct hostmem_heap_block
#include <hostmem_heap.h>

Representation of a memory-allocation as produced by hostmem_buffer_alloc(…)

Public Members

size_t size
int free
struct hostmem_heap_block *next
struct hostmem_heap
#include <hostmem_heap.h>

A pre-allocated heap providing memory for a buffer-allocator.

Public Members

struct hostmem_hugepage memory

A hugepage-allocation; can span multiple hugepages.

struct hostmem_heap_block *freelist

Pointers to description of free memory in the heap.

struct hostmem_config *config

Pointer to hugepage configuration.

size_t nphys

Number of hugepages backing ‘memory’.

uint64_t *phys_lut

An array of physical addresses; on for each hugepage in ‘memory’.

hostmem_dma.h

Hugepage-backed malloc-like allocator for DMA in userspace

This header provides a minimal, header-only allocator for use in user-space drivers requiring DMA-capable memory. Allocations are backed by hugepages and are guaranteed to be contiguous in both virtual and physical address space — up to hugepage granularity.

Interface

  • void *hostmem_dma_malloc(size_t size); Allocate a block of memory of the given size.

  • void hostmem_dma_free(void *ptr); Frees a block previously returned by hostmem_dma_malloc().

  • uint64_t hostmem_dma_v2p(void *virt); Resolve a virtual address to its corresponding physical address.

Usage

You must call hostmem_dma_init() before any allocation is made, and hostmem_dma_term() after all memory has been freed. The allocator does not support lazy initialization.

Caveats

  • Physical contiguity is guaranteed only up to the system’s hugepage size. On most systems, this is 2MB.

  • Sub-hugepage allocations may span multiple hugepages, resulting in reduced physical contiguity. This may be addressed in a future update.

  • Alignment is currently to the system’s page size (typically 4KB).

Roadmap

Planned improvements include:

  • hostmem_dma_calloc() for zero-initialized memory

  • Sub-hugepage contiguity enforcement

Version

0.7.0

Functions

static inline void hostmem_dma_free(struct hostmem_heap *heap, void *ptr)

Free the DMA-capable memory pointed to by ptr

If ptr is NULL, no operation is performed.

Parameters:
static inline void *hostmem_dma_malloc(struct hostmem_heap *heap, size_t size)

Allocate size bytes of DMA-capable memory.

Parameters:

  • size – Number of bytes to allocate. Passing size=0 is considered invalid-input by hostmem_dma_malloc().

Returns:

On success, a pointer to the allocated memory is returned. On error, NULL is returned and errno set to indicate the error.

static inline void *hostmem_dma_malloc_aligned(struct hostmem_heap *heap, size_t size, size_t alignment)

Allocate size bytes of DMA-capable memory aligned to given ‘alignment’.

Parameters:

  • size – Number of bytes to allocate.

  • alignment – Boundary to align to.

Returns:

Pointer to the allocated memory, or NULL on failure.

static inline void *hostmem_dma_alloc_array(struct hostmem_heap *heap, size_t elem_count, size_t elem_size)

Allocate elem_count * elem_size bytes of DMA-capable memory.

If elem_count * elem_size is larger than the size of a hugepage, elem_size must be a divisor of the hugepage size.

Parameters:

  • elem_count – Number of elements to allocate. Passing elem_count=0 is considered invalid-input by hostmem_dma_alloc_array().

  • elem_size – Number of bytes to allocate per element. Passing elem_size=0 is considered invalid- input by hostmem_dma_alloc_array().

Returns:

On success, a pointer to the allocated memory is returned. On error, NULL is returned and errno set to indicate the error.

static inline uint64_t hostmem_dma_v2p(struct hostmem_heap *heap, void *virt)

Resolve the physical address of a given virtual address.

Parameters:
Returns:

Physical address corresponding to the given virtual address.

dma-buf

dmabuf.h

Representation of a dma-buf and its physical pages

A generic interface compatible with any dma-buf. A dma-buf descriptor can be obtained from host memory (memfd via udmabuf) or from device memory, e.g. CUDA or ROCm.

This header is dependency-free: it describes a dma-buf and segments its pages, and needs nothing beyond libc. Resolving the DMA addresses behind a dma-buf in the first place needs the out-of-tree dmabuf_import module and lives in <upcie/experimental/dmabuf_import.h>.

Version

0.7.0

Functions

static inline int dmabuf_pp(struct dmabuf *dmabuf)

Print information about the given dma-buf and each of it’s pages.

static inline int dmabuf_get_lut(struct dmabuf *dmabuf, size_t nphys, uint64_t *phys_lut, uint64_t page_size)

Get LUT (lookup table) from dma-buf.

The pages in the dma-buf might span multiple physical pages. This function creates a LUT segmented to fit the provided page_size.

NOTE: Requires pre-allocated phys_lut

static inline int dmabuf_get_granule_lut(struct dmabuf *dmabuf, uint64_t *lut, size_t nlut, uint64_t granule)

Summarise a dma-buf into one address per granule.

Where dmabuf_get_lut() expands the scatter list per page, this collapses it per granule, for a translator indexing by va >> granule_shift.

Each granule is verified physically contiguous rather than assumed: an exporter may split a contiguous run at its own page size, which is harmless, but a genuine discontinuity would make base + offset resolve wrongly.

And it is tolerant at both ends, because neither vendor’s export lines up with the allocation size it reports. An export describing more than nlut granules is fine, the surplus is ignored; so is a final granule the export only partially covers, which is what an allocation not ending on a granule boundary produces. What is not fine is a granule the export does not reach at all, since nothing would fill its entry.

Parameters:

  • dmabuf – Attached dma-buf to read the scatter list from

  • lut – Destination, nlut entries, one per granule from the start

  • nlut – Number of granules to fill

  • granule – Bytes per entry; a power of two

Returns:

0 on success, -EINVAL on bad arguments or a granule the export does not reach, -EOPNOTSUPP when a granule is not contiguous.

struct dmabuf_page
#include <dmabuf.h>

Public Members

uint64_t addr

Address of a page.

uint64_t len

Length of the page (can span multiple phys pages)

struct dmabuf
#include <dmabuf.h>

Public Members

int fd

dma-buf file descriptor

size_t npages

Number of pages in the dma-buf.

struct dmabuf_page *pages

Array of pages in the dma-buf.

NVMe

nvme_controller.h

Rudimentary Representation of Controller, BAR Mapping, Registers, and Derived Values

This header defines basic structures and access patterns for working with an NVMe controller, including BAR-space mappings, controller registers, and values derived from register content.

Version

0.7.0

Functions

static inline void nvme_controller_close(struct nvme_controller *ctrlr)
static inline int nvme_controller_open(struct nvme_controller *ctrlr, const char *bdf, struct hostmem_heap *heap)

Disables the NVMe controller at ‘bdf’, sets up admin-queues and enables it again.

static inline int nvme_controller_delete_io_qpair(struct nvme_controller *ctrlr, struct nvme_qpair *qpair)

Deletes the submission-queue and completion-queue and frees host-side resources.

Sends Delete I/O SQ and Delete I/O CQ admin commands to the controller, then releases the host DMA memory and returns the queue ID to the free pool.

Parameters:

  • ctrlr – Pointer to a pre-allocated NVMe controller

  • qpair – Pointer to a queue-pair (from nvme_controller_create_io_qpair)

Returns:

0 on success, negative errno on error. Resources are freed regardless.

static inline int nvme_controller_create_io_qpair(struct nvme_controller *ctrlr, struct nvme_qpair *qpair, uint16_t depth)

Allocates a submission-queue, a completion-queue, and wraps them in the nvme_qpair struct.

struct nvme_controller
#include <nvme_controller.h>

This is one way of combining the various components needed.

Public Members

struct pci_func func

The PCIe function and mapped bars.

struct nvme_qpair aq

Admin qpair.

uint64_t qids[NVME_QID_BITMAP_WORDS]

Allocation status of IO queues.

struct hostmem_heap *heap

Heap for DMA-capable memory.

void *buf

IO-buffer for identify-commands, io-qpair-creation etc.

uint32_t csts

Controller Status Register Value.

uint32_t cap

Controller Capabilities Register Value.

uint32_t cc

Controller configuration Register Value.

int timeout_ms

Command timeout in milliseconds (derived from cap.to)

nvme_qpair.h

NVMe Queue Pair Abstraction

This header defines a minimal software abstraction for managing NVMe queue pairs (SQ/CQ) in a user-space NVMe driver context. It provides basic functionality for queue setup, submission, completion handling, and doorbell notification.

A queue pair is represented by ‘struct nvme_qpair’, which includes memory-mapped pointers to the submission and completion queues, doorbell registers, and associated tracking state (head, tail, phase).

Key functions include:

nvme_qpair_init(): Initializes a queue pair and allocates DMA memory for SQ/CQ. nvme_qpair_term(): Frees resources associated with a queue pair. nvme_qpair_reap_cpl(): Polls the CQ for a completion, updates head/phase, and rings CQ doorbell. nvme_qpair_sqdb_ring(): Notifies the controller by ringing the SQ doorbell. nvme_qpair_enqueue(): Writes the given command into the SQ nvme_qpair_submit_sync(): Submits a command and waits synchronously for its completion.

See also: nvme_qid.h for queue ID (qid) management.

Version

0.7.0

Functions

static inline void nvme_qpair_term(struct nvme_qpair *qp)
static inline int nvme_qpair_init(struct nvme_qpair *qp, uint32_t qid, uint16_t depth, uint8_t *bar0, struct hostmem_heap *heap)

Initialize a queue-pair on the given controller.

static inline int nvme_qpair_reap_cpl(struct nvme_qpair *qp, int timeout_ms, struct nvme_completion *cpl)

Reaps at most a single completion and informs the controller via qp->cqdb.

Parameters:

  • qp – A queue-pair as represented by ‘struct nvme_qp’

  • cpl – Completion when one is reaped

  • timeout_ms – Timeout in milliseconds

Returns:

Pointer to a valid completion, or NULL on timeout.

static inline void nvme_qpair_sqdb_update(struct nvme_qpair *qp)

Update the submission queue tail doorbell if needed.

This function writes the current SQ tail index to the MMIO doorbell register for the given queue pair, notifying the controller of new commands. To avoid redundant MMIO writes, the function checks whether the tail value has changed since the last call. The last written value is tracked in nvme_qpair->tail_last_written.

Parameters:

  • qp – Pointer to the NVMe queue pair whose SQ doorbell should be updated.

static inline int nvme_qpair_enqueue(struct nvme_qpair *qp, struct nvme_command *cmd)

Enqueue a command into an NVMe submission queue of a nvme_qpair

That is, writes it into the submission queue memory and increments the tail-pointer, it does not write the tail to the sq-doorbell.

Parameters:

  • qp – The queue-pair

  • cmd – Command to submit

  • user – Optional opaque pointer returned on completion

Returns:

On success 0 is returned. On error then negative errno is set to indicate the error.

static inline int nvme_qpair_submit_sync(struct nvme_qpair *qp, struct nvme_command *cmd, int timeout_ms, struct nvme_completion *cpl)

Submits a command on the given qpair, waits for completion, and populates cpl.

This is intended for synchronous I/O or Admin commands where the caller manages the payload and sets up PRP1/PRP2 manually. The function does not modify or validate the PRP fields.

Parameters:

  • qp – Pointer to the submission queue pair.

  • cmd – Pointer to the command to submit; cid will be assigned.

  • timeout_ms – Timeout in milliseconds to wait for command completion.

  • cpl – Pointer to a completion structure to receive the result.

Returns:

On success 0 is returned. On error, negative errno is returned to indicate the error.

static inline int nvme_qpair_submit_sync_contig_prps(struct nvme_qpair *qp, struct hostmem_heap *heap, void *dbuf, size_t dbuf_nbytes, struct nvme_command *cmd, int timeout_ms, struct nvme_completion *cpl)

Submits a command with a contiguous PRP payload, waits for completion, and populates cpl.

This is intended for synchronous I/O or Admin commands using a physically contiguous buffer. The function prepares the PRP entries automatically using the provided heap and dbuf, sets up the command, submits it on the given qpair, and waits for completion.

Parameters:

  • qp – Pointer to the submission queue pair.

  • heap – Pointer to the host memory heap used for resolving physical addresses.

  • dbuf – Pointer to the data buffer to be described via PRPs.

  • dbuf_nbytes – Size of the data buffer in bytes.

  • cmd – Pointer to the command to submit; cid will be assigned and PRPs set.

  • timeout_ms – Timeout in milliseconds to wait for command completion.

  • cpl – Pointer to a completion structure to receive the result.

Returns:

On success 0 is returned. On error, negative errno is returned to indicate the error.

static inline int nvme_qpair_submit_sync_iov_prps(struct nvme_qpair *qp, struct hostmem_heap *heap, struct iovec *dvec, size_t dvec_cnt, struct nvme_command *cmd, int timeout_ms, struct nvme_completion *cpl)

Submits a command with an iovec PRP payload, waits for completion, and populates cpl.

This is intended for synchronous I/O commands using scatter-gather buffers. The function prepares the PRP entries automatically using the provided heap and dvec, sets up the command, submits it on the given qpair, and waits for completion.

Parameters:

  • qp – Pointer to the submission queue pair.

  • heap – Pointer to the host memory heap used for resolving physical addresses.

  • dvec – Array of iovec structures describing the data segments.

  • dvec_cnt – Number of elements in the dvec array.

  • cmd – Pointer to the command to submit; cid will be assigned and PRPs set.

  • timeout_ms – Timeout in milliseconds to wait for command completion.

  • cpl – Pointer to a completion structure to receive the result.

Returns:

On success 0 is returned. On error, negative errno is returned to indicate the error.

struct nvme_qpair
#include <nvme_qpair.h>

Public Members

void *sq

VA-Pointer to DMA-capable memory backing the Submission Queue (SQ)

void *cq

VA-Pointer to DMA-capable memory backing the Completion Queue (CQ)

void *sqdb

Pointer to Submission Queue Doorbell Register in bar0.

void *cqdb

Pointer to Completion Queue Doorbell Register in bar0.

uint32_t qid

The admin: queue-id == 0 ; io: queue-id > 0;.

uint16_t depth

Length of the queue-pair.

uint16_t tail

Submissin Queue Tail Pointer.

uint16_t tail_last_written

Last tail-value written to DB-reg. init to UINT16_MAX.

uint16_t head

Completion Queue Head Pointer.

uint8_t phase
uint8_t _rsdv[3]
struct nvme_request_pool *rpool

Command Identifier tracking and user-callback.

struct hostmem_heap *heap

For allocation / free of DMA-capable SQ/CQ entries.

nvme_command.h

Rudimentary Representation of Commands and Completions

This header defines minimal representations of NVMe commands and their completions, suitable for low-level or embedded NVMe driver implementations.

Version

0.7.0

struct nvme_completion
#include <nvme_command.h>

Public Members

uint32_t cdw0
uint32_t rsvd
uint16_t sqhd
uint16_t sqid
uint16_t cid
uint16_t status
struct nvme_command
#include <nvme_command.h>

Public Members

uint8_t opc
uint8_t fuse
uint16_t cid
uint32_t nsid
uint64_t rsvd2
uint64_t mptr
uint64_t prp1
uint64_t prp2
uint32_t cdw10
uint32_t cdw11
uint32_t cdw12
uint32_t cdw13
uint32_t cdw14
uint32_t cdw15

nvme_request.h

NVMe Request Abstraction

This header defines a minimal software abstraction for managing NVMe command identifiers (CIDs) in user space. The abstraction uses a fixed-size pool of struct nvme_request, each assigned a CID, along with a freelist-based allocator for constant-time allocation and release.

This is not part of the NVMe specification, but is useful for tracking user-submitted commands while they are in flight and associating user-defined metadata with each command.

Caveat

assert() is used here. thus instead of a segfault, you will get a nice message like::

nvme_request_get: Assertion cid < NVME_REQUEST_POOL_LEN failed.

Of course, this comes at a cost, so, make sure e.g. meson disables assert on release builds.

The stack implementation has an upper-bound of NVME_REQUEST_POOL_LEN elements.

Version

0.7.0

Defines

NVME_REQUEST_POOL_LEN

Functions

static inline void nvme_request_pool_init(struct nvme_request_pool *pool)

Initialize a request-pool.

When intending to use PRPs associated with the commands, then also use:

  • nvme_request_pool_{init,term}_prps()

static inline void nvme_request_pool_term_prps(struct nvme_request_pool *pool, struct hostmem_heap *heap)
static inline int nvme_request_pool_init_prps(struct nvme_request_pool *pool, struct hostmem_heap *heap)
static inline void nvme_request_pool_term_prps_dmamem(struct nvme_request_pool *pool, struct dmamem_heap *heap, size_t prp_offset)

Release the PRP-list scratch region held by a dmamem-backed request pool.

The counterpart to nvme_request_pool_init_prps_dmamem. The caller provides the same heap and the prp_offset returned by init; the pool struct itself is caller-owned.

static inline int nvme_request_pool_init_prps_dmamem(struct nvme_request_pool *pool, struct dmamem_heap *heap, size_t *prp_offset_out)

Populate a request pool with per-request PRP-list scratch from a dmamem_heap.

Sibling of nvme_request_pool_init_prps: same layout (one 4 KiB page per request, addressed through pool->reqs[i].prp / .prp_addr). The scratch region is virtually contiguous, so the per-request VA is a plain stride, but its IOVA is resolved per page via dmamem_heap_at_iova: on a LUT dmamem the NVME_REQUEST_POOL_LEN * 4 KiB scratch spans several hugepages whose physical pages are not contiguous, so translating the base once and adding a stride would corrupt every entry past the first hugepage boundary. Arithmetic dmamems resolve to the same linear result.

Parameters:

  • pool – Caller-owned request pool (already nvme_request_pool_init’d).

  • heap – dmamem_heap the scratch region is carved from.

  • prp_offset_out – Heap offset of the scratch region, for later term.

Returns:

0 on success, negative errno on allocation failure.

static inline struct nvme_request *nvme_request_alloc(struct nvme_request_pool *pool)

Allocates a request object from the pool.

The returned request has a valid CID and may be used for command submission.

Parameters:

  • pool – The request pool to allocate from.

Returns:

On success, a pointer to a request is returned. On error, NULL is returned and errno set to indicate the error.

static inline void nvme_request_free(struct nvme_request_pool *pool, uint16_t cid)

Free a request previously allocated with nvme_request_alloc().

This marks the cid (command-identifier) as available for reuse.

The cid must no longer be referenced in any submission or completion queue &#8212; that is, the associated command must be fully completed, and any processing of the completion must be done. Only then is it safe to free the request.

If you’re using submit-on-completion (i.e., reusing the request immediately), there is no need to call this function &#8212; the cid is implicitly reused.

Parameters:

  • pool – The request pool the cid came from.

  • cid – The command identifier to mark as available again.

static inline struct nvme_request *nvme_request_get(struct nvme_request_pool *pool, uint16_t cid)

Retrieve the request object associated with the given ‘cid’.

The intended purpose here is to obtain the request-object associated with a command upon its completion.

Parameters:

  • pool – The request pool the CID belongs to.

  • cid – The command identifier.

Returns:

Pointer to the corresponding request object.

static inline uint64_t nvme_request_prp_retranslate(struct hostmem_heap *heap, void *virt)

Re-translate a PRP entry that crosses a hugepage boundary.

Kept out-of-line so the rare boundary path does not inline a v2p into the hot contig builder; the common in-hugepage case strides physically from PRP1, and the inlined stride loop keeps large-I/O builds free of any per-entry call.

static inline void nvme_request_prep_command_prps_contig(struct nvme_request *request, struct hostmem_heap *heap, void *dbuf, size_t dbuf_nbytes, struct nvme_command *cmd)

Prepare the PRP entries for a command with a contiguous data buffer.

Describes dbuf in the command’s PRP1/PRP2 fields, building a PRP list in request when the buffer spans more than two pages, so the controller can access it while the command is in flight.

static inline void nvme_request_prep_command_prps_iov(struct nvme_request *request, struct hostmem_heap *heap, struct iovec *dvec, size_t dvec_cnt, struct nvme_command *cmd)

Prepare the PRP list for a command with an iovec (scatter-gather) data buffer.

This function initializes the Physical Region Page (PRP) entries in the given NVMe command (cmd) using the provided request and an array of iovec entries. Each iovec entry is assumed to be page-aligned and allocated from the given heap.

static inline int nvme_request_prep_command_prps_contig_dmamem(struct nvme_request *request, struct dmamem *dmem, void *dbuf, size_t dbuf_nbytes, struct nvme_command *cmd)

Prepare the PRP entries for a command whose contiguous data buffer lives inside a dmamem_heap.

Sibling of nvme_request_prep_command_prps_contig for the dmamem world. Behaviour depends on the underlying dmamem’s translator:

ARITHMETIC: the heap sits on one contiguous IOAS mapping, so every subsequent page is at prp1 + i * pagesize. Single dmamem_va_to_iova at the start; the rest is arithmetic.

LUT: pages inside a hugepage still stride physically from prp1, but crossing a hugepage boundary requires re-translation. Same shape as the hostmem_heap variant above: stride within the granule, dmamem_va_to_iova (via the registry table) at each boundary.

The translator dispatch is one predictable compare; a process runs one translator for its lifetime, so it predicts perfectly after warmup.

static inline int nvme_request_prep_command_prps_iov_dmamem(struct nvme_request *request, struct dmamem *dmem, struct iovec *dvec, size_t dvec_cnt, struct nvme_command *cmd)

Prepare the PRP entries for a command with a scatter-gather data buffer whose iovec elements live inside a dmamem_heap.

Sibling of nvme_request_prep_command_prps_iov for the dmamem world. Each iovec base is translated via dmamem_va_to_iova so both ARITHMETIC and LUT dmamems work.

struct nvme_request
#include <nvme_request.h>

Public Members

uint16_t cid

The NVMe command identifier.

uint8_t rsvd[6]
void *user

An arbitrary pointer for caller to pass on to completion.

uint64_t prp_addr

Use this when constructing command.PRP2.

void *prp

Use this when constructing the PRP-list itself.

struct nvme_request_pool
#include <nvme_request.h>

Public Members

struct nvme_request reqs[NVME_REQUEST_POOL_LEN]
uint16_t stack[NVME_REQUEST_POOL_LEN]
size_t top
void *prps

Pointer to pre-allocated memory directly mapped to each reqs.