Python consumers¶
The Python bindings of a C library are used together with other Python
libraries, and those libraries set requirements on the bindings: how memory
is passed between them without a copy, who frees it, when it may be reused,
and which thread waits for IO. This page surveys the libraries that the
bindings of a storage library such as xNVMe are used with, and lists the
requirements they set. The requirements motivate the design of the
python and cython targets.
The libraries¶
- NumPy
Arrays in host memory. NumPy takes and exports memory through the buffer protocol of PEP 3118, the array interface, and DLPack (
numpy.from_dlpack()). An array made withnumpy.frombuffer()shares the memory of the object it is made from and keeps a reference to that object as itsbase.- PyTorch
Tensors in host or GPU memory.
torch.frombuffer()makes a tensor that shares the memory of an object with the buffer protocol, and its behaviour is undefined for memory that is not on the CPU (torch.frombuffer).torch.from_dlpack()takes memory on any device. Host memory that a GPU copies from asynchronously must be pinned; memory that PyTorch did not allocate is pinned withtorch.cuda.cudart().cudaHostRegister()(pinned memory). The caching allocator of PyTorch reuses the memory of a tensor as soon as the tensor is freed.- CuPy
Arrays in GPU memory, exchanged through the CUDA array interface and DLPack.
cupy.cuda.UnownedMemorytakes a pointer, a size and an owner, an object that CuPy keeps referenced while it uses the memory. CuPy also allocates pinned host memory,cupy.cuda.alloc_pinned_memory()(CuPy memory).- PyCUDA
GPU memory, and page-locked host memory as NumPy arrays:
pycuda.driver.pagelocked_empty()allocates one, andpycuda.driver.register_host_memory()page-locks the memory of an existing array (PyCUDA).- RAPIDS
GPU dataframes and arrays: cuDF, cuML and others, which allocate device memory through RMM. RMM allocates in stream order, and its
rmm_cupy_allocatorwraps anrmm.DeviceBufferin a CuPyUnownedMemory(RMM and CuPy). The libraries exchange arrays through the CUDA array interface.- KvikIO
The file IO of RAPIDS, through GPUDirect Storage (cuFile) where it is available, and through POSIX IO otherwise.
CuFile.pread()andCuFile.pwrite()take a host or device buffer, split the IO into tasks run by a pool of threads, and return anIOFuture, whoseget()waits for the IO and returns the number of bytes (KvikIO). GPUDirect Storage is used only for IO aligned to a page.- Triton
GPU kernels written in Python. A kernel argument that has a
data_ptr()method and adtypeattribute is passed to the kernel as a pointer (triton.jit).- CuTe DSL
The Python DSL of CUTLASS for GPU kernels.
cute.runtime.from_dlpack()makes a tensor of the memory of any DLPack tensor, without a copy. The tensor is valid only while the tensor it is made from is alive, andassumed_alignmust divide its address (CuTe DSL).- ggml
A C library of tensors for machine learning, the base of llama.cpp. Its Python bindings,
llama-cpp-pythonandggml-python, are ctypes bindings written by hand. Theggufpackage of llama.cpp reads a GGUF model file throughnumpy.memmap, and each tensor it reads is a view of the mapped file.- vLLM
LLM inference on PyTorch. A KV connector moves the KV cache of vLLM to and from other storage, through methods vLLM calls during the forward pass:
start_load_kv(),wait_for_layer_load(),save_kv_layer()andwait_for_save().wait_for_save()blocks until every save is done, so that the paged KV buffer is not overwritten before it is saved (vLLM KV connector). The API is marked experimental. vLLM can run a worker process for each GPU.- LMCache
A KV cache layer for vLLM, which stores KV cache in CPU memory, on local disk, through GPUDirect Storage, and in remote stores. A storage backend implements
submit_put_task(), which stores aMemoryObjand returns a future,submit_prefetch_task()andget_blocking()(LMCache backend). The hot cache of LMCache is a pinned CPU buffer.- Jupyter
Interactive use. IPython shows an object by its
_repr_pretty_(),_repr_html_()or other_repr_*_()method, and byrepr()otherwise (IPython display). Help and completion read docstrings and signatures.
The requirements¶
Memory the library allocates, such as a buffer for IO, is an object with the buffer protocol, so NumPy, PyTorch, KvikIO and the bindings of other libraries use it without a copy. The object frees the memory when it is collected, and not while an array made from it is alive.
A parameter that C reads or writes as an array of bytes takes any object with the buffer protocol, its size in bytes.
Memory a GPU reads or writes asynchronously is pinned. Memory the library allocates may be pinned already, or be pinned by the caller with
cudaHostRegister(). A copy that the bindings make is neither, so the bindings pass the caller’s memory to C, not a copy.GPU memory is exchanged through DLPack and the CUDA array interface, which NumPy, PyTorch, CuPy, RAPIDS and the CuTe DSL take; the buffer protocol covers host memory only. Both carry a stream, on which the consumer synchronizes before using the memory.
Memory given to an IO that completes after the call returns is kept alive until the IO completes. The caching allocator of PyTorch reuses freed memory at once, and vLLM and LMCache wait for saves to complete before the memory is reused. KvikIO and LMCache return a future for each IO.
Waiting for IO does not hold the GIL, so that the Python threads of vLLM and LMCache run meanwhile, and KvikIO waits in threads of its own.
A process opens its own devices and queues, as vLLM can run a process for each GPU.
Sizes and addresses are aligned as the device requires. GPUDirect Storage and the CuTe DSL require alignment too.
An object shows its value in Jupyter, and every function has a docstring and a signature.
The bindings today¶
Requirements 1, 2 and 3 are met by both targets. Memory that <stem>_alloc
allocates is a Buffer, and a void * @arrparam and a registration
take the memory of the caller’s object, not a copy. The cython module also
takes any buffer for an array of numbers. Requirement 4 is met in one
direction: the bindings take a DLPack tensor and a CUDA array, but a
Buffer gives its memory through the buffer protocol only, as nothing in
the C API says whether the memory is a device’s. The bindings ask
__dlpack__() for no stream, so the caller synchronizes before the call.
Requirement 9 is met: records print as YAML, and the functions of both modules
have docstrings and signatures. Under requirement 6, the cython module
holds the GIL during a call; Cython code that calls C through the .pxd may
release it.
Requirement 5 is not met. The bindings keep memory that C uses after a call returns alive only while it is registered. Requirements 7 and 8 are for the caller and the C library; the bindings do not check them.