Inference at the
Speed of Silicon

Overview

Inferviqo is a unified inference engine that bridges the memory efficiency of vLLM's PagedAttention with the high-throughput request handling of SGLang's RadixAttention. Rather than choosing between memory or speed, you get both — in a single Python library with a clean API.

The engine is designed for production ML serving infrastructure: it ships with hardware-specific kernels, continuous batching, Prometheus telemetry, and full NVFP4/MXFP4 quantization support out of the box.

System Requirements

To fully utilize Inferviqo's custom kernels, the following hardware is recommended:

• NVIDIA: GB300, GB200, H100 SXM5/PCIe — for NVFP4/MXFP4 kernel optimizations
• AMD: MI350X, MI355X — full hardware-specific custom kernel compilation
• OS: Linux (Ubuntu 22.04+ recommended)
• CUDA: 12.1 or newer (ROCm 5.7+ for AMD)
• Python: 3.10 or newer
• RAM: ≥ 64 GB system memory recommended for large models

Inferviqo degrades gracefully on older hardware (A100, A6000) using generic kernel paths, but maximum performance requires the above GPU generations.

Installation

Install the core library directly via pip. Pre-compiled wheels for specialized GPU clusters are available in our private registry — contact us for access.

# Install from PyPI
pip install inferviqo

# Verify installation
python -c "import inferviqo; print(inferviqo.__version__)"

Quick Start

This example loads a model, applies hardware-specific kernel optimizations, and runs generation using continuous batching and radix cache warm-up.

from inferviqo import QuantizedModel, InferenceEngine

# 1. Load your model from HuggingFace Hub or a local path
model = QuantizedModel.load("deepseek-v4-67b")

# 2. Apply hardware-specific kernel optimizations for your GPU
model.optimize_kernels(
    target_gpu="GB300",
    precision="nvfp4"       # or "mxfp4", "fp8", "bf16"
)

# 3. Initialize the inference engine with continuous batching
engine = InferenceEngine(
    model=model,
    enable_radix_cache=True,
    max_batch_size=256,
    max_seq_len=32768
)

# 4. Generate — engine handles batching automatically
response = engine.generate(
    "Explain the principles of PagedAttention.",
    max_tokens=1024,
    temperature=0.7
)
print(response.text)

QuantizedModel

Loads and quantizes a model checkpoint. Supports HuggingFace Hub IDs and local directory paths.

class QuantizedModel:

    @classmethod
    def load(cls, model_path_or_id: str, **kwargs) -> 'QuantizedModel':
        """Load a model from HuggingFace Hub or local path."""
        ...

    def optimize_kernels(self, target_gpu: str, precision: str = "nvfp4") -> None:
        """Compile and cache hardware-specific CUDA/ROCm kernels."""
        ...

QuantizedModel.load() — Parameters

optimize_kernels() — Parameters

InferenceEngine

The core serving engine. Manages KV cache allocation, continuous batching, and the radix prefix cache across concurrent sessions.

generate() — Parameters