vllm

library0.28.0pypypiunverified

vLLM is a high-throughput and memory-efficient inference and serving engine for large language models (LLMs). It utilizes various optimization techniques, such as PagedAttention, to significantly improve LLM serving performance. Currently at version 0.19.0, vLLM maintains a rapid release cadence with frequent updates and new feature additions.

llm-agents ai-ml

pip install vllm

github ↗package ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · VLLM

vllm

llm-agentspythonv0.28.0

Install

—

Import

—

Disk

—

Pass rate

0/ 10

Env Coverage0 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v? · pip install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

build_error

glibc

py 3.10–3.910 runs

timeout

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

LLM

✓ from vllm import LLM

SamplingParams

✓ from vllm import SamplingParams

This quickstart demonstrates how to initialize the vLLM engine with a specified model and generate text for multiple prompts using custom sampling parameters. Ensure you have a CUDA-enabled GPU and adequate VRAM for the chosen model. The model will be downloaded from Hugging Face if not available locally.

import os from vllm import LLM, SamplingParams # For demonstration, use a small model. Replace with your desired model, e.g., 'mistralai/Mistral-7B-Instruct-v0.2' # If the model is not found locally, vLLM will attempt to download it from Hugging Face. # Ensure you have sufficient GPU memory for the chosen model. model_name = os.environ.get("VLLM_MODEL", "facebook/opt-125m") # Initialize the LLM engine llm = LLM(model=model_name) # Prepare prompts and sampling parameters prompts = [ "Hello, my name is", "The capital of France is", "Write a short poem about a cat." ] sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=100) # Generate text outputs = llm.generate(prompts, sampling_params) # Print the results for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

vllm --version

Debug

Known issues

breakingStarting from v0.14.0, asynchronous scheduling is enabled by default. This changes the execution flow and might affect existing scripts. Some configurations like pipeline parallel, CPU backend, and certain speculative decoding methods are not yet supported with async scheduling.

fix

To disable async scheduling and revert to the synchronous behavior, use the `--no-async-scheduling` flag when running the vLLM server or set `enable_async=False` in Python API calls where applicable.

affects: >=0.14.0

breakingvLLM v0.14.0 introduced a hard requirement for PyTorch 2.9.1 and the default wheels are compiled against CUDA 12.9. Using older PyTorch versions or incompatible CUDA versions can lead to installation failures or runtime errors.

fix

Ensure your PyTorch installation matches the required version and CUDA compatibility. Consider using `pip install vllm --extra-index-url https://download.pytorch.org/whl/cuXX` where `XX` is your CUDA version (e.g., `cu121`, `cu122`, `cu129`).

affects: >=0.14.0

gotchaUsers on CUDA 12.9+ may encounter `CUBLAS_STATUS_INVALID_VALUE` errors. This is often caused by a CUDA library mismatch with the installed PyTorch.

fix

Try removing system CUDA paths from `LD_LIBRARY_PATH` (e.g., `unset LD_LIBRARY_PATH`), or install vLLM with `uv pip install vllm --torch-backend=auto`, or specify the PyTorch CUDA wheel during installation (e.g., `pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129`).

affects: >=0.17.0

gotchaSupport for new model architectures, like Gemma 4 in v0.19.0, often requires a minimum `transformers` library version. Missing this requirement can lead to model loading failures.

fix

Upgrade your `transformers` library to the recommended version. For Gemma 4, `transformers>=5.5.0` is required. Always check the release notes for specific model requirements.

affects: >=0.19.0

gotchaServing Qwen3.5 models with FP8 KV cache on B200 GPUs in v0.18.0 was noted to have degraded accuracy.

fix

If encountering accuracy issues with this specific model/hardware/quantization combination, consider using a different KV cache precision (e.g., FP16) or consult later patch releases for potential fixes.

affects: 0.18.0

Errors

Common errors & fixes

CMake Error at build/lib.linux-x86_64-cpython-310/vllm/attention/CMakeLists.txt:10 (find_package): Could not find a package configuration file provided by "nvcc" with any of the following names: nvccConfig.cmake nvcc-config.cmake

vLLM's installation requires the NVIDIA CUDA Toolkit's `nvcc` compiler to be correctly installed and accessible in the system's PATH.

fix

Ensure the appropriate NVIDIA CUDA Toolkit version (compatible with your GPU and PyTorch) is installed and its `bin` directory is added to your system's PATH. Verify with `nvcc --version`.

ModuleNotFoundError: No module named 'vllm.entrypoints.llm'

The `LLM` class was moved directly into the top-level `vllm` package in recent versions (e.g., 0.19.0), invalidating the old `vllm.entrypoints.llm` import path.

fix

Update your import statement from `from vllm.entrypoints.llm import LLM` to `from vllm import LLM`.

RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU Y; Z MiB total capacity; A MiB already allocated; B MiB free; C MiB reserved in total by PyTorch)

The chosen model, batch size, or maximum sequence length configuration exceeds the available GPU VRAM, causing an out-of-memory error.

fix

Reduce the `max_model_len`, `max_num_batched_tokens`, `max_seqs`, or `gpu_memory_utilization` parameters when initializing `LLM`, or use a smaller model.

Upgrade

Version history

0.28.0latest on PyPI · released Aug 26, 2026

Audit

Dependencies

transformersoptionalRequired for specific model architectures like Gemma 4, typically >=5.5.0.

torchrequiredCore deep learning framework. Specific CUDA versions (e.g., cu121, cu129) are critical for GPU compatibility.

Agent activity

64 hits · last 30 days

node

Perplexity

OpenAI (training)

Resources

githubhttps://github.com/vllm-project/vllm ↗

packagehttps://pypi.org/project/vllm/ ↗

homepagehttps://vllm.io ↗

docshttps://docs.vllm.ai/en/latest/ ↗