Can I start a complete HTTP service with the offline example?

No. The documented release supplies offline inference. A complete online system still needs service integration and components such as full KV cache management, stage orchestration and replica-level load balancing.

Does DeepInfer support R1, V3 or every DeepSeek model?

The checked README lists DeepSeek-V4.1-Flash. Do not infer support for other model families from the framework name. Confirm the current repository and model-specific examples before choosing weights.

Can I run this on a consumer GPU or native Windows?

The checked prerequisites specify Linux, NVIDIA SM100 or Huawei Ascend 950 and matching toolchains. This guide does not establish native Windows/macOS or consumer-GPU compatibility, and eight-device examples are not single-device instructions.

Are the framework and model weights covered by the same license?

DeepInfer is released under Apache 2.0. Model weights and third-party components have their own terms; consult the linked model card and dependency licenses separately. Framework licensing does not replace model licensing.

Has this guide reproduced the published benchmarks?

No. It explains official documentation and links to the original charts and scripts. No GPU benchmark or model download was performed for this page. Reproduction requires matching resources, configuration and metric definitions.

DeepSeek · Open-source inference framework

DeepInfer: architecture, setup and offline inference

DeepInfer is an inference framework: software that executes model weights. Start with supported hardware, understand the released engine, then follow the documented Linux build and offline examples.

Official DeepInfer repository

Independent documentation guide. We have not installed DeepInfer or reproduced its GPU benchmarks. This page does not host inference, execute commands or download model weights.

Sources checked: 2026-10-25

Check support before installing

The current README lists the following platforms and model family. Support for older DeepSeek models, other accelerators, native Windows or macOS is not established by that list.

Documented prerequisites · not a promise that any one device can fit the model
Requirement	NVIDIA path	Ascend path
Supported model family	DeepSeek-V4.1-Flash	DeepSeek-V4.1-Flash
Accelerator	NVIDIA SM100	Huawei Ascend 950
Operating system	Linux	Linux
Python environment	Python 3.12	Python 3.12
Compiler	GCC ≥ 13.0	GCC ≥ 13.0
Platform toolkit	CUDA 13	CANN 9.2

Device support and memory capacity are different checks. Confirm device count, memory, interconnects and checkpoint requirements for your configuration. Consumer GPUs and a Python installation alone do not satisfy these prerequisites.

Official README

How the offline engine fits together

This simplified flow describes an offline run. It does not represent a complete production HTTP service or a full cluster topology.

01
Offline requests
Prompts and generation parameters
02
Scheduling
Organize batches and execution work
03
Model execution
Prefill context, then decode tokens
04
Sampling
Select tokens under configured rules
05
Output
Generated sequences and diagnostics

Optimized kernels

A kernel is a device-level compute routine. DeepInfer combines specialized kernel libraries with its execution runtime; examine the linked README for the current library set and platform support.

Attention DP / CP, MoE EP / EPLB

DP means data parallelism; CP means context parallelism. MoE is mixture of experts, EP distributes experts, and EPLB balances expert work. These describe engine parallelism, not a ready-made service load balancer.

Speculative decoding and SWA replay

Draft tokens can be proposed and verified together. DSpark schedules verification using confidence. SWA means sliding-window attention; bounded replay reduces repeated prefill work by retaining a suitable suffix. Advanced settings depend on model configuration.

Scheduling, sampling and structured output

Asynchronous scheduling overlaps CPU scheduling and device execution. Structured output constrains generation to formats such as JSON schema, regex or EBNF grammar. Batch-invariant execution and deterministic sampling have separate configuration requirements.

The README currently marks batch-invariant inference and deterministic sampling as NVIDIA-only. Deterministic sampling additionally requires the documented settings and a fixed sequence seed. This tutorial does not enable advanced optimization flags by default.

Official README Inference deployment Q&A

Released engine vs complete online service

Read the release scope separately from the deployment Q&A’s description of DeepSeek’s production system. Not every production component described there is shipped as part of the repository.

Capabilities and the remaining integration work
Area	Released engine capability	For a complete online service
Execution	Offline generation, model execution, scheduling and sampling.	Request ingress, service lifecycle and operational integration; do not treat the offline example as an HTTP server.
Cache	Engine-level KV use and bounded replay capabilities.	The complete KV cache system listed as outside this release, including the infrastructure-coupled management layer.
Parallelism and balancing	MoE expert parallelism and expert-parallel load balancing (EPLB).	Load balancing across data-parallel replicas for prefill and decode.
Stage separation	Much of the required engine functionality is implemented.	Deployment and orchestration of disaggregated vision encoding, prefill and decode.

DeepSeek Recipe is a separate protocol conversion and token encoding/decoding library. Its existence should not be read as a bundled, production-ready API server in the offline DeepInfer release.

DeepSeek Recipe protocol library Inference deployment Q&A

Build from source on Linux

Use a machine that meets the prerequisites above. Commands below are documentation examples only. Choose one accelerator platform and use a separate, empty Python environment for that build.

1. Clone and activate Python 3.12

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
python3.12 -m venv .venv
source .venv/bin/activate

Remain in the DeepInfer repository root with the virtual environment active. Review the current build script before execution, record your source revision and model revision, and reserve sufficient disk and memory for dependencies and weights.

The main path assumes SM100 hardware and CUDA 13 are already prepared. Keep DEEPINFER_ACCELERATOR exported in the same environment when running DeepInfer. The build installs dependencies and compiles vendored components.

2. NVIDIA / Linux build

export DEEPINFER_ACCELERATOR=nvidia
./build.sh

For Ascend 950 with CANN 9.2, use ascend in a separate matching environment instead of running the NVIDIA build. Do not combine the two toolchains or expect the CUDA-specific steps to apply.

Alternative: Ascend / Linux build

export DEEPINFER_ACCELERATOR=ascend
./build.sh

Source build script Official README

Run an offline inference example

Default checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017. A first run can download weights into the Hugging Face cache. This is a large-model run, not a lightweight CPU smoke test.

The example explicitly selects eight NVIDIA devices and uses --ep 8. It assumes those devices are present and have sufficient resources. It is not a universal single-card command. Ascend users should follow the platform-specific official examples and configure their device visibility accordingly.

NVIDIA example · eight matching visible devices

export DEEPINFER_ACCELERATOR=nvidia
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
python examples/offline_inference.py --ep 8

Use an existing local checkpoint

export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py --ep 8

Run the local-checkpoint alternative in the same configured NVIDIA shell. Replace /path/to/DeepSeek-V4.1-Flash-1017 with a real compatible Hugging Face checkpoint directory containing config.json and weight files. The environment variable changes the default model path; it does not override an explicitly supplied --model_name.

Official examples Offline inference script Model weights and model license

Read the performance results correctly

The official README’s prefill chart uses 4 SM100 GPUs and its decode chart uses 64, each within one scale-up domain. These configurations describe the published charts, not a universal minimum or a benchmark we ran.

Stage, metric and interpretation
Metric	Unit / scope	How to interpret it
Prefill throughput	tokens/s/GPU	Rate of processing input tokens. Read alongside input length, cache coverage and batch size; it is not complete request latency.
Decode avg_SPS	steps/s	Steady-state decoding steps per second. A step is not necessarily one accepted output token when speculative decoding is enabled.
Decode throughput accounting	Full-acceptance convention	The example reports throughput using a full draft-acceptance convention. Compare draft length and true acceptance rate before treating it as delivered output tokens/s.
Per-user speed and latency	Per request / workload	High aggregate throughput does not guarantee fast replies for each request. Compare the same hardware, concurrency, context and measurement boundaries.

Record the source revision, hardware, device count, parallelism, batch settings, cached prefix, draft length and acceptance assumptions when reproducing a curve. Profiling perturbs execution; do not derive steady-state performance from profiler trace timings. No universal speedup or API price follows from these charts.

Official prefill chart Official decode chart Benchmark reproduction script Official examples

Advanced options are not installation defaults

DSpark, confidence-based verification, SWA bounded replay and reproducible sampling each have model and platform conditions. Start with the documented example, then inspect the README and example configuration before changing optimization flags or interpreting a result difference.

Official README Official examples

Troubleshooting checklist

Unsupported hardware: Check the accelerator architecture against the current README before compiling. A successful Python installation does not establish kernel compatibility.
Build or compiler failure: Confirm Linux, GCC, Python and the matching CUDA/CANN versions. Keep the environment isolated and inspect the first failing build step rather than installing unrelated toolchain versions.
Platform variable missing: Export DEEPINFER_ACCELERATOR as nvidia or ascend in the activated environment, and retain it for subsequent execution. Choose the platform matching the installed hardware and libraries.
Checkpoint cannot be resolved: Use a valid model ID or an existing compatible directory with config.json and weights. Check permissions and the explicit --model_name argument before assuming the environment override is active.
Wrong device count or insufficient memory: Check visible devices, parallel configuration and model memory requirements. Reducing a device count is not automatically valid; choose a documented configuration that fits available hardware.

Frequently asked questions

Official sources and model background

Verify current support and commands before execution. The repository can change after the date below. Our linked model article provides background, not additional hardware compatibility guarantees.