xLLM uses gflags to manage service startup parameters. --model <PATH> is the only required flag. For native configuration, explicitly supplied command-line flags override JSON configuration values, which override compiled defaults. The tables below are grouped by the Config classes in /xllm/core/framework/config, with one Config per section. The ConfigJsonUtils section contains the common JSON config-file flags.
Backend model type: llm for text-only models, vlm for multimodal models, or dit for diffusion models.
task
string
"generate"
Model task, for example generate, embed, or mm_embed.
enable_return_embedding_modality_tags
bool
false
Whether full sequence embeddings include per-token modality tags.
limit_image_per_prompt
int32
8
Maximum number of images per prompt. Only applies to multimodal models.
max_encoder_cache_size
int64
0
Maximum GPU/NPU memory size in MB for the encoder cache per worker. 0 disables the encoder cache.
reasoning_parser
string
""
Reasoning parser, for example auto, glm45, glm47, glm5, qwen3, qwen35, or deepseek-r1.
tool_call_parser
string
""
Tool-call parser, for example auto, qwen25, qwen3, qwen35, qwen3_coder, kimi_k2, deepseekv3, deepseekv32, deepseekv4, deepseekv41, glm45, glm47, or glm5.
enable_qwen3_reranker
bool
false
Whether to enable the Qwen3 reranker.
flashinfer_workspace_buffer_size
int32
134217728
Reserved FlashInfer workspace buffer for intermediate attention results in split-k attention. Default is 128 MiB.
enable_return_mm_full_embeddings
bool
false
Whether VLM models return ViT embeddings and sequence embeddings.
mm_download_headers
string
""
Service-level default HTTP headers for multimodal downloads, as a JSON object. Per-request headers take precedence. Example: {"Authorization":"Bearer xxx"}.
use_audio_in_video
bool
false
Whether to decode both audio and video when the input is a video.
audio_max_upload_file_mb
int32
25
Maximum speech-to-text audio upload size in MiB. Must be positive.
audio_max_decode_duration_s
int32
600
Maximum decoded speech-to-text audio duration in seconds. 0 disables the limit.
use_cpp_chat_template
bool
true
Use native C++ chat templates for supported models, for example deepseek_v32, deepseek_v4, and deepseek_v41. Set to false to fall back to Jinja for debugging.
Enable VMM for model weights. Independent of enable_vmm_kvcache. NPU only.
enable_rolling_load
bool
false
Stream DiT layer weights through a fixed number of HBM slots. NPU only.
rolling_load_num_cached_layers
int32
2
Number of decoder layer weight slots to keep in HBM when enable_rolling_load=true.
rolling_load_num_rolling_slots
int32
-1
Number of rolling slots used by decoder rolling load. Fixed slots are rolling_load_num_cached_layers - rolling_load_num_rolling_slots. -1 means auto, min(2, preload_count). Must be in [-1, rolling_load_num_cached_layers].
Maximum KV Cache size in bytes. With virtual memory, caps the shared KV/weight pool. 0 leaves the explicit byte ceiling unset and derives capacity from available memory.
max_memory_utilization
double
0.8
Fraction of device memory used for model inference, including model weights and KV Cache. Must be in (0, 1] for physical KV Cache and fixed VMM. Elastic VMM ignores this ratio during startup sizing, arena reservation, and physical growth; capacity follows live available memory and observed execution headroom.
kv_cache_dtype
string
"auto"
KV Cache dtype for quantization. auto aligns with model dtype and disables quantization. int8 enables INT8 quantization and is only supported on the MLU backend.
indexer_cache_dtype
string
"auto"
Indexer cache dtype for models with an indexer cache. Supported values are auto and int8. auto aligns with model dtype and disables indexer cache quantization. int8 enables INT8 indexer cache quantization.
enable_prefix_cache
bool
true
Whether to enable prefix cache in the block manager. See Prefix Cache.
kv_events_config
string
""
KV residency event publisher configuration as a JSON object string. An empty string disables reporting. See the fields and example below.
enable_in_batch_prefix_cache
bool
false
Whether to cache admitted prefill full blocks into the prefix cache so that later requests in the same batch can share them.
max_linear_state_cache_slots
int64
0
Maximum number of active linear-attention state cache slots. 0 derives an automatic capacity from the available KV Cache budget.
xxh3_128bits_seed
uint32
1024
Default XXH3 128-bit hash seed.
enable_vmm_kvcache
bool
false
Enable VMM for KV Cache. Independent of enable_vmm_weights; false uses the original physical KV Cache implementation. NPU only.
enable_shared_kv_cache
bool
false
Share one fixed physical arena between target Full KV, Linear state extents and compatible Full-only MTP draft KV. Target and draft use separate bank views with the same final page indices; Linear state can reuse space in draft banks. Draft Full row geometry must match the target, and draft layers must fill whole companion banks. Native NPU Torch Qwen hybrid execution remains limited to ordinary eager decoding. The Python path supports eager/chunked prefill, ACL decode/verify and ordinary Qwen3.5/3.8 MTP.
vmm_elastic
bool
true
When either VMM flag is enabled, grow active logical slots and physical backing on demand, and release idle backing. false uses a fixed resident pool.
vmm_memory_reserve_bytes
int64
0
Additional device memory to keep free when elastic virtual memory grows, in bytes. Supplements runtime execution headroom; must be nonnegative.
vmm_startup_timeout_ms
int32
120000
Positive timeout in milliseconds applied separately to the master’s node-registration wait and each non-master’s physical-pool readiness wait during VMM startup. Independent of per-request RPC timeouts; each wait has its own timer.
phy_page_granularity_size
int64
2097152
Granularity size of one physical page in bytes, default 2 MiB, for continuous KV Cache.
The following fields belong to the JSON object in kv_events_config:
Field
Type
Default
Description
publisher
string
"zmq"
Publisher type. Currently only zmq is supported.
endpoint
string
Required
Base ZeroMQ publish address for vLLM-style KV events, for example tcp://*:5557. Must be a nonempty TCP address.
replay_endpoint
string
""
Base ZeroMQ address for snapshot recovery requests, for example tcp://*:6557. An empty string disables this endpoint. Recovery replies contain the current full snapshot and its matching live stream sequence.
motor_pool_endpoint
string
""
Base ZeroMQ publish address for Motor local Host copy confirmation events, for example tcp://*:7557. An empty string disables this stream.
topic
string
""
Published ZeroMQ topic. Consumer subscription filters must match this topic.
hwm
int32
10000
ZeroMQ send queue high-water mark in messages. Must be positive.
snapshot_interval_seconds
int32
60
Interval between periodic full residency snapshots, in seconds. Must be positive.
Each DP rank uses the base TCP port plus dp_rank. The event, replay and Host confirmation port ranges must not overlap, and every port must be between 1 and 65535. For example, with DP=2 and endpoint=tcp://*:5557, the two ranks publish on 5557 and 5558.
Add the following flag to an existing launch command to enable event publication and snapshot recovery:
Current support covers ordinary non-PD text inference with prefix caching enabled, flat KV layouts and local Host offload. Events describe the Device/Host residency of reusable prefix blocks; Host copies are advertised after successful copy acknowledgements. The publisher runs on a background thread. Multi-resource composite/linear-state layouts, multimodal inference, virtual-memory KV, PD separation and external KV stores are not yet supported.
Whether to enable schedule overlap, also known as asynchronous scheduling. See Async Scheduling.
prefill_scheduling_memory_usage_threshold
double
0.95
Memory usage threshold during prefill scheduling.
enable_chunked_prefill
bool
true
Whether to enable chunked prefill.
max_tokens_per_chunk_for_prefill
int32
-1
Maximum number of tokens per chunk in the prefill stage. -1 uses the default policy.
chunked_match_frequency
int32
2
Sequence prefix-cache match frequency.
priority_strategy
string
"fcfs"
Request priority strategy, for example fcfs, priority, or deadline.
enable_mix_batch
bool
true
Whether to run prefill and decode in the same batch. Forced to false when CP or MTP is active.
aggressive_coeff
double
1.0
Aggressive coefficient for MixScheduler urgency judgment.
starve_threshold
double
1.0
Starvation threshold coefficient for MixScheduler.
enable_starve_prevent
bool
true
Whether to enable anti-starvation behavior in MixScheduler.
enable_dp_fair_token_budget
bool
true
Whether to cap each DP group at max_tokens_per_batch / dp_size tokens per scheduling round on disaggregated PD PREFILL instances, floored at one prefill chunk.
request_queue_size
int32
100000
Capacity of the scheduler request queue. Requests arriving while the queue is full are rejected at admission.
Context parallel size. Backend and model support is required; 1 disables context parallelism.
kv_split_size
int32
1
KV Cache split width. 0 falls back to cp_size (legacy). 1 disables KV split: each CP rank stores the full KV and skips the prefix AllGather. Other K values (K divides cp_size) shard KV across K ranks while token-CP still uses cp_size.
tp_size
int64
1
Tensor parallelism size. Only used for DiT models.
sp_size
int64
1
Sequence parallelism size. Only used for DiT models.
cfg_size
int64
1
Classifier-free guidance parallelism size. Only used for DiT models.
vae_size
int64
1
VAE patch parallelism size. Only used for DiT models.
communication_backend
string
"hccl"
NPU communication backend, for example lccl or hccl. Uses hccl when DP is enabled.
enable_mm_encoder_dp
bool
false
Whether to enable encoder data parallelism for multimodal models.
enable_multi_stream_parallel
bool
false
Whether to enable computation/communication parallelism with two streams and two micro batches in the prefill stage. See Multi-Stream Parallelism.
micro_batch_num
int32
1
Number of micro batches used for multi-stream parallelism.
Master address for multi-node distributed serving, for example 10.18.1.1:9999.
worker_memory_master_node_addr
string
"127.0.0.1:19889"
Rank 0 address where worker memory RPC servers register their endpoints, for example 10.18.1.1:9999. Used only when virtual memory is enabled and nnodes > 1.
nnodes
int32
1
Number of multi-node nodes.
node_rank
int32
0
Rank of the current node.
etcd_addr
string
""
etcd address used to save instance metadata.
etcd_namespace
string
""
Optional etcd namespace prefix for all xLLM keys, for example prod-a.
enable_service_routing
bool
false
Whether to enable xLLM service routing.
xservice_rpc_protocol
string
"baidu_std"
Protocol used by XServiceClient to communicate with the service layer: baidu_std or http. Does not change P/D engine RPC or KV transfer protocols.
xservice_address
string
""
Direct HTTP service control address (host:port) for instance registration, heartbeats and metadata queries. Requires xservice_rpc_protocol=http and is mutually exclusive with etcd_addr. Per-request result callbacks use the service address supplied in the request.
Maximum suffix-tree depth for suffix speculative decoding.
speculative_suffix_max_cached_requests
int32
-1
Maximum number of globally cached requests for suffix speculation. -1 means unlimited; 0 disables it.
speculative_suffix_use_tree_spec
bool
false
Whether to use tree-based suffix speculation instead of path speculation.
enable_probabilistic_draft
bool
false
Drafts sample proposals from their distribution (emitting q for rejection sampling). The default is greedy argmax. Only MTP, DSpark, DFlash2, Eagle3 support it.
Whether to use predicted latency for latency-aware scheduling.
profile_max_prompt_length
int32
2048
Maximum prompt length used for profiling.
max_global_ttft_ms
int32
std::numeric_limits<int32_t>::max()
Global TTFT threshold in milliseconds.
max_global_tpot_ms
int32
std::numeric_limits<int32_t>::max()
Global TPOT threshold in milliseconds.
enable_profile_kv_blocks
bool
true
Whether to generate KV Cache blocks for profiling.
disable_ttft_profiling
bool
false
Whether to disable TTFT profiling.
enable_online_profile
bool
false
Enable the online timeline endpoints (/start_profile, /stop_profile). Supports Ascend NPU and the documented profiling backends.
profile_backend
string
"torch"
Online profiling backend. torch (default) records CPU+NPU using torch_npu on Ascend, or CPU+CUDA using Kineto. ascend records native CANN data for export with msprof. cuda toggles the external nsys capture range. See Profiling a Running Service for deployment requirements.
profile_dir
string
""
Output directory for torch and ascend traces. Empty means the current working directory.
Native extension implementation family for registered components, including models and executors. An empty value preserves normal selection. The selected family must be registered; components without registrations in that family keep their normal selection.
enable_graph
bool
false
Whether to enable graph execution for the decode phase to reduce kernel-launch overhead and device idle time. Supports CUDA Graph, ACL Graph (NPU), MLU Graph, and DCU Graph. See Graph Mode.
enable_graph_double_buffer
bool
true
Whether to enable double-buffered ACL graph persistent params and graph instances for NPU schedule-overlap decode.
enable_graph_mode_decode_no_padding
bool
false
Whether decode graph capture uses the actual num_tokens instead of a padded shape.
enable_prefill_piecewise_graph
bool
false
Whether to enable piecewise CUDA graph for the prefill phase. Attention runs in eager mode while other operations are captured in CUDA graphs.
enable_graph_vmm_pool
bool
true
Whether to enable a VMM-backed CUDA graph memory pool for multi-shape graph memory reuse.
max_tokens_for_graph_mode
int32
2048
Maximum number of tokens for graph execution. 0 means no limit.
acl_graph_decode_batch_size_limit
int32
16
Decode batch size threshold for ACL graph on NPU. When the actual decode batch size exceeds this value, ACL graph decode falls back to eager mode to avoid OOM.
block_draft_graph_warmup_max_kv_len
int32
0
Pre-capture power-of-two KV capacity buckets for independent Python block-draft ACL graphs (DSpark/DFlash2, including GLM MLA drafts). 0 covers the full model context; a positive value selects the bucket covering that length; -1 retains lazy capacity capture. Required existing warmup shapes are always retained, even above a positive limit. Requests beyond the warmup limit can still grow graphs lazily. More buckets increase startup time and draft graph memory. Does not change target, MTP, context-write or optional whole-round unified graph warmup.
enable_shm
bool
false
Whether to enable shared memory for model execution.
use_contiguous_input_buffer
bool
true
Whether to use a contiguous device input buffer for model execution.
input_shm_size
uint64
1024
Input shared-memory size. Default is 1GB.
output_shm_size
uint64
128
Output shared-memory size. Default is 128MB.
random_seed
int32
-1
Random seed for the random number generator. -1 means no fixed seed.
enable_dsa_multi_stream
bool
false
Local overlap in the Python GLM indexer: K/weights projections run on a side stream and stay fused when query/cache rows match; the main stream joins before updating the index cache. Non-interleaved RoPE additionally overlaps Q projection/RoPE with cache preparation. Main attention preparation finishes before the indexer. Requires a supported model backend and stream-capable device.
enable_attn_dp_weight_sharding
bool
false
Shard Python GLM MLA head-dependent attention weights across TP and DP. Requires DP > 1, CP = 1, layerwise split = 1, and attention heads divisible by TP × DP. Attention still executes TP-local heads on each token owner, with three collectives per layer. MoE uses its independent MoE TP/EP groups.
Fused MC2 mode for NPU. -1 uses the auto default, 0 disables fused MC2, positive values enable dense matmul-allreduce, 1 uses DispatchFFNCombine for MoE, 2 uses DispatchGmmCombineDecode for MoE.
enable_dspark_native_sas
bool
false
Enable native NPU DSpark SparseAttnSharedkv semantics. Older operators that reject non-empty ori_sparse_indices may terminate during tiling; keep this disabled to use q_len=1 compatibility mode.
