Offline Inference

To facilitate users in quickly using xLLM for offline inference, we provide Python script examples for launching offline inference.

LLM

LLM inference example: https://github.com/xLLM-AI/xllm/blob/main/examples/generate.py

Embedding

Generate embedding example: https://github.com/xLLM-AI/xllm/blob/main/examples/generate_embedding.py

VLM

VLM inference example: https://github.com/xLLM-AI/xllm/blob/main/examples/generate_vlm.py