Run a private LLM with vLLM

1. Pick a server#

A 70B-parameter model in FP8 needs about 80 GB of GPU memory plus room for the KV cache. gpu-l40s-4x or one H200 is a good start.

2. Start vLLM#

docker run -d --gpus all -p 8000:8000 \
  -v /models:/models vllm/vllm-openai:latest \
  --model /models/Qwen3-72B-Instruct --tensor-parallel-size 4

3. Call the API#

curl http://10.20.0.15:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "/models/Qwen3-72B-Instruct", "messages": [{"role": "user", "content": "Merhaba!"}]}'

Any OpenAI SDK works — change only the base_url.

Performance tips#

• Use --tensor-parallel-size equal to the number of GPUs
• Enable prefix caching for chat workloads with long system prompts
• Store model weights on local NVMe, not on network storage