# Run a private LLM with vLLM

> Serve an open-weight model behind an OpenAI-compatible API on your own GPUs — data never leaves your server.

Source: https://docs.avenlith.com/en/llm-inference  
Category: AI & GPU  
Last updated: 2026-09-22

## 1. Pick a server

A 70B-parameter model in FP8 needs about 80 GB of GPU memory plus room for the KV cache. `gpu-l40s-4x` or one H200 is a good start.

## 2. Start vLLM

```bash
docker run -d --gpus all -p 8000:8000 \
  -v /models:/models vllm/vllm-openai:latest \
  --model /models/Qwen3-72B-Instruct --tensor-parallel-size 4
```

## 3. Call the API

```bash
curl http://10.20.0.15:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "/models/Qwen3-72B-Instruct", "messages": [{"role": "user", "content": "Merhaba!"}]}'
```

Any OpenAI SDK works — change only the `base_url`.

> **Warning:** > vLLM has no authentication. Keep it on a private network or put it behind a load balancer with an API key check.

## Performance tips

- Use `--tensor-parallel-size` equal to the number of GPUs
- Enable prefix caching for chat workloads with long system prompts
- Store model weights on local NVMe, not on network storage