How to measure LLM API latency yourself

Short answer

Send a few dozen streaming requests with your real prompt, record when the first content token arrives and when the stream ends, and report the median and the 95th percentile, not the average. The open-source script llm-latency does this for any OpenAI-compatible or Anthropic-native endpoint in one command, with no dependencies beyond Python 3.9.

Published latency numbers, including the ones we publish, come from someone else's prompts, region and network. The only numbers that describe your users are the ones measured from where your code runs. This page shows how to get them.

1. Run it

git clone https://github.com/amdmsz/llm-latency && cd llm-latency
export LLM_API_KEY=sk-...

# OpenAI-compatible endpoint (Chat Completions)
python3 llm_latency.py --base-url https://api.openai.com/v1 --model gpt-6.1-sol -n 20

# Anthropic Messages API
python3 llm_latency.py --api anthropic --base-url https://api.anthropic.com --model claude-opus-5-5 -n 20

Any gateway or self-hosted server that speaks one of the two formats works the same way; point --base-url at it. Repeat --model to compare several models in one run.

The output is one row per model:

model        n  success  ttft p50  ttft p95  total p50  total p95  tok/s p50

2. Use your own prompt

The default prompt is short. Latency depends heavily on input length and on how much the model has to think, so measure with something representative:

python3 llm_latency.py --base-url https://api.openai.com/v1 --model gpt-6.1-sol \
  --prompt-file prompt.txt --max-tokens 1024 -n 50 --json run.json --csv run.csv

--json and --csv keep the per-request results so you can look at the slow ones.

3. Read the columns

Column	What it is	What to watch for
ttft	Seconds from sending the request to the first visible content token	For reasoning models this includes thinking time. That is what your user waits for
total	Seconds until the stream ends	Grows with max_tokens and output length
tok/s	Output tokens divided by generation time	A ~ prefix means the server sent no usage block and tokens were estimated
success	HTTP 2xx, at least one content token, and a clean end of stream	A stream cut off without its terminator counts as a failure even with status 200

4. Three mistakes that make the numbers wrong

Reporting the average. A few very slow requests pull the mean far from what most requests look like, and hide how bad the slow ones are. Report p50 and p95.

Overstating output speed on reasoning models. The token count a server returns usually includes thinking tokens, but if you only start the clock at the first visible token, you divide all tokens by part of the time. The script handles this in two ways: on OpenAI-style endpoints it subtracts reasoning tokens when the server reports them, and on the Anthropic format it starts the speed clock at message_start. If you write your own measurement, check which tokens your count includes.

Trusting a p95 from twenty requests. With 20 samples the p95 is close to "the slowest one". It is fine for spotting a problem. For a number you will set timeouts or alerts from, run a few hundred.

5. Make it routine

The script exits with a non-zero code when any request fails, so it can gate a deploy or run from cron:

python3 llm_latency.py --base-url "$BASE" --model "$MODEL" -n 10 --json "latency-$(date +%F).json" \
  || echo "LLM endpoint check failed"

Keep the JSON files. A week of them tells you more than any single run.

Limits

• Every request is billed by your provider. Keep -n modest and --max-tokens realistic.
• With -c above 1 you are also measuring rate limits and queueing, which is useful but a different number.
• Client-side timing includes your network. That is the point, but it means results from your laptop and from your servers will differ.