Streaming

Streaming a response

Set "stream": true and the reply arrives as server-sent events: one data: line per chunk, each an OpenAI chat.completion.chunk, ending with data: [DONE].

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"A token"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" is…"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

The OpenAI SDKs handle this for you (stream=True in Python, stream: true in Node.js). With cURL, pass -N so the output isn't buffered.

Token usage in a stream

To receive the token counts in a final chunk before [DONE], as with OpenAI, send:

"stream_options": {"include_usage": true}

Per-request metrics

Send X-P9-Debug-Metrics: true on a streaming request, and after data: [DONE] Paranine sends one more event with that request's figures:

data: [DONE]

data: {"p9_metrics":{"ttft_ms":112,"tok_s":48.7,"cost_micro_pkr":45120}}

The OpenAI SDKs stop reading at [DONE] and ignore the extra event, so the header is safe to send everywhere. To read the figures, parse the stream yourself and look for p9_metrics:

curl -N https://sovereign.paranine.com/v1/chat/completions \
  -H "Authorization: Bearer $P9_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-P9-Debug-Metrics: true" \
  -d '{"model":"paranine/gpt-oss-120b(Global)","stream":true,"messages":[{"role":"user","content":"Hello"}]}'

The same measurements, for every request, are in your request logs. There, time to first token is counted from the moment Paranine passes your request to the model, so it reads a little lower than ttft_ms here.