Streaming
By default, when you request a completion from Empower, the entire completion is generated and sent back in a single response. For longer completions, the wait for a response can extend over several seconds.
To receive responses more promptly, you can ‘stream’ the completion as it’s being generated. This approach allows you to begin printing or processing the beginning of the completion before it is completely finished. This is particularly useful in chat experiences where providing users with the best experience involves delivering responses as soon as they are ready.
How to use
To stream completions, set stream=true when calling the chat completions api.
This returns an object that streams the response as data-only server-sent events. To extract chunks,
use the delta field instead of the message field.
Empower supports the streaming in both message and tool use responses.
Code Example
The Empower API is fully compatible with the OpenAI API. Below, we will provide a few code examples demonstrating how to utilize this compatibility using the OpenAI SDK:
from openai import OpenAI
client = OpenAI(
base_url="https://app.empower.dev/api/v1",
api_key="YOUR_API_KEY"
)
response = client.chat.completions.create(
model="mistralai/Mixtral-8x7B-Instruct-v0.1",
messages=[
{"role": "user", "content": "Who won the world series in 2020?"},
{
"role": "assistant",
"content": "The Los Angeles Dodgers won the World Series in 2020.",
},
{"role": "user", "content": "Where was it played?"}],
stream=True
)
content = ''
# Streaming response is an object that streams the response as data-only server-sent events.
# Here we iterate over the chunks in the response object to get the data
for chunk in response:
# Extract the delta from the chunk
content_delta = chunk.choices[0].delta.content
if content_delta:
content += content_delta
print(content)
Output
The
The
The 2
The 20
The 202
The 2020
The 2020 World The 2020 World Series
The 2020 World Series was The 2020 World Series was played
The 2020 World Series was played at The 2020 World Series was played at Gl
The 2020 World Series was played at Globe The 2020 World Series was played at Globe Life
...