By default, when you request a completion from Empower, the entire completion is generated and sent back in a single response. For longer completions, the wait for a response can extend over several seconds.

To receive responses more promptly, you can ‘stream’ the completion as it’s being generated. This approach allows you to begin printing or processing the beginning of the completion before it is completely finished. This is particularly useful in chat experiences where providing users with the best experience involves delivering responses as soon as they are ready.

​
How to use

To stream completions, set stream=true when calling the chat completions api. This returns an object that streams the response as data-only server-sent events. To extract chunks, use the delta field instead of the message field.

Empower supports the streaming in both message and tool use responses.

​
Code Example

The Empower API is fully compatible with the OpenAI API. Below, we will provide a few code examples demonstrating how to utilize this compatibility using the OpenAI SDK:

python
from openai import OpenAI

client = OpenAI(
    base_url="https://app.empower.dev/api/v1",
    api_key="YOUR_API_KEY"
)


response = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[
        {"role": "user", "content": "Who won the world series in 2020?"},
        {
            "role": "assistant",
            "content": "The Los Angeles Dodgers won the World Series in 2020.",
        },
        {"role": "user", "content": "Where was it played?"}],
    stream=True
)

content = ''
# Streaming response is an object that streams the response as data-only server-sent events.
# Here we iterate over the chunks in the response object to get the data
for chunk in response:
    # Extract the delta from the chunk
    content_delta = chunk.choices[0].delta.content

    if content_delta:
        content += content_delta
    print(content)

Output

The
The
The 2
The 20
The 202
The 2020
The 2020 World                                                                                                                                                                                                                                                                                                              The 2020 World Series
The 2020 World Series was                                                                                                                                                                                                                                                                                                   The 2020 World Series was played
The 2020 World Series was played at                                                                                                                                                                                                                                                                                         The 2020 World Series was played at Gl
The 2020 World Series was played at Globe                                                                                                                                                                                                                                                                                   The 2020 World Series was played at Globe Life
...
Query a Deployed ModelJSON mode and Structured Output