Documentation
Start typing to search…

Streaming

A complex pipeline may take time to execute. To keep users updated during the process, Pipeline offers response streaming, delivering real-time updates of the pipeline run.

Enable Response Streaming

To enable response streaming, simply add the header Accept: text/event-stream to the request. Pipeline will return responses in text/event-stream format, providing real-time updates.

export INSTILL_API_TOKEN=********

curl -X POST
'http://localhost:8080/v1beta/namespace/NAMESPACE_ID/pipelines/PIPELINE_ID/trigger' \
--header "Content-Type: application/json" \
--header "Accept: text/event-stream" \
--header "Authorization: $INSTILL_API_TOKEN" \
--data '{ "data": [ { "variable": { "prompt": "hello world" } } ] }'

Example Stream

event: PIPELINE_STATUS_UPDATED
data: {"updateTime":"2024-10-01T19:40:31.800966675Z","batchIndex":0,"status":{"completed":false,"errored":false,"started":true}}

event: COMPONENT_STATUS_UPDATED
data: {"updateTime":"2024-10-01T19:40:32.128045472Z","componentID":"openai-0","batchIndex":0,"status":{"completed":false,"errored":false,"skipped":false,"started":true}}

event: COMPONENT_INPUT_UPDATED
data: {"updateTime":"2024-10-01T19:40:32.128772046Z","componentID":"openai-0","batchIndex":0,"status":{"completed":false,"errored":false,"skipped":false,"started":true},"input":{"model":"gpt-4o-mini","n":1,"prompt":"hello world","response-format":{"type":"text"},"system-message":"You are a helpful assistant.","temperature":1,"top-p":1}}

event: COMPONENT_OUTPUT_UPDATED
data: {"updateTime":"2024-10-01T19:40:33.181563573Z","componentID":"openai-0","batchIndex":0,"status":{"completed":false,"errored":false,"skipped":false,"started":true},"output":{"texts":["Hello! How can I assist you today?"],"usage":{"completion-tokens":0,"prompt-tokens":0,"total-tokens":0}}}

event: PIPELINE_OUTPUT_UPDATED
data: {"updateTime":"2024-10-01T19:40:33.182456024Z","batchIndex":0,"status":{"completed":false,"errored":false,"started":true},"output":{"o":["Hello! How can I assist you today?"]}}

event: PIPELINE_STATUS_UPDATED
data: {"updateTime":"2024-10-01T19:40:33.397610043Z","batchIndex":0,"status":{"completed":true,"errored":false,"started":true}}

Supported Events

Pipeline currently provides the following events:

  • PIPELINE_STATUS_UPDATED: Updates the pipeline's status
  • PIPELINE_OUTPUT_UPDATED: Updates the pipeline's output
  • PIPELINE_ERROR_UPDATED: Updates error messages in the pipeline
  • COMPONENT_STATUS_UPDATED: Updates a component's status
  • COMPONENT_INPUT_UPDATED: Updates a component's input
  • COMPONENT_OUTPUT_UPDATED: Updates a component's output
  • COMPONENT_ERROR_UPDATED: Updates error messages in a component

Each event shares the following common fields:

  • updateTime: The timestamp of the event
  • batchIndex: Indicates the data index in the batch
  • status: Reflects the status of the pipeline or component

In-Component Streaming

Pipeline not only supports component-level streaming (events for when a component starts, errors, or completes) but also offers in-component streaming. If a component itself supports streaming, such as text generation from LLM models, Pipeline can stream these updates to the user in real time.

Currently, in-component streaming uses an override mode, meaning each new output replaces the previous one for the same component. In future updates, delta mode will be introduced to reduce transmission overhead and improve performance.

Updated about 6 hours ago