Create Pipeline

Pipeline in Instill Core is a deployable resource that functions as an API service. Designed specifically for unstructured data ETL, it provides a flexible and powerful way to build data workflows. At its core, a pipeline recipe defines a directed acyclic graph (DAG) workflow that can be executed end-to-end via manual triggers, remote events, or scheduled runs.

Pipeline offers multiple methods for creating a pipeline, including HTTP and gRPC APIs, the Python and TypeScript SDKs, and Console.

Create Pipeline via API

curl -X POST 'http://localhost:8080/v1beta/namespaces/NAMESPACE_ID/pipelines' \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $INSTILL_API_TOKEN" \
--data '{
	"id": "PIPELINE_ID",
	"description": "A brief description of your pipeline",
	"rawRecipe": "#The YAML recipe of your pipeline"
}'

from instill.clients import init_pipeline_client

pipeline = init_pipeline_client(api_token="INSTILL_API_TOKEN", url="http://localhost:8080")
pipeline.create_pipeline(
	namespace_id="NAMESPACE_ID",
	pipeline_id="PIPELINE_ID",
	description="A brief description of your pipeline",
	raw_recipe="#The YAML recipe of your pipeline",
)
pipeline.close()

NAMESPACE_ID refers to the user or organization ID, and PIPELINE_ID is the pipeline ID. The resource ID must start with a lowercase letter (a–z) or an underscore, followed by up to 31 characters that may include lowercase letters (a–z), digits (0–9), hyphens, or underscores. The total length must be between 1 and 32 characters.

INSTILL_API_TOKEN is the API token that should be created in advance.

Pipeline Object

In the Create Pipeline and all APIs in Manage Pipelines, all endpoints follow a consistent structure for request and response bodies. Below are the key fields:

• id: The pipeline's unique identifier.
• uid: The immutable UID of the pipeline.
• name: The full resource name of the pipeline.
• description: A brief description of the pipeline.
• readme: A README for the pipeline.
• rawRecipe: The pipeline's recipe in YAML format. For more details, see Pipeline Recipe.
• recipe: The JSON representation of the YAML recipe (read-only).
• sharing: The sharing settings for the pipeline. For more information, refer to Share Pipeline.
• releases: Associated releases of the pipeline. For more information, refer to Release Pipeline.
• stats: Statistical data related to the pipeline.

For additional details, please refer to the API reference.

Example Pipeline Object:

{
  "name": "users/test/pipelines/sample",
  "uid": "fd9d9e8f-8533-45ea-b8c9-f4029d9ad1af",
  "id": "sample",
  "description": "This is a sample pipeline",
  "recipe": {
    "version": "v1beta"
  },
  "createTime": "2024-10-01T02:54:18.594935Z",
  "updateTime": "2024-10-01T02:54:18.594935Z",
  "deleteTime": null,
  "sharing": {
    "users": {
      "*/*": {
        "enabled": true,
        "role": "ROLE_EXECUTOR"
      }
    },
    "shareCode": null
  },
  "metadata": {},
  "ownerName": "users/test",
  "releases": [],
  "readme": "",
  "permission": {
    "canEdit": true,
    "canTrigger": true,
    "canRelease": false
  },
  "owner": {
    ...
  },
  "dataSpecification": null,
  "tags": [],
  "stats": {
    "numberOfRuns": 0,
    "lastRunTime": "0001-01-01T00:00:00Z",
    "numberOfClones": 0
  },
  "rawRecipe": "version: v1beta\n\n# ---------- Data ----------\n# Variables that manually trigger the pipeline and can be referenced in component actions\n# Structure example:\n# variable:\n#   key:              # Unique identifier for the variable.\n#     type:  # Data type, e.g., image, string, array:string.\n#     title:          # Title of this input field.\n#     description:    # Introduction of what should be input. \n#\n# variable:\n\n# Custom user-defined output\n# Structure example:\n# output:\n#   key:      # Unique identifier for the output.\n#     title:  # Title of this output field.\n#     value:  # Can be a value or use ${} to reference data.\n#\n# output:\n\n# ---------- Schema ----------\n# Component actions executed during the pipeline run\n# Click \"⌘O\" to add a new component\n# component:\n",
  "sourceUrl": "",
  "documentationUrl": "",
  "license": "",
  "profileImage": "http://localhost:8080/v1beta/users/test/pipelines/sample/image",
  "endpoints": {
    "webhooks": {}
  }
}

Create Pipeline via Console

To create a new pipeline from Console, follow these steps:

1. Launch Console via a local Instill Core deployment at http://localhost:3000, or by selecting the Go to Console button in the bottom-left of the Instill Agent interface if you are using Instill Core as a managed service.
2. Navigate to the Pipelines page using the navigation bar.
3. Click the + Create Pipeline button.
4. Select the Owner (namespace).
5. Enter the name for your Pipeline.
6. Enter the description for your Pipeline.
7. Set up your Pipeline to be public or private.
8. Click the Create button.
9. Console will redirect you into the pipeline editor.