Set up Component

Set up Component via Connection

Integration Connections allow users to set up components that connect with external services.

Create Component Connection via Console

You can store the connection settings to such services from the Console > Settings > Integrations page.

The Connected section shows the connections already created, grouped by application. You can explore the supported integrations and create a new connection through the Connect button.

Connect through OAuth 2.0

If an integration supports OAuth 2.0 to connect with the third-party service (e.g., GitHub, Slack), you won't need to fill in the connection details. The connection flow will be as simple as authorizing the Instill Core application to interact with that vendor on your behalf.

You need to follow the vendor's instructions to connect via OAuth, which will typically involve creating an application on that service. In order to integrate your Instill Core deployment with a vendor via OAuth, you'll need to copy your app's client ID and client secret and set them as environment variables on instill-core.

If your deployment isn't configured to support OAuth, you'll still be able to create connections. In this case, you'll need to fill out a form with the connection details, which you'll need to fetch from the vendor's site.

Create Component Connection via API

This endpoint allows for creating a connection associated with a specific namespace.

export INSTILL_API_TOKEN=********

curl -X POST 'http://localhost:8080/v1beta/namespaces/NAMESPACE_ID/connections' \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $INSTILL_API_TOKEN" \
--data '{
  "id": "CONNECTION_ID",
  "integrationId": "openai",
  "method": "METHOD_DICTIONARY",
  "setup": {}
}'

from instill.clients import init_pipeline_client

pipeline = init_pipeline_client(api_token="INSTILL_API_TOKEN", url="http://localhost:8080")
pipeline.create_connection(
  namespace_id="NAMESPACE_ID",
  connection_id="CONNECTION_ID",
  integration_id="openai",
)
pipeline.close()

Set up Component with a Connection in Pipeline Recipe

component:
  sql-0:
    type: sql
    input:
      table-name: weights
      update-data: ${variable.new-weights}
    setup: ${connection.service-1-dev}
    task: TASK_UPDATE

It offers several benefits over in-component setup declarations:

• Making pipeline recipes more compact.
• Centralizing credential management in a single page.
• Connecting with 3rd party services via OAuth 2.0.

Set up Component Using Secret

In Pipeline, certain components require the configuration of credential data, such as passwords, API keys, and tokens. The protection and privacy of these credentials are paramount. To prevent potential exposure of sensitive data, pipeline not only prohibits users from setting credential data as plaintext within the recipe but also stores all credential data in a specified secret space. This approach ensures that sensitive data is kept secure and inaccessible to unauthorized individuals. Users can manage their secrets by navigating to Console > Settings > Secrets.

Create Secrets via Console

1. Proceed to Console > Settings > Secrets.

2. Press the Create Secret button.

3. Input a unique key and the corresponding credential data.

4. Press the Create Secret button to save the secret and make it available for pipeline configurations.

Create Secret via API

export INSTILL_API_TOKEN=********

curl -X POST 'http://localhost:8080/v1beta/namespaces/NAMESPACE_ID/secrets' \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $INSTILL_API_TOKEN" \
--data '{
  "id": "Your secret ID",
  "description": "A brief description of your secret",
  "value": "The secret value"
}'

from instill.clients import init_pipeline_client

pipeline = init_pipeline_client(api_token="INSTILL_API_TOKEN", url="http://localhost:8080")
pipeline.create_secret(
   namespace_id="NAMESPACE_ID",
   secret_id="Your secret ID",
   description="A brief description of your secret",
   value="The secret value",
)
pipeline.close()

Set up Component using Secret in Pipeline Recipe

When setting up a pipeline, users can refer to stored secrets within component configurations using a specific syntax. Rather than incorporating plaintext credentials, users can securely load secret data by employing the following syntax:

${secret.my-secret-key}

This method ensures that the actual credential data is never exposed in the pipeline configuration, maintaining the integrity and security of the system.