Improvements

Pipeline

• Added per-component output display in the editor to aid debugging.
• Enhanced UX to keep Ask task fully visible during scroll.
• Added smart hint suggestions before typing (Ctrl + Space).
• Updated UI for “not found” Public Pipelines & Models, redirecting users to the Playground tab.

Bug Fixes

Pipeline

• Fixed missing JSON output in Playground and pipeline builder for the Crawler component.
• Fixed lack of feedback when renaming a component fails.
• Fixed issue where only partial credit-related error messages were shown across the pipeline.
• Corrected logic for displaying the appropriate UI for JSON outputs.
• Fixed non-functioning button in the pipeline preview tab.
• Fixed error where CMDO would break after adding a random component.
• Fixed disappearing organization name and logo after creation.
• Replaced "Response key" with "Response title" in the Playground output view.
• Fixed bug where “ask-the-catalog” pipeline output chunks appeared as 'object'.

Component

• Fixed issue with jq JSON operator filtering.

Console

• Fixed jittering of dialog windows after clicking the Clone button.

To file an issue please usethis template or raise them in our Discord community.

We are improving vendor connection resources by introducing Instill Integrations to easily setup and contain the connection details of any 3rd party vendors (i.e. OpenAI API key, Pinecone URL and API key, etc.) on Instill VDP. This will ensure higher observability by ensuring you see what vendor has been authorized for each integration and improve overall user experience when applied in the pipeline builder.

In contrast to Secret Management, this new integration feature will separate the component setup from the variable which means:

• Secrets will be used for avoiding repetition and will not be tied to any component type
• Integrations will be used to store connection setup details all in one place

To setup an integration, you can either do it through the Integrations page or within the pipeline builder.

Configuring integrations

The Integrations page will allow you to easily configure and protect your external connections all in one place, allowing you to quickly search & filter available integrations and configure new ones.

To setup your personal new integration, simply follow these steps:

1. Go to your Profile > Settings > Integrations
2. Click Connect on any available 3rd party vendor
3. Fill in the connection details

To setup your organizations new integration, simply follow these steps:

1. Go to your Profile > Your Organization > Organization > Integrations
2. Click Connect on any available 3rd party vendor
3. Fill in the connection details

Using integrations

When writing a pipeline recipe in the pipeline builder, you can reference the connection in the setup section within your component declaration:

component: 
  library-db:
    type: sql
    input:
      table-name: books
      update-data: ${variable.books}
    setup: ${connection.my-library-1}  # <-- reference connection 
    task: TASK_UPDATE

Next, we will also enable connecting with 3rd party services via OAuth.

Learn more about this feature by joining ourDiscord where we showcase these new features.

The Instill SDK provides a Python-based interface for interacting with Instill Core features, including data, pipeline and model operations.

Please note: Our Python SDK supports up to and including Python 3.11.

Data - Instill Artifact

Perform key data operations such as:

• List, create, update, and delete catalogs
• Upload, process, and manage files within a catalog
• View and search file chunks
• Ask catalog to retrieve relevant information based on files

Pipeline - Instill VDP

Handle pipeline tasks:

• Create and run pipelines

Model - Instill Model

Work with AI models:

• Create and serve models
• Run model for inference

Check out Instill AI Cookbooks to learn how to use the Python SDK:

• Generate structured insights from URL Notebook
• Generate structured outputs from LLMs Notebook

The Python SDK is undergoing continuous improvements. We'd love to hear your feedback! Please share your thoughts in the #feedback channel on ourDiscord community.

Improvements

Artifact

• Renamed all APIs to follow a consistent new structure.

Pipeline

• Clicking a preview component now centers its code in the left editor panel.
• Made the left sidebar in the Low-Code Editor resizable.
• Enhanced the "Add Component" dropdown by removing null entries.
• Rounded component borders and increased text size for improved readability.
• Added description hints for each input key in components.
• Removed the deprecated “test version” button from the interface.

Bug Fixes

Pipeline

• Fixed issue preventing Asana from being correctly added as a component.
• Fixed spinning loader persisting after a successful pipeline run.
• Corrected indentation display when adding components.
• Added missing Public/Private labels in certain scenarios.
• Fixed issues when converting JPG images to text.
• Eliminated white padding issue in the image preview widget.

Python SDK

• Fixed error when triggering organization pipelines via InstillClient.trigger_org_pipeline_release().

To file an issue please usethis template or raise them in our Discord community.

Chroma is an AI-native open-source embedding database.

We added a Chroma Component on💧 Instill VDP to expand your vector database options and enable you to build and search Chroma's' vector databases.

Support tasks:

• Batch Upsert
• Upsert
• Query
• Delete
• Create Collection
• Delete Collection

Freshdesk is a cloud-based customer support platform that was founded with the mission of enabling companies of all sizes to provide great customer service. Our goal is simple: make it easy for brands to talk to their customers and make it easy for users to get in touch with businesses.

We added a Freshdesk Component on💧 Instill VDP to enable more Customer Relationship Management (CRM) capabilities when building your pipelines. This application component focuses on using Freshdesk's ticketing system.

Support tasks:

• Get Ticket
• Create Ticket
• Reply To Ticket
• Create Ticket Note
• Get All Conversations
• Get Contact
• Create Contact
• Get Company
• Create Company
• Get All
• Get Product
• Get Group
• Get Agent
• Get Skill
• Get Role

Asana is a web and mobile "work management" platform designed to help teams organize, track, and manage their work.

We added an Asana Component on💧 Instill VDP to empower project management workflow automation. Build unique pipeline automation that you can mix and match across various vendors using Instill Components like data, application and AI.

Support tasks:

• Goal
• Task
• Project
• Portfolio

To request a new component, please gohere and click "Request A Component".

The new pipeline editor contains a pipeline preview with a streaming feature which is useful for you to observe how data flows through the components when running your pipeline. It will provide visibility into the status of each component in your pipeline run.

You can enable streaming mode by:

1. Create or use an existing pipeline such as structured-web-insights
2. Clone this pipeline by clicking Clone in the top right
3. Go into editor mode by clicking Edit in the top right
4. Setup your Google Search Engine (see setup docs or README)
5. Fill in the input form and hit Run

Learn more about this feature by joining ourDiscord where we showcase these new features.

Improvements

Pipeline

• Reduced font size for code blocks for better readability.
• Added support for manual saving using Ctrl+S / Cmd+S.
• Preview panel is now visible on the Overview page.
• Improved default UI for pipeline editor output.
• Improved UX for creating multiline variables (was previously unclear to users).

Components

• Redesigned input/output interface for the Instill Artifact component.
• Added Jira integration: users can now create issues directly from pipelines.

Bug Fixes

Pipeline

• Fixed an issue where auto-save could desynchronize frontend and backend states, causing accidental reversion.
• Fixed version dropdown on the Overview page not displaying all available versions.
• Fixed streaming functionality not working in Safari.
• Fixed issue where output images weren’t displaying properly in the console.

Documentation

• Fixed broken documentation link for the Instill Model component.

We are introducing a completely new way of building your pipelines on Instill VDP to shift our tool to focus on better supporting developers in the AI and data domain. The goal of VDP is to help you dynamically build versatile pipelines for complex and unique use cases that require minimal maintenance with high data observability.

Recipe as Code

The new pipeline builder UI allows you to edit YAML recipes next to the pipeline preview for improved debugging and data flow visualization. This means you can edit recipe.yml files in your own editors like VSCode and simply import them into VDP, streamlining the recipe sharing process. Additionally, the editor offers smart hints and syntax highlighting, making it easier to build and modify recipes efficiently.

Watch this tutorial on how to build recipes in the new pipeline editor:

Quick tips:

• Use Control+O or Cmd+O shortcut to add components easily
• Use Control+K or Cmd+K shortcut to search for things
• Use Control+S or Cmd+S shortcut to save your recipe manually
• Use Control+Enter or Cmd+Enter shortcut to run the pipeline
• Click the Getting Started button at the bottom-left corner to view the guide in the pipeline preview panel.

High Observability

The pipeline preview offers a visual representation of data flow between components, striking a balance between code-level control and visual management. This feature is particularly beneficial for data engineers working on complex pipelines, as it provides real-time streaming feedback to simplify debugging. The preview also enhances component presentation, making it easier to build, test, and maintain sophisticated data pipelines efficiently.

You can try exploring the recipe of the structured-text-summarizer pipeline to transform a paragraph of unstructured text into structured JSON output in the new pipeline editor.

Learn more about this feature by joining ourDiscord where we showcase these new features.

We are introducing Runs, a logging system for pipeline and model executions. This feature provides execution logs for the currently selected pipeline or model that will help you better track credit usage, monitor performance and get snapshots of each executions metadata which is useful for debugging.

To get started, simply follow these steps:

Pipeline Logging Runs

1. Go to any pipeline such as the contract-reviewer
2. In the Playground, run the pipeline
3. Go to "Runs" tab to view your history of pipeline executions
4. Click into any Run ID to view specific metadata related to this pipeline execution

Clicking into a specific Run ID will include useful information including run duration, credit consumption, trigger source, recipe, input/output results, and each components metadata.

Model Logging Runs

1. Go to any Instill Model such as the instill-ai/llava-1-6-13b.
2. In the Playground, run the model.
3. Go to "Runs" tab to view your history of model executions
4. Click into any Run ID to view specific metadata related to this model execution

Powered by open-source Milvus, Zilliz delivers the most performant and cost-effective vector database for AI at any scale.

We added Zilliz Component on💧 Instill VDP to expand your vector database options and enable you to build and search Zilliz's vector database.

Support tasks:

• Vector Search
• Upsert
• Batch Upsert
• Delete
• Create Collection
• Drop Collection
• Create Partition
• Drop Partition

To request a new component, please gohere and click "Request A Component".

Improvements

• Added create issue task to Jira Component
• Added new fields in Website Crawl
• Incorporated I/O tasks in Instill Artifact Component
• Added Excel pivot table to Markdown conversion in Document Component
• Implemented code snippet extraction in Web operator
• Enhanced SQL component with SSL/TLS input (base64 encoded) and engine relocation
• Expanded HubSpot Component with deal/ticket updates and task retrieval
• Added XLSX to Text/Markdown conversion support for Catalog
• Introduced PDF to Images conversion in Document component
• Improved Artifact component with base64 upload, multiple file support, and filename output
• Added file upload interruption notification

Bug Fixes

• Fixed mobile device login/signup buttons
• Corrected Public Model Playground Page button wording
• Resolved text highlight omission issue
• Fixed blank Floating Window UI bug
• Addressed Catalog file deletion issue

To report a bug or improvement, please create an issue on GitHubhere.

Milvus is an open-source vector database built for GenAI applications. Install with pip, perform high-speed searches, and scale to tens of billions of vectors with minimal performance loss.

We added Milvus Component on💧 Instill VDP to expand your vector database options and enable you to build and search Milvus' vector databases.

Support tasks:

• Vector Search
• Upsert
• Delete
• Drop Collection
• Create Partition
• Drop Partition

To request a new component, please go here and click "Request A Component".

All Instill Cloud users automatically receive 10,000 Instill Credits for free each month. Instill Credits help reduce the need to pre-configure any AI components by skipping the need for creating accounts or API keys on 3rd party services.

If you run out, simply follow these steps to top up:

Personal Credits:

1. Click the top-right avatar
2. Go to Settings > Billing > Credits

Organization Credits:

1. Click the top-right avatar
2. Go to Settings > Organization
3. Select a specific organization
4. Go to Billing > Credits

Learn more about this feature by joining ourDiscord where we showcase these new features.