Pipeline Builder is an interactive, AI-powered assistant that helps you build data Pipelines in Cribl Edge using plain-language prompts. Instead of building a Pipeline manually, you can describe what you want to do with your data and Pipeline Builder will generate a working Pipeline for you.

Pipeline Builder gives you an easy way to explore what Cribl Edge can do, especially if you’re new to the platform or want help tackling more complex use cases. You stay in control: Pipeline Builder’s suggestions are editable, and you can review, modify, or view outputs from the suggestions before using them in production.

Here are a few common ways you can use Pipeline Builder:

• Structure raw data to match a standard schema or convert from one schema to another: The data for many Destinations follow specific schemas. If events are improperly formatted, some Destinations may drop data or store them in ways that make the data difficult to find and use. With Cribl Edge, you can convert your raw data into normalized schemas, using Pipeline Builder to do the heavy lifting around configuring Functions. Pipeline Builder is optimized for transforming data into the OCSF, UDM, and ECS schemas. You can also upload your own schema objects to convert to other schemas. See Convert Data into a Custom Schema for information about using custom schemas or schemas that aren’t yet officially supported.
• Apply common Functions to transform or enrich your data: Pipeline Builder can apply Cribl Edge’s most frequently used Functions based on your intent. You don’t need to know Regex or JavaScript syntax or which Functions to use up front. Just describe your goals and Pipeline Builder builds the appropriate logic for you.
• Filter and route data dynamically: Use plain-language prompts to define filtering or routing logic without writing Regex or JavaScript expressions manually. Pipeline Builder builds the first draft of the filtering and control logic. You can use its suggestions as-is or fine-tune as needed.

How to Use Pipeline Builder ​

This section explains the overall workflow for using Pipeline Builder to build Pipelines.

Before You Begin ​

Cribl AI is available by default, so Pipeline Builder is available without any setup. To manage AI settings, from Settings, select the Global tab, and then select AI Settings.

For details, see Cribl AI Availability.

Launch Pipeline Builder ​

You can launch Pipeline Builder in three ways:

• Through QuickConnect.
• Through the Pipelines page.
• Inside a Pipeline through the Add Function menu.

Select a tab for more detailed instructions about each method for launching Pipeline Builder:

Build a Pipeline with Pipeline Builder ​

The interactive flow of Pipeline Builder follows a few general phases:

1. Capture sample data: When you first launch Pipeline Builder, you see a chat window that prompts you to capture sample data. Choose a method for providing sample data, such as capturing a live sample from a Source, selecting an existing sample file, or pasting raw events directly into the chat.

   Why does it need sample data?

   Pipeline Builder uses this data to build, preview, test, and validate the data transformations it creates. It helps you compare inbound and outbound data to ensure the Pipeline produces the expected output.

2. Choose an objective: After receiving your sample data, Pipeline Builder summarizes the data and presents options for transforming it. Select the objective that matches your goal, and provide any additional information Pipeline Builder requests. When you transform data into OCSF, Pipeline Builder recommends the most suitable OCSF class for your data and asks you to approve or adjust the mapping, instead of asking you to choose from a list of classes. Pipeline Builder maps all available fields from your data, including fields such as Source and Destination IP addresses, to produce more complete events.

3. Review the Pipeline: Pipeline Builder generates the Pipeline automatically and presents the output events in the chat window so you can see what the transformed data looks like. It displays the name of the new Pipeline in the editor tab and repeats it in the chat so you can identify the Pipeline later. Expand the Functions in the editor to explore the settings it applied. Pipeline Builder saves the Pipeline automatically, so there is no approval step. It then runs your sample events through the saved Pipeline and, if you chose a target schema, checks that the output events match that schema. If validation finds issues, Pipeline Builder revises the Pipeline and validates it again, continuing until validation passes or further revisions no longer improve the result. Use natural language prompts in the chat window to request changes and continue to iterate until the Pipeline meets your needs. If you don’t want a change that Pipeline Builder saved, you can roll it back.

4. Return to QuickConnect: Because Pipeline Builder has already saved the Pipeline, select Back to QuickConnect at the top of the chat window to return to QuickConnect. Select Save Connection to add the Pipeline to your deployment.

   Take note of the Pipeline ID displayed in the chat window. You use this ID to locate the Pipeline on the Pipelines page later.

5. Confirm the Pipeline is present: Confirm that the Pipeline you built with Pipeline Builder appears on the Pipelines page or in QuickConnect. To find it in QuickConnect, select the connection you used to launch Pipeline Builder, then select its Pipeline icon and select Pipeline to view the associated Pipelines.

Edit or Update a Pipeline With Pipeline Builder ​

When you open an existing Pipeline and launch Pipeline Builder, the Pipeline loads automatically. You can then choose how to proceed:

• Describe your edits directly: Tell Pipeline Builder what changes you want to make, and it applies them to the loaded Pipeline.
• Capture sample data first: Optionally collect sample events to understand the Pipeline’s current behavior or test changes before committing to them.

After Pipeline Builder applies your edits, it presents the output events in the chat window so you can review the results. Use natural language prompts to continue refining the Pipeline as needed.

Pipeline Builder applies your edits to the existing Pipeline and saves it automatically, so there is no approval step and no prompt to save a copy. It always overwrites the Pipeline you opened. To keep the original unchanged, copy the Pipeline before you launch Pipeline Builder, and edit the copy.

After saving, Pipeline Builder validates the result and revises the Pipeline if validation finds issues, as it does when you build a new Pipeline. Validation requires sample events, so if you edit a Pipeline without capturing or providing sample data, Pipeline Builder does not validate the result. If you don’t want a change that Pipeline Builder saved, you can roll it back.

You can edit any existing Pipeline with Pipeline Builder, not just Pipelines that were previously generated with Pipeline Builder.

To start an edit session, launch Pipeline Builder from one of these entry points:

• From inside a Pipeline, select Add Function, then Edit with AI.
• From QuickConnect, select a connection that has an existing Pipeline, then open Pipeline Builder.

Edit the Pipeline in the Editor Panel ​

You can also change the Pipeline yourself while you chat with Pipeline Builder. Make your changes in the Pipeline editor panel next to the chat window, then select Save to write them to the Pipeline. Pipeline Builder treats the edited Pipeline as the current version, so its next response builds on your changes.

Roll Back Pipeline Changes ​

Because Pipeline Builder saves automatically, use rollback to undo a change that you don’t want. Each time Pipeline Builder saves, it records how to reverse that save: restore the previous version of the existing Pipeline, or, if that save created the Pipeline, delete it.

To roll back a change:

1. In the chat window, hover over the most recent completed step. The Rollback button appears when you hover over that step, and only that step offers it.

2. Select Rollback, and then confirm when Pipeline Builder prompts you.

Pipeline Builder reverses that step, rewinds the conversation to it, and lets you continue from there. If you roll back a save, Pipeline Builder restores the previous version of the Pipeline. If that save created the Pipeline, Pipeline Builder deletes it. To step back further, for example after several saves, repeat the rollback. Each rollback reverses the most recent remaining step.

Guidelines and Better Practices ​

When working with Pipeline Builder, keep these guidelines in mind:

• Always review the results, even though validation is automatic: Pipeline Builder saves and validates each Pipeline without waiting for your approval. Validation checks that your sample events run through the Pipeline and, if you chose a target schema, that the output events match it. It does not confirm that the Pipeline does what you intended. As with any AI tool, check that the output events match your expectations, and roll back any change that doesn’t. Trust, but verify.
• Pipeline Builder may take different paths to meet an objective: AI tools do not always solve problems in the exact same way every time, which sometimes takes some getting used to. This behavior is by design, but just be aware that the results may vary.

Use effective prompts to ensure the best results from the Pipeline Builder. Some general guidelines:

• Be specific: To receive high-quality results, be clear and explicit with your requests. Provide exact names of fields that you want to reference or create.
• Provide examples: For instance, if you want to mask a specific piece of text, describe the target text in detail and explain what it should be masked to (for example, Mask all IPv4 IP addresses to X.X.X.X).
• Ask questions: Use the chat window to answer general questions about Pipelines. For example, you could ask the chatbot, What Pipeline Functions can I use to redact sensitive information like IP addresses?

Feature Support ​

Cribl does not make any warranty regarding the availability, uptime, performance, or accuracy of Billy, including PII Detection and Prevention. Please read the Supplemental Terms for Cribl AI for more information.

Requirements ​

By default, Pipeline Builder requires internet access to reach the Cribl-managed AI backend. If you configure a Custom AI Provider, Pipeline Builder instead requires network connectivity to the configured provider endpoint and does not need to reach the Cribl-managed backend.

Supported Deployment Types ​

Pipeline Builder is available:

• For Cribl.Cloud, hybrid, and on-prem deployments.
• For Cribl Edge 4.12.0 and newer.

Supported Schemas ​

Pipeline Builder specializes in transforming data into these schemas:

For OCSF, Pipeline Builder recommends a version based on your sample data. To map to a different supported version, select Pick from our library when Pipeline Builder prompts you to choose a target schema, then select the version you want.

Supported Functions and Integrations ​

Pipeline Builder can build Pipelines using these Functions:

• Aggregations
• Code
• Comment
• Drop
• Eval
• Flatten
• Lookup
• Mask
• Numerify
• Regex Extract
• Regex Filter
• Rename
• Serialize
• Unroll

Pipeline Builder can support all available Sources and Destinations.

What Data the Pipeline Builder Can Access ​

The Pipeline Builder assistant has access to:

• The natural language messages that you write in the chat window.
• The sample events that you capture, select, paste, or generate, subject to token limits.
• The Pipeline that you are building or editing, including any changes that you make in the editor panel.
• The target schema that you select, including any custom schema.
• The output events that result from running your sample events through the Pipeline.
• The IDs of the Sources, Destinations, sample files, and Pipelines that you select during the session.

Pipeline Builder does not access other user or Organization data beyond what is in the session. For details about how Cribl AI features handle your data, see Cribl AI and Your Data.