Use Cribl Products to Change the Shape, Size, or Quality of Data

Cribl provides powerful data transformation capabilities, allowing you to accommodate analysis tools or convert data types as needed. Transforming data usually involves altering the structure, format, and content of the data to fit specific requirements. You can transform data at any stage where Cribl is involved. Here, we’re focusing on processing data through a Pipeline. Use Functions to manipulate data, such as filtering, enriching, extracting, and aggregating. Here are just a few of the Functions you can use to transform data:

• Regex Extract: Extracts a portion of the raw event, and places it into a specified field. It’s useful for reorganizing and capturing data that matches a regular expression pattern. See regex filtering and lookups with regex.
• Lookup: Enriches event data by adding fields like hostname, name, id, and type using a key such as the host IP. It is typically used to add context or additional information to events from external databases or tables. For details, refer to the Lookup Function and Search Lookups. If your data feed changes regularly or is larger than a few million rows, we recommend using Redis to host the lookup file.
• Eval: Executes arbitrary expressions to generate new fields or modify existing ones. This can be used to perform calculations, manipulate strings, or format and transform event data.
• GeoIP: Correlates source IPs to a geographic database, thereby enriching IP address data with their corresponding geographical information like country, city, and lat/long coordinates. Check out our GeoIP and Threat Feed Enrichment sandbox.
• Aggregation: Use the Aggregations Function to group multiple records into summarized data, which is beneficial for presenting high-level trends without the overhead of every single transactional detail. Cribl Search provides many aggregation functions.
• Sampling: Processes a percentage of incoming events. This is particularly useful for maintaining a manageable Dataset for analyses in environments with a vast data inflow.
• For more complex transformations that are not covered by built-in functions, use the Code Function, where custom JavaScript code can be written to manipulate and cast data as needed.

Note that to test and enable data flow through your configured Pipeline, you’ll need to provision Workers – see Managing Cribl.Cloud Worker Groups.

Check out this conceptual video on transforming data. By using these powerful features, you can ensure that vast amounts of machine-generated data are effectively utilized and fit for varied analytical needs.

The Cribl Docs team created this content with assistance from AI. Using AI-generated text allows us to quickly create expanded content with more detailed scenarios, solutions, and examples. Because AI-generated content isn’t always completely accurate, we review, test, and augment it before publishing.