Selected Cribl Stream Destinations support writing out data as files in the Apache Parquet columnar storage format.

The following Destinations support the Parquet format:

• Azure Blob Storage
• Azure Data Explorer
• Cloudflare R2
• Databricks
• Filesystem/NFS
• Google Cloud Storage
• MinIO
• Amazon S3 Compatible Stores
• Amazon Security Lake

You can select the Parquet format from the Destination’s General > Data format drop-down. (An exception is the Amazon Security Lake Destination, which does not have the drop-down, because it uses the Parquet format exclusively). When the Parquet format is selected, you can configure the Parquet settings for the Destination, including selecting the Parquet schema to use.

You can automatically generate a Parquet schema based on the events of each Parquet file the Destination writes, by toggling on Parquet Settings > Automatic schema for the Destination. The automatically-generated schema preserves the exact structure of the ingested events, with no data lost. Every field in the generated schema will have a data type of JSON, STRING, INT_64, UNSIGNED_INT_64, or DOUBLE.

You can refer to a single Parquet schema in multiple Destinations. When editing them, be aware that any change will affect all places it is used in.

When logging is set to debug, you will see what schema is associated with the Parquet files that Cribl Stream writes out. On Linux, you can use the Cribl Stream CLI’s parquet command to view a Parquet file, its metadata, or its schema.

Cribl Stream supports two kinds of schemas:

• Parquet schemas for writing data from a Cribl Stream Destination to Parquet files, described on this page.
• JSON schemas for validating JSON events.

These schemas serve different purposes. When using C.Schema, omit the schema kind to validate against a JSON schema, or set the schema kind to parquet to validate against a Parquet schema.

Validate Events Against a Parquet Schema ​

You can check whether an event conforms to a schema in the Parquet Schemas library before a Destination writes Parquet. Use this while you are building a schema or Pipeline, and optionally to divert events that would otherwise be dropped at write time.

To see whether an event conforms, use:

C.Schema('<schema-id>', { kind: 'parquet' }).validate(__e)

The expression returns true for a conforming event and false for a non-conforming event. It does not modify the event. If the schema ID is unknown or misspelled, the expression also returns false without throwing an error.

You can use this expression anywhere Cribl Stream accepts JavaScript, including Data Preview, a Pipeline Function, a Route filter, or an Output Router filter.

Typical uses:

• Schema and Pipeline development. In Data Preview or an Eval Function, call explain() to see which field failed and why. Refine the schema or event before you send data to a Parquet Destination.
• Keep non-conforming events. A Parquet Destination drops rows that violate the schema. To keep those events, use validate() in a filter. Send conforming events to the Parquet Destination. Send the rest to a non-Parquet Destination, such as S3 writing JSON to a separate bucket.

Schema validation is computationally expensive compared with a typical filter. Test throughput in your environment before you run this check on every event in a high-volume path. In filters that process every event, use validate(). Use explain() when you need failure details.

Parquet schema validation approximates the behavior of the Parquet writer. It considers values valid when the writer can coerce them, such as the string "42" for an integer field. Some data types or conversions can still fail when the Destination writes the event.

For method signatures and return values, see C.Schema.

Automatic Parquet Schema Generation ​

When you decide whether or not to generate Parquet schemas automatically, consider these pros and cons.

Advantages:

• Configuration is minimal.
• There’s no need to know how events you want to ingest are structured.
• There’s no need to change Parquet schema when event structure changes.

Disadvantages:

• Writing Parquet files takes longer than with predefined schemas. Note that schema complexity increases linearly with the number of events used to generate the schema.
• Automatically-generated Parquet schemas cannot be as fine-tuned as predefined ones, to capture certain nuances. For example, encoding will always be PLAIN, compression will always be SNAPPY.
• Some aspects of event structure (such as maps and sets) cannot be captured in the schema.
• Some aspects of time-related fields cannot be captured in the schema.
• Because the system produces a new Parquet schema for each file written, there may be differences between successive schemas. This can limit your ability to concatenate a collection of Parquet files.

Add Parquet Schema ​

If you are not using automatic schema generation, or if there’s no pre-existing schema that suits your data, before configuring a Destination for Parquet output, you should add an existing Parquet schema, or create a new Parquet schema that suits the data you’re working with.

You do not need to start from scratch: Cribl provides sample Parquet schemas for you to clone and then customize as needed.

To add a new Parquet schema:

1. In the sidebar, select Worker Groups and choose a Worker Group.
2. In the Worker Groups submenu, select Processing.
3. If you are adding the schema to a Pack, select Packs and choose your Pack.
4. Select Knowledge, then Parquet Schemas.
5. Select Add Parquet schema.
6. Enter an ID for the regex and an optional Description.
7. In Schema, paste the Parquet schema content.
8. Select Save.

Modify Parquet Schema ​

You can modify existing Parquet schemas, but Cribl strongly recommends that you first clone the schema, and give the clone its own distinct name.

1. Select the schema you want to start with, to open it in a modal.
2. Select Clone Parquet Schema to open the New Parquet schema modal.
3. Give the new schema a name and description.

Cloning vs Modifying Schemas ​

Modifying an existing schema does not propagate your modifications to its clones. Cloning and renaming schemas is the safest approach, because it ensures that your Destination will use the newly modified version of the schema.

If you do not clone and rename the schema (that is, you leave the schema name unchanged), you still must re-select the schema in the Destination’s Parquet schema drop-down to bring the modified version into the Destination.

Parquet Format Used by Cribl Stream ​

Different Parquet readers and writers behave differently. Keep the following guidelines in mind when working with Parquet in Cribl Stream.

In the Parquet Schema editor, you express your Parquet schema in JSON. This does not look like the examples in the Parquet spec, which have their own syntax (like the repetition/name/type triple), but they are functionally equivalent. See the examples below.

The editor provides autocompletion and validation to guide you. (However, Cribl Stream currently does not fully support autocompletion on deeply nested schemas.)

File Extensions ​

Cribl Stream can read Parquet files only if they have the extension .parquet, .parq, or .pqt.

Field Content ​

When Cribl Stream writes to a Parquet file:

• If the data contains a field that is not present in the schema - i.e, an extra field - Cribl Stream writes out the parent rows, but omits the extra field.

• If the data contains a field that is present in the Parquet schema, but whose properties violate the schema, Cribl Stream treats this as a data mismatch. Cribl Stream drops the rows containing that field - it does not write those rows to the output Parquet file at all.

• If the data contains JSON, the JSON must be stringified. Otherwise, Cribl Stream treats this as a data mismatch, and does not write out the row. For example, this valid (but not stringified) JSON will trigger a data mismatch: { "name": "test"}.

  The same JSON in stringified form will work fine: "{\"name\": \"test\"}".

Data Types ​

Cribl Stream supports:

• All primitive types.
• All logical types.
• All converted types.

Converted types have been superseded by logical types, as described in the Apache Parquet docs. Cribl Stream can read Parquet files that use converted types, but will write out the same data using corresponding logical types.

Repetition Type ​

You have three alternatives when defining a field’s Repetition type:

• Set optional to true.
• Set repeated to true.
• Set neither optional nor repeated. This implicitly sets the Repetition type to required, and it is the default.

Usage guidelines:

• Do not set both optional and repeated to true.
• Do not use the required key at all.
• Instead of omitting optional, you have the option to include it, but set it to false.
• Instead of omitting repeated, you have the option to include it, but set it to false.
• If any field’s Repetition type is repeated, Cribl Stream represents this field as a single key whose value is an array - not as separate key-value pairs with identical keys.

Encodings ​

• Among the *DICTIONARY encodings, Cribl Stream supports only DICTIONARY. Trying to assign the unsupported encodings PLAIN_DICTIONARY or RLE_DICTIONARY will produce an error.

• BYTE_STREAM_SPLIT encoding can be used only with DOUBLE or FLOAT types, and otherwise produces errors.

• The RLE and all DELTA* encodings also produce errors.

Parquet Limitations ​

Cribl Stream does not support writing Parquet files which employ any of the following:

• Parquet Modular Encryption.
• The Parquet Bloom Filter.
• Separation of metadata and/or column data into multiple files.

Cribl Stream does not support reading or writing Parquet files which employ the deprecated INT96 data type.

Examples of Parquet Schema as JSON ​

These examples should give you a hint of how to express Parquet schema in JSON. For a full explanation of the Parquet syntax, see the Parquet spec.

List Example ​

In this example, the whole schema is named the_list, and its structure consists of a container named list whose zero or more elements are each named element. Note that in Parquet schema, to specify that a field is nested (that is, contains other fields), you just give it the type group.

message schema {
  REQUIRED group the_list (LIST) {
    REPEATED group list {
      REQUIRED BYTE_ARRAY element (STRING);
    }
  }
}

Expressed as JSON, the same schema is more spread-out, because to specify that a field is nested, you give it a sub-field named fields. (This is equivalent to declaring the nested field’s type as group.)

{
  "the_list": {
    "type": "LIST",
    "fields": {
      "list": {
        "repeated": true,
        "fields": {
          "element": {
            "type": "STRING"
          }
        }
      }
    }
  }
}

Map Example ​

In this example, the whole schema is named the_map, and its structure consists of a container named key_value whose zero or more pairs of elements each contain a key and a value element. Both the key and the value are strings in this case, but they could be other data types.

message schema {
  REQUIRED group the_map (MAP) {
    REPEATED group key_value {
      REQUIRED BYTE_ARRAY key (STRING);
      REQUIRED BYTE_ARRAY value (STRING);
    }
  }
}

{
  "the_map": {
    "type": "MAP",
    "fields": {
      "key_value": {
        "repeated": true,
        "fields": {
          "key": {
            "type": "STRING"
          },
          "value": {
            "type": "STRING"
          }
        }
      }
    }
  }
}

Array of Arrays Example ​

In this example, the whole schema is named array_of_arrays, and its structure consists of a container named list whose zero or more elements - which are themselves lists - are each named element. Within each of those element lists are one or more additional list/element structures.

message schema {
  REQUIRED group array_of_arrays (LIST) {
    REPEATED group list {
      REQUIRED group element (LIST) {
        REPEATED group list {
          REQUIRED INT64 element;
        }
      }
    }
  }
}

Expressed as JSON, the same schema is cumbersome to read compared to Parquet, which was designed to express nested structures concisely.

{
  "array_of_arrays": {
    "type": "LIST",
    "fields": {
      "list": {
        "repeated": true,
        "fields": {
          "element": {
            "type": "LIST",
            "fields": {
              "list": {
                "repeated": true,
                "fields": {
                  "element": {
                    "type": "INT64"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}