On This Page

Home / Stream/ Setting Up Cribl Stream/ About Connected Environments/Migrate Data from On-Prem to Cribl.Cloud

Migrate Data from On-Prem to Cribl.Cloud

You can use Connected Environments to migrate data from an on-prem deployment to a Cribl.Cloud deployment. If you previously had an on-prem deployment of Cribl and you are moving forward with a Cloud-first or Cloud only solution, you can use connected environments to onboard your data from a on-prem Worker Nodes to their counterpart Cribl.Cloud Worker Nodes.

You can transfer data from on-prem Worker Nodes to the Cribl.Cloud Worker Nodes without incurring any data transfer costs.

Limitations
​

This workflow should only be used to permanently migrate data from an on-prem environment to a Cribl.Cloud environment. In other words, this workflow is intended as a one-time operation to onboard data from on-prem to Cribl.Cloud.

It should not be used as a method of regularly transferring data between on-prem and Cribl.Cloud environments. Instead, see Transfer Data Between Workspaces or Environments for more information about the current data transfer capabilities between Cribl Worker Nodes that report to the same Leader or different Leaders.

Overview of the Data Migration Workflow
​

The workflow for migrating data from on-prem Worker Nodes to Cribl.Cloud Worker Nodes has these phases:

  1. Create a Connected Environment. This process generates the information you need to set up the data flow between the on-prem Worker Nodes and the Cribl.Cloud Worker Nodes.

  2. On-Prem Worker Nodes: Configure HTTP or TCP Destination. Add an auth token on the on-prem Worker Nodes to configure the first half of the connection between the on-prem Destination and the Cribl.Cloud Source.

  3. Cribl.Cloud Worker Nodes: Configure HTTP or TCP Source. Add a matching auth token on the Cribl.Cloud Worker Nodes to complete the connection.

  4. Validate the Data Flow. Confirm that the data is properly flowing from the on-prem Worker Nodes to the Cribl.Cloud Worker Nodes.

Create a Connected Environment
​

To begin the process of migrating data from on-prem to Cribl.Cloud, you need to first set up a Connected Environment on the on-Prem Leader. This process generates the information you need to set up the data flow between on-prem Worker Nodes and the Cribl.Cloud Worker Nodes.

To set up a Connected Environment, follow the instructions on How to Connect On-Prem Leaders to Cribl.Cloud. These instructions also explain how to:

  • Optionally set up environment variables at the time you create the Connected Environment.
  • Use a command-line script to set up a Connected Environment if needed.

Ensure that you also follow the instructions to Enable the On-Prem Leader Connection to Cribl.Cloud.

On-Prem Worker Nodes: Configure HTTP or TCP Destination
​

On the on-prem deployment, select a Worker Node and configure a new or existing Cribl HTTP or Cribl TCP Destination. These Destinations enable internal data transfer between Cribl nodes to prevent data transfer billing costs.

To configure an HTTP or TCP Destination:

  1. Follow the general configuration instructions on the Cribl HTTP or Cribl TCP Destination documentation.

  2. In the Auth Token section of the configuration modal, select Auth Token to set up a shared secret between this Destination and the Cribl.Cloud Source:

    • Token secret (text secret): Select the name of the secret that you have created previously or create a new secret using the Create button. See Create and Manage Secrets for specific instructions about creating secrets.
    • Enable token: Toggle the token on to enable it.
    • Description: Optionally, enter a description summarizing the purpose of the secret.

    If you don’t see the Auth Token section of the Destination, it means you have not set up a Connected Environment for your on-prem Leader.

  3. Commit and deploy the changes.

You can add multiple tokens if you need to rotate auth tokens.

Cribl.Cloud Worker Nodes: Configure HTTP or TCP Source
​

On the Cribl.Cloud deployment, select a Worker Node and configure a new or existing Cribl HTTP or Cribl TCP Source. These Sources enable internal data transfer between Cribl nodes to prevent data transfer billing costs.

To configure an HTTP or TCP Source, follow the same steps listed in the previous section about Configuring HTTP or TCP Destination for the on-prem Worker Nodes.

When you select or create the secret from the Token secret (text secret) menu, this secret must match the value of the secret used on the on-prem Cribl HTTP or Cribl TCP Destination.

Commit and deploy after making these configuration changes.

Validate the Data Flow
​

As a final step, confirm that the data is properly flowing from the on-prem Worker Nodes to the Cribl.Cloud Worker Nodes. A few methods for confirming data flow include:

  • Capture sample data on your Cribl HTTP or TCP Source on your Cribl.Cloud Worker Nodes and verify it looks correct. See Add Sample Data for specific instructions.
  • In the configuration modal for your Cribl HTTP or TCP Source on your Cribl.Cloud Worker Nodes, select the Charts tab to view data events coming in to verify you see activity. You can also compare this to the outgoing data on the Cribl HTTP or TCP Destination on your on-prem Worker Nodes.