On This Page

Home / Edge/ Setting Up Cribl Edge/ About Connected Environments/Migrate Data from On-Prem to Cribl.Cloud

Migrate Data from On-Prem to Cribl.Cloud

You can use Connected Environments to migrate data from an on-prem deployment to a Cribl.Cloud deployment. If you previously had an on-prem deployment of Cribl and you are moving forward with a Cloud-first or Cloud only solution, you can use connected environments to onboard your data from a on-prem Edge Nodes to their counterpart Cribl.Cloud Edge Nodes.

You can transfer data from on-prem Edge Nodes to the Cribl.Cloud Edge Nodes without incurring any data transfer costs.

Limitations
​

This workflow should only be used to permanently migrate data from an on-prem environment to a Cribl.Cloud environment. In other words, this workflow is intended as a one-time operation to onboard data from on-prem to Cribl.Cloud.

It should not be used as a method of regularly transferring data between on-prem and Cribl.Cloud environments. Instead, see Transfer Data Between Workspaces or Environments for more information about the current data transfer capabilities between Cribl Edge Nodes that report to the same Leader or different Leaders.

Overview of the Data Migration Workflow
​

The workflow for migrating data from on-prem Edge Nodes to Cribl.Cloud Edge Nodes has these phases:

  1. Create a Connected Environment. This process generates the information you need to set up the data flow between the on-prem Edge Nodes and the Cribl.Cloud Edge Nodes.

  2. On-Prem Edge Nodes: Configure HTTP or TCP Destination. Add an auth token on the on-prem Edge Nodes to configure the first half of the connection between the on-prem Destination and the Cribl.Cloud Source.

  3. Cribl.Cloud Edge Nodes: Configure HTTP or TCP Source. Add a matching auth token on the Cribl.Cloud Edge Nodes to complete the connection.

  4. Validate the Data Flow. Confirm that the data is properly flowing from the on-prem Edge Nodes to the Cribl.Cloud Edge Nodes.

Create a Connected Environment
​

To begin the process of migrating data from on-prem to Cribl.Cloud, you need to first set up a Connected Environment on the on-Prem Leader. This process generates the information you need to set up the data flow between on-prem Edge Nodes and the Cribl.Cloud Edge Nodes.

To set up a Connected Environment, follow the instructions on How to Connect On-Prem Leaders to Cribl.Cloud. These instructions also explain how to:

  • Optionally set up environment variables at the time you create the Connected Environment.
  • Use a command-line script to set up a Connected Environment if needed.

Ensure that you also follow the instructions to Enable the On-Prem Leader Connection to Cribl.Cloud.

On-Prem Edge Nodes: Configure HTTP or TCP Destination
​

On the on-prem deployment, select a Edge Node and configure a new or existing Cribl HTTP or Cribl TCP Destination. These Destinations enable internal data transfer between Cribl nodes to prevent data transfer billing costs.

To configure an HTTP or TCP Destination:

  1. Follow the general configuration instructions on the Cribl HTTP or Cribl TCP Destination documentation.

  2. In the Auth Token section of the configuration modal, select Auth Token to set up a shared secret between this Destination and the Cribl.Cloud Source:

    • Token secret (text secret): Select the name of the secret that you have created previously or create a new secret using the Create button. See Create and Manage Secrets for specific instructions about creating secrets.
    • Enable token: Toggle the token on to enable it.
    • Description: Optionally, enter a description summarizing the purpose of the secret.

    If you don’t see the Auth Token section of the Destination, it means you have not set up a Connected Environment for your on-prem Leader.

  3. Commit and deploy the changes.

You can add multiple tokens if you need to rotate auth tokens.

Cribl.Cloud Edge Nodes: Configure HTTP or TCP Source
​

On the Cribl.Cloud deployment, select a Edge Node and configure a new or existing Cribl HTTP or Cribl TCP Source. These Sources enable internal data transfer between Cribl nodes to prevent data transfer billing costs.

To configure an HTTP or TCP Source, follow the same steps listed in the previous section about Configuring HTTP or TCP Destination for the on-prem Edge Nodes.

When you select or create the secret from the Token secret (text secret) menu, this secret must match the value of the secret used on the on-prem Cribl HTTP or Cribl TCP Destination.

Commit and deploy after making these configuration changes.

Validate the Data Flow
​

As a final step, confirm that the data is properly flowing from the on-prem Edge Nodes to the Cribl.Cloud Edge Nodes. A few methods for confirming data flow include:

  • Capture sample data on your Cribl HTTP or TCP Source on your Cribl.Cloud Edge Nodes and verify it looks correct. See Add Sample Data for specific instructions.
  • In the configuration modal for your Cribl HTTP or TCP Source on your Cribl.Cloud Edge Nodes, select the Charts tab to view data events coming in to verify you see activity. You can also compare this to the outgoing data on the Cribl HTTP or TCP Destination on your on-prem Edge Nodes.