Cloud Data Ingestion and Replication

All-in-one ingestion and replication for files, databases, apps, and streams

Get started with Application Ingestion and Replication

Overview

Application Ingestion and Replication can transfer data from Software-as-a-Service (SaaS) and on-premise applications to cloud-based data lakes, data warehouses, and event streaming platforms. The SaaS and on-premise applications used in your business or organization store large amounts of business-critical data daily.

You can use Application Ingestion and Replication to transfer the data stored by your applications to targets that can handle large volumes of data. After you transfer the data to the target, you can consolidate the data and use it for various purposes, such as advanced data analytics and data warehousing.

Application Ingestion and Replication can perform the following types of load operations:

Initial load: Loads source data read at a single point in time to a target. After the data is loaded, the ingestion job ends. You can use this load type to materialize a target to which incremental changes will be sent later.
Incremental load: Loads data changes continuously or until the ingestion job is stopped or ends. The ingestion job loads the changes that have occurred since the last time it ran or from a specific start point. You can use this load type to keep data in the target up to date so that you can make informed decisions for your business or organization based on the latest data.
Initial and Incremental load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects continuously.

Why Best

Solution Capabilities

Use Cases

Architecture

Supported Source and Target

Create Application Ingestion and Replication Connections

Configure the Application Ingestion and Replication Task

Implementing First Salesforce to Snowflake Application Ingestion & Replication Job

Types of Apply Mode

For incremental load and combined initial and incremental load jobs with Snowflake targets, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target.

Apply modes like Soft Deletes and Audit apply to specific targets. Refer to the resource below for additional details.

The options are:

Target Properties

Standard: Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target.
Soft Deletes: Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
- Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
Audit: Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, transaction ID, and before image.
- Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.

Default Data Type Mappings in Application Ingestion Jobs

Handling Source Schema Changes

Overriding Schema Drift Options

Running & Deploying Application Ingestion and Replication Job

Asset Management

Monitoring Application Ingestion and Replication Job

Performance Tuning Guide

Troubleshooting

Recommended Webinars

Stay informed about upcoming expert-led webinars to deepen your knowledge of cloud data ingestion and replication, enabling seamless and efficient data movement - View upcoming webinars