Cloud Data Ingestion and Replication

Application Ingestion and Replication can transfer data from Software-as-a-Service (SaaS) and on-premise applications to cloud-based data lakes, data warehouses, and event streaming platforms. The SaaS and on-premise applications used in your business or organization store large amounts of business-critical data daily. You can use Application Ingestion and Replication to transfer the data stored by your applications to targets that can handle large volumes of data. After you transfer the data to the target, you can consolidate the data and use it for various purposes, such as advanced data analytics and data warehousing. 

Application Ingestion and Replication can perform the following types of load operations:

• Initial load: Loads source data read at a single point in time to a target. After the data is loaded, the ingestion job ends. You can use this load type to materialize a target to which incremental changes will be sent later. 
• Incremental load: Loads data changes continuously or until the ingestion job is stopped or ends. The ingestion job loads the changes that have occurred since the last time it ran or from a specific start point. You can use this load type to keep data in the target up to date so that you can make informed decisions for your business or organization based on the latest data.
• Initial and Incremental load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects continuously. 

• Unified solution for ingestion and replication from on-premises and SaaS apps with minimal overhead on the source system
• Supports various mechanisms for replication from SAP –application layer, database layer, or hybrid architecture
• No breakage of pipelines in case of schema changes at SAP and other app sources

Easy: 

• Easy 4-step wizard for data engineers to ingest and replicate from SAP and other SaaS sources
• Out of the connectivity to SAP using various mechanisms using ABAP, ODP along with trigger and log-based CDC

Efficient: 

• Efficient bulk load and CDC capture with no source footprint 
• Performant way of ingesting and applying CDC onto CDW, CDL, and messaging systems

Cost-Effective: 

• Ingest tens of thousands of application objects along with CDC in minutes
• Automatic schema drift handling– for source schema changes

You can use Application Ingestion and Replication to solve multiple business problems.

Following are some of the use cases of Application Ingestion and Replication:

• Data warehousing: Organize the data of SaaS and on-premise applications by transferring it to a cloud-based data warehouse system. After an initial batch load of data to the data warehouse, Application Ingestion, and Replication can replicate data changes continuously from a source application to keep the data up to date in the data warehouse.
• Advanced data analytics: Consolidate your application data in data lakes and data warehouses for extensive analysis that helps in making informed business decisions.
• Utilizing application data in other data processing applications: Keep data lakes and data warehouses synchronized with SaaS and on-premise sources and provide up-to-date data to other applications for processing.

Application Ingestion and Replication requires the following components to run an application ingestion and replication job: 

• Secure Agent: Secure Agent is the program that runs tasks and enables secure communication between your organization and Intelligent Data Management Cloud (IDMC) Services. When you run an application ingestion and replication job, the metadata of the ingestion task is pushed to the Secure Agent which enables the ingestion job to process the data.
• Database Ingestion service: It is a microservice that Secure Agent uses to run both application ingestion and replication and database ingestion and replication tasks.
• Intelligent Data Management Cloud (IDMC) Services interface: IDMC provides a web-based interface to create and deploy an application ingestion and replication task.
• Source and target APIs: The application ingestion and replication job uses the source APIs to retrieve the data from the source objects and the target APIs to load the data to the target. For an incremental load operation, the ingestion job identifies the changes that are made to the source objects after a specific date and time and retrieves the changes at intervals defined in the associated ingestion task.

The sources that Applications Ingestion task support depend on whether the application ingestion tasks transfer a point-in-time snapshot of data in a batch initial load operation or load incremental change data from a specific start point.

The targets that the Applications Ingestion task support depend on the sources specified for the Application Ingestion and Replication tasks.

Refer to the video below for creating a Salesforce Mass Ingestion Connection with

1. JWT Bearer Flow Authentication
2. Username-Password Flow Authentication

In Data Integration, use the application ingestion task wizard to configure the application ingestion task.

On the wizard pages, complete the following configuration tasks:

1. Define basic task information, such as the task name, project location, runtime environment, and load type.
2. Configure the source.
3. Configure the target.
4. Configure the task schedule and runtime options.

The article below describes how to set up Mass Ingestion Applications and create your first application ingestion job to replicate data from a Salesforce source to a Snowflake target.

For incremental load and combined initial and incremental load jobs with Snowflake targets, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. The options are:

1. Standard: Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target.
2. Soft Deletes: Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
   • Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
3. Audit: Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, transaction ID, and before image. 
   • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
     

Apply modes like Soft Deletes and Audit apply to specific targets. Refer to the resource below for additional details.

This reference provides default data-type mappings for source applications and target databases. When application ingestion generates the target table and data, it uses these mappings in the resource below.

Also, when you configure a target for an application ingestion task, you can optionally define data-type mapping rules to override the default mappings of source data types to target data types.

You can configure Application Ingestion and Replication tasks to automatically detect source schema changes, also called schema drift, and handle these changes on the target. This feature is available only for the incremental load and combined initial and incremental load tasks.

You can override the schema drift options when you resume an application ingestion job that is in the Stopped, Aborted, or Failed state. The overrides affect only those objects that are currently in the Error state because of the Stop Object, Stop Table, Stop Report, or Stop Job schema drift option. Use the overrides to correct or resolve these errors.

You can run commands in the Cloud Mass Ingestion Command Line Interface (CLI) to create, view, and deploy application ingestion and database ingestion tasks and to list, start, stop, redeploy, undeploy, and monitor ingestion jobs.

You can manage Mass Ingestion assets such as projects, folders, and ingestion tasks from the Explore page.

You can perform the following management tasks depending on your user role and permissions:

• Edit ingestion tasks.
• Copy projects, folders, and ingestion tasks.
• Move folders and ingestion tasks.
• Rename projects, folders, and ingestion tasks.
• Delete projects, folders, and ingestion tasks.
• Apply tags so you can filter assets on the Explore page.
• Configure user permissions for projects, folders, or assets.
• Use source control to manage versions of projects, folders, and tasks.
• Migrate assets between organizations.

Monitor the progress, performance, and status of  Data Ingestion and Replication jobs from the following services:

• Monitor
• Operational insights
• Data Integration

Find more details by clicking on the documentation link below.

Performance Tuning and Best Practices for Mass Ingestion Tasks

Learn how to troubleshoot an application ingestion and replication task by clicking on the resource below.