Cloud Data Ingestion and Replication

Step into your learning journey and progress steadily towards mastering the Cloud Data Ingestion and Replication (CDIR) service with ease and confidence.

Why Best

Data Replication is the foundation for enterprises aiming to be GenAI-ready and ensure the latest data is available for analytics and reporting. CDIR simplifies data ingestion and replication from various sources. It is the only unified solution for data ingestion, synchronization, and replication from various sources, including databases, applications, streaming, and files.

It supports different use cases:

• Cloud Data Lake Ingestion
  • Mass ingestion of files into cloud and on-premises data lakes
  • Streaming and IoT data ingestion into a data lake
  • Mass ingestion of on-premises database content into a cloud or on-premises data lake
• Data Warehouse Modernization or Migration
  • Mass ingestion of on-premises database, data warehouse, and mainframe content into a cloud data warehouse (example: Snowflake)
  • Synchronize ingested data with Change Data Capture (CDC)
• Kafka for Real-Time Analytics
  • Log files and clickstream ingestion 
  • CDC ingestion
  • IoT data ingestion
  • Easily ingests and replicates enterprise data using batch, streaming, real-time, and CDC into cloud data warehouses, lakes, relational databases, and messaging hubs.

Features

• Step-by-Step Wizard: Easily design and create ingestion tasks.
• Deployment and Management: Streamline scheduling, real-time monitoring, and lifecycle management
• Versatile Connectivity: Out-of-the-box support for various sources and targets
• Scalability: Handle billions of rows and millions of files within hours

Solution Capabilities

• ​Easy:  A 4-step wizard for data engineers to ingest and replicate files, databases, applications, CDC, and streaming data
• Efficient: 
  • Automatic CDC and schema drift to ingest and replicate data into cloud data warehouses and data lakes. Includes  “Audit mode” and “Soft Delete” for various customer use cases.
  • Offers multiple low-latency CDC mechanisms (log-based, query-based, trigger-based, API-based) and robust data validation.
• Cost-Effective: Reduce delays by democratizing data availability and multiplying the integration workforce with a user-friendly and wizard-driven interface.

Pre-Requisites

Before you start this course, it is recommended to complete the following prerequisites, which consist of three focused modules:

1. Setup and User Management: Activate ingestion and replication services while managing user roles for secure, efficient access.
2. Batch Ingestion Tasks: Use CDIR for smooth batch ingestion from databases and local files to the cloud.
3. Real-Time and Application Ingestion Tasks: Enable real-time streaming and application data ingestion to the cloud with CDIR.

Completing these modules will prepare you to engage with the main course material confidently.

High-level Architecture

Customer Value

Marathon Oil

The notion that Data is the new Oil hasn’t been so real. Informatica’s CDIR helped Marathon Oil ingest at top speed onto their Cloud Datawarehouse Snowflake. A lot of data that goes between on-premises and the cloud goes through the CDIR application.

CDIR scaled to the requirement from Marathon Oil and the monitoring module helped have a constant check on the lifecycle of the data.