Cloud Data Ingestion and Replication

Streaming Ingestion and Replication ingests high-volume, real-time data from sources like logs, clickstreams, social media, and IoT to on-premises and cloud storage. It allows tracking and monitoring of the ingestion progress. For operational intelligence or real-time data warehousing, it processes data before it becomes obsolete. Simple transformations ensure the ingested data is ready for analytics.

• Unified solution for ingestion from a variety of messaging and IoT protocol sources with native connectivity
• Simple and easy way to build and manage streaming data ingestion with a wizard-driven interface
• Ability to apply simple transformations as part of the ingestion pipeline

Easy:

• Easy 4-step wizard for data engineers to ingest streaming data from various sources
• Alerting and monitoring of the ingestion pipeline in real-time dashboards

Efficient:

• High performant ingestion way of ingesting files with support for various file formats
• Leverages out of the box connector for messaging & IoT systems across on-premise and cloud sources such as Kafka, JMS, EventHub, Google PubSub, Amazon Kinesis, MQTT, and OPC to data lakes

Cost-Effective:

• Edge transformations to enrich and cleanse data across multiple formats such as JSON, and XML before ingestion
• Schema agnostic ingestion– for addressing schema changes in message format

Consider using Streaming ingestion and Replication in the following scenarios:

• Real-time analytics. Ingest streaming and IoT data into messaging systems, such as Apache Kafka or Amazon Kinesis, for real-time analytics. Real-time analytics can help companies identify business opportunities and revenue streams that can result in increased profits and improved customer service.
• Data integration. Ingest streaming and IoT data into cloud data lakes, such as Amazon S3, for integrating data in real-time to provide up-to-the-minute information.

For supported source and target versions, please refer to the KB article below:

Before you define connections for the streaming ingestion and Replication tasks, ensure that you have a license for the required connectors for your source and target types. Connections provide access to data in cloud and on-premises applications, platforms, databases, and flat files. Connection definitions include the location of the source or target, the runtime environment, and the other properties specific to the connection type.

Before you can create a connection, ensure that the correct connectors for your sources and targets are available in Intelligent Data Management Cloud (IDMC) Services. The supported connectors vary by type of ingestion task. To create a connection or search for an existing connection, use the Administrator service. After you configure connection properties, the connection becomes available for use within the organization.

In Data Integration, use the Streaming Ingestion and Replication task wizard to configure a streaming ingestion task. You define a source and a target, and you can optionally define a transformation to transform the data.

Transformations are part of a Streaming Ingestion and Replication task. Transformations represent the operations that you want to perform when ingesting streaming data. Each transformation performs a specific function. For example, a Filter transformation filters data from the ingested data based on a specified condition.

When you create a Streaming Ingestion and Replication task, adding a transformation is optional. Each transformation type has a unique set of options that you can configure. Refer to the below resources for more details about the transformations in Streaming Ingestion and Replication.

You can configure additional runtime options for the Streaming Ingestion and Replication task. Runtime options include settings to manage reject events and to notify users about errors.

After you create a Streaming Ingestion and Replication task, you must deploy it on the Secure Agent for the task to run as a job. Before deploying the task, ensure that the Secure Agent configured for the task is running. You can undeploy a Streaming Ingestion and Replication job from the Secure Agent. When you undeploy a streaming ingestion job, the Secure Agent does not save the previous state histories of the job. So, when you deploy the streaming ingestion task again, the task runs as a completely new job

You can stop and resume Streaming Ingestion and Replication jobs. Stop or resume a job on the My Jobs page in Data Integration, the All Jobs and Running Jobs pages in Monitor, or the Mass Ingestion page in Operational Insights. When you stop a streaming ingestion job, the Secure Agent saves the previous state histories of the job. When you resume the stopped streaming ingestion job, the job starts running from the last saved state.

Use streaming ingestion resources to deploy, undeploy, start, stop, copy, and update Streaming Ingestion and Replication tasks and to monitor streaming ingestion jobs.

Monitor the progress, performance, and status of Data Ingestion and Replication jobs from the following services:

• Monitor: All Jobs and Running Jobs
• Operational Insights:  Overview and All Jobs
• Data Integration: Recent Jobs and My Jobs

Our Troubleshooting section is designed to assist you in resolving common issues or challenges you may encounter. Whether you're experiencing technical difficulties, operational issues, or unexpected errors, this section provides step-by-step guidance and solutions. Explore the troubleshooting article below to troubleshoot effectively and efficiently, ensuring smooth operation and minimizing downtime.

The Frequently Asked Questions (FAQ) section aims to provide answers to common queries and concerns you may have. Browse through the topics below to find clear and concise explanations that can help streamline your experience with us.