Cloud Data Ingestion and Replication

• Unified solution for ingestion from a variety of messaging and IoT protocol sources with native connectivity
• Simple and easy way to build and manage streaming data ingestion with a wizard-driven interface
• Ability to apply simple transformations as part of the ingestion pipeline

Consider using Streaming ingestion and Replication in the following scenarios:

• Real-time analytics. Ingest streaming and IoT data into messaging systems, such as Apache Kafka or Amazon Kinesis, for real-time analytics. Real-time analytics can help companies identify business opportunities and revenue streams that can result in increased profits and improved customer service.
• Data integration. Ingest streaming and IoT data into cloud data lakes, such as Amazon S3, for integrating data in real-time to provide up-to-the-minute information.

In Data Integration, use the Streaming Ingestion and Replication task wizard to configure a streaming ingestion task. You define a source and a target, and you can optionally define a transformation to transform the data.

Transformations are part of a Streaming Ingestion and Replication task. Transformations represent the operations that you want to perform when ingesting streaming data. Each transformation performs a specific function. For example, a Filter transformation filters data from the ingested data based on a specified condition.

When you create a Streaming Ingestion and Replication task, adding a transformation is optional. Each transformation type has a unique set of options that you can configure. Refer to the below resources for more details about the transformations in Streaming Ingestion and Replication.

After you create a Streaming Ingestion and Replication task, you must deploy it on the Secure Agent for the task to run as a job. Before deploying the task, ensure that the Secure Agent configured for the task is running. You can undeploy a Streaming Ingestion and Replication job from the Secure Agent. When you undeploy a streaming ingestion job, the Secure Agent does not save the previous state histories of the job. So, when you deploy the streaming ingestion task again, the task runs as a completely new job

You can stop and resume Streaming Ingestion and Replication jobs. Stop or resume a job on the My Jobs page in Data Integration, the All Jobs and Running Jobs pages in Monitor, or the Mass Ingestion page in Operational Insights. When you stop a streaming ingestion job, the Secure Agent saves the previous state histories of the job. When you resume the stopped streaming ingestion job, the job starts running from the last saved state.

Monitor the progress, performance, and status of Data Ingestion and Replication jobs from the following services:

• Monitor: All Jobs and Running Jobs
• Operational Insights:  Overview and All Jobs
• Data Integration: Recent Jobs and My Jobs