Cloud Data Ingestion and Replication

To transfer a large number of files between on-premises and cloud repositories, use File Ingestion and Replication. When setting up a task, specify the source and target endpoints for the transfer. You can configure the task to transfer multiple files in batches to enhance performance. Additionally, you can schedule the task and configure it to perform actions like compression, decompression, encryption, or decryption of files. This setup allows for efficient file management and monitoring of transfers.

• High-performant ingestion for files from on-premises and cloud.
  
• Simple and easy way to build and manage File Ingestion and Replication tasks with a wizard-driven interface.
• Built-in capability for encryption, virus scans, and email alerting as well as compression/decompression.
• Supports the ability to handle files as and when they come in using File Listeners.

Easy:

• Easy 4-step wizard for data engineers to ingest files and use file listener.
• Supports file actions like compression/decompression, virus scans, and encryption/decryption with monitoring.

Efficient:

• High-performant way of ingesting files with support for various file formats.
• Leverages Mass File Transfer to move files from on-prem or cloud sources.

Cost-Effective:

• File Listener capability to transport files as events as and when they land in a directory.
• Orchestration support for downstream transformation on files.

• Ingest files that need to be transferred from cloud/on-premise repositories to Data lakes and data warehouses
• Transfer files from remote FTP/SFTP/FTPS servers with high performance and scalability
• Supports all file types of any size

File Ingestion and Replication supports a variety of different sources including Advanced FTP/SFTP/FTPS, Amazon S3, Azure Blob Storage, Azure Data Lake, Google Storage, HDFS, and even local files. Some of the key targets supported include Advanced FTP/SFTP/FTPS, Amazon S3, Amazon Redshift, Azure Data Lake, Google Cloud Storage, Google Big Query, Snowflake, and Databricks Delta. For more details on the supported sources and targets, see the KB article below:

You can perform several file-processing actions with the File Ingestion and Replication task:

• Compress: Compress files using Zip, Tar, Bzip2, or Gzip. Password protection is available for Zip files.
• Decompress: Decompress files using Unzip, Untar, Bunzip2, or Gunzip. Password protection is available for Unzip files.
• Encrypt: Encrypt files using Pretty Good Privacy (PGP), compatible with Gnu Privacy Guard (GPG). Please refer to the documentation below.
• Decrypt: Decrypt files using PGP, compatible with GPG. Please refer to the documentation below.
• File Operations: Flatten or rename files in the target directory, noting that flattening may overwrite files with the same name.
• Virus Scan: Detect viruses using the Internet Content Adaptation Protocol (ICAP). The file transfer fails if a virus is found.

You can manually run a File Ingestion and Replication task or schedule it to run at a specific time or when a file is ready. Notifications can be set up for task failures or infected file detections. A File Ingestion and Replication task can handle multiple jobs simultaneously to improve performance and scalability, though concurrent jobs might cause issues with duplicate files. Running batches of files in parallel can reduce processing time, with the maximum number of parallel batches determined by the fmi-task-max-pool-size and fmi-task-core-pool-size properties. High thread counts might impact the performance of other jobs on the same Secure Agent.

• Check that your organization has licenses for Mass Ingestion Files and the FMI packages.
• The Mass ingestion application is running on the Secure Agent.
• Source and target connections exist, based on the sources from where you want to transfer files and the targets to where you want to transfer files.

In the wizard, complete these steps:

• Enter the task name, project location, and runtime environment.
  
• Configure the source.
• Configure the target.
• Optionally, add file-processing actions.
• Optionally, set runtime options.
• Click Save anytime to save your progress. When done, click Finish to save and close the wizard.

• To run the task manually from the Task Details page, click Run.
• To run a task on a schedule, edit the task in the File Ingestion and Replication task wizard to associate the task with a schedule.
• To run a task using REST API, use REST API File Ingestion and Replication resources.

You can optimize the performance of File Ingestion and Replication tasks by configuring an appropriate file size and batch size.

Source File Size

For large source files, consider splitting them into multiple smaller files, as processing multiple small files is faster than processing a single large file. The recommended size for each split file is around 500 MB to 1 GB.

Batch size

When configuring a File Ingestion and Replication job to write to a target, the number of files written with each COPY command affects performance. You should specify a batch size in the source properties of the task to optimize this performance. The default batch size is 5. Snowflake Cloud Data Warehouse V2 Connector's maximum batch size is 1000 when writing from Amazon S3 or Azure Blob Storage. Similarly, the Databricks DB SQL Connector allows a maximum batch size of 1000 from these sources. For other sources, the batch size must be between 1 and 20.

Consider the following guidelines to optimize performance:

• When you specify a batch size, you must also ensure that the source file is split into multiple smaller files to optimize performance.
  
• For optimized performance, batch size should equal or close to the number of files to be loaded in the target.
• Increasing the number of parallel batches with batch size can further improve performance.
• For information about the appropriate batch size value that you can specify for different sources and targets in a File ingestion task, see the Local folder source properties topic in the File Ingestion help.

The job results for each File Ingestion and Replication task instance show the status, success, and error statistics. To view detailed information, click the task name on the My Jobs page, accessible from the Data Integration Home page, Monitor pages, or Mass Ingestion page in Operational Insights. You can also download the job.

Use File Ingestion and Replication resources to start a job, see the activity log, view, create, update and delete a task.