Cloud Data Ingestion and Replication

To transfer a large number of files between on-premises and cloud repositories, use File Ingestion and Replication. When setting up a task, specify the source and target endpoints for the transfer. You can configure the task to transfer multiple files in batches to enhance performance. Additionally, you can schedule the task and configure it to perform actions like compression, decompression, encryption, or decryption of files. This setup allows for efficient file management and monitoring of transfers.

• High-performant ingestion for files from on-premises and cloud.
  
• Simple and easy way to build and manage File Ingestion and Replication tasks with a wizard-driven interface.
• Built-in capability for encryption, virus scans, and email alerting as well as compression/decompression.
• Supports the ability to handle files as and when they come in using File Listeners.

You can perform several file-processing actions with the File Ingestion and Replication task:

• Compress: Compress files using Zip, Tar, Bzip2, or Gzip. Password protection is available for Zip files.
• Decompress: Decompress files using Unzip, Untar, Bunzip2, or Gunzip. Password protection is available for Unzip files.
• Encrypt: Encrypt files using Pretty Good Privacy (PGP), compatible with Gnu Privacy Guard (GPG). Please refer to the documentation below.
• Decrypt: Decrypt files using PGP, compatible with GPG. Please refer to the documentation below.
• File Operations: Flatten or rename files in the target directory, noting that flattening may overwrite files with the same name.
• Virus Scan: Detect viruses using the Internet Content Adaptation Protocol (ICAP). The file transfer fails if a virus is found.

You can manually run a File Ingestion and Replication task or schedule it to run at a specific time or when a file is ready. Notifications can be set up for task failures or infected file detections. A File Ingestion and Replication task can handle multiple jobs simultaneously to improve performance and scalability, though concurrent jobs might cause issues with duplicate files. Running batches of files in parallel can reduce processing time, with the maximum number of parallel batches determined by the fmi-task-max-pool-size and fmi-task-core-pool-size properties. High thread counts might impact the performance of other jobs on the same Secure Agent.

• Check that your organization has licenses for Mass Ingestion Files and the FMI packages.
• The Mass ingestion application is running on the Secure Agent.
• Source and target connections exist, based on the sources from where you want to transfer files and the targets to where you want to transfer files.

You can optimize the performance of File Ingestion and Replication tasks by configuring an appropriate file size and batch size.

Source File Size

For large source files, consider splitting them into multiple smaller files, as processing multiple small files is faster than processing a single large file. The recommended size for each split file is around 500 MB to 1 GB.

Batch size

When configuring a File Ingestion and Replication job to write to a target, the number of files written with each COPY command affects performance. You should specify a batch size in the source properties of the task to optimize this performance. The default batch size is 5. Snowflake Cloud Data Warehouse V2 Connector's maximum batch size is 1000 when writing from Amazon S3 or Azure Blob Storage. Similarly, the Databricks DB SQL Connector allows a maximum batch size of 1000 from these sources. For other sources, the batch size must be between 1 and 20.

Consider the following guidelines to optimize performance:

• When you specify a batch size, you must also ensure that the source file is split into multiple smaller files to optimize performance.
  
• For optimized performance, batch size should equal or close to the number of files to be loaded in the target.
• Increasing the number of parallel batches with batch size can further improve performance.
• For information about the appropriate batch size value that you can specify for different sources and targets in a File ingestion task, see the Local folder source properties topic in the File Ingestion help.