Overview
The Python Transformation feature in Informatica Cloud Data Integration (CDI) is supported exclusively within CDI-Elastic (CDI-E). This transformation allows users to write and execute Python scripts as part of their data integration workflows, enabling advanced data processing and customization. It is specifically designed for use in advanced mode, which utilizes Spark on Kubernetes.
Key use cases include:
- Processing complex logic or advanced data manipulations directly within the integration pipeline.
- Leveraging Python libraries for statistical, machine learning, or natural language processing tasks.
Pre-Requisites
- CDI-Elastic (CDI-E) Advanced Mode Environment: Python Transformation is supported only in elastic mappings of CDI-E and does not work in standard CDI mappings.
- Linux-Based Secure Agent: Python Transformation can only be configured on Linux-based Secure Agents; it is not supported for Windows-based Secure Agent setups.
- Python Installation: Set up Python (e.g., Python 3.6.5 or later) on the Secure Agent machine:
- Install Python binaries in the <Secure Agent directory>/ext/python/ directory.
- Add resource files (if needed) to the same directory.
- Advanced Cluster Configuration: Mappings with Python transformations require an advanced cluster based on Kubernetes.
- Dependencies: Ensure proper installation of all required Python libraries and resource files. If utilizing a custom installation, specify the paths in the Secure Agent.
Self-Service Resources
Goals
- Enable robust customization in data pipelines using Python scripting.
- Provide a tool for executing advanced transformations and leveraging external Python libraries.
- Support use cases that require computation beyond standard transformations.
Outcome
- Functional integration of Python logic within CDI-E mappings.
- Enhanced capabilities for handling complex business logic and data transformations.
- Improved flexibility through utilization of Python's rich ecosystem of libraries.
- Integration Developer: For designing and implementing Python logic in CDI-E mappings.
- Cloud Architect: For setting up and managing advanced clusters and execution environments.
- Administrator: For configuring Secure Agent setups and ensuring infrastructure readiness.
Actions
Add to Favorites
Engagement Details
Catalog Type
Ask An Expert
Engagement Category
Feature Clarity
Products
Cloud Data Integration - Elastic (CDI-E)
Engagement Type
Ask An Expert
Adoption Stage
Implement
Focus Area
Adoption - Technical
Functional
Engagement ID
AAE-CDIE-003
Disclaimer
- All the topics covered in the Success Accelerators/Ask An Expert sessions are intended for guidance and advisory only. This is implicit and it will not be called out under the scope of each engagement.
- Customers need to include their relevant technical/business team members highlighted in each engagement topic to derive the best out of each engagement.
- Customers need to perform any hands-on work by themselves leveraging the guidance from these engagements.
- Customers need to work with Informatica Global Customer Support for any product bugs/issues and troubleshooting.