Challenge
The z/OS and i5/OS platforms have been and continue to be very powerful platforms that host a significant amount of valuable corporate data. Access demand to the data contained on these platforms from distributed systems and cloud-based environments has grown significantly. Many sites have implemented Informatica’s PowerExchange and PowerCenter to meet this demand making z/OS and i5/OS based data available to other platforms and target databases. The amount of data contained in z/OS and i5/OS sources is typically very large. The process of refreshing the data in these target databases, on a regular schedule, is not often feasible because of the system resources required along with the amount of time it takes to complete. Sites have also implemented PowerExchange Change Data Capture (CDC) to maintain the data contained in the target databases. PowerCenter is then used to consume the CDC data that has been captured by the z/OS and i5/OS PowerExchange environments.
Description
To reduce the number of resources being consumed, remote logging should be implemented for z/OS and i5/OS CDC sources. The PowerExchange Logger running on the Linux, UNIX, or Windows is used to extract CDC data from the z/OS and i5/OS platforms and relog that data to a less costly system. Multiple PowerCenter CDC sessions can then use the PowerExchange Listener running on Linux, UNIX, or Windows to retrieve the CDC data from the remote logger.
Below is a remote logging architecture diagram from a z/OS perspective.
For z/OS and i5/OS sources, the remote logging of CDC data to a Linux, UNIX, or Windows system has the following benefits:
- Moves resource-intensive, column-level processing and Unit-of-Work (UOW) Cleanser processing off of the z/OS or i5/OS system onto the Linux, UNIX, or Windows system where the PowerExchange Logger runs.
- Extracts change data from the z/OS PowerExchange Logger log files or DB2 for i5/OS journal receivers in a single pass and transmits that data over the network to the PowerExchange Logger running on Linux, UNIX, or Windows. The data is then available for PowerCenter CDC sessions to process thru the PowerExchange Listener running on Linux, UNIX, or Windows. This single-pass processing reduces network traffic and avoids the overhead of multiple data extraction reads.
- Reduces costly CPU usage, disk space, and CDC processing time on the z/OS or i5/OS source system.
After remote logging has been implemented for z/OS or i5/OS CDC sources, the CDC data becomes available for consumption by other Informatica on-premises and cloud-based products. Examples of these products include PowerExchange CDC Publisher and Informatica Intelligent Cloud Services (IICS).
For additional information on remote logging, refer to the “Remote Logging of Data” chapter in either the PowerExchange CDC Guide for z/OS or the PowerExchange CDC Guide for i5/OS.
Below are diagrams and process flows that describe the remote logging process from a z/OS perspective. Several of the diagrams and process flows also describe how the CDC data is retrieved from the remote logger. In an i5/OS environment, the process is conceptually the same.
Remote Logging Process Flow from a z/OS PowerExchange CDC Environment to a Linux PowerExchange Logger
| Activity | Activity Description |
| 1 | The PowerExchange Logger loads the dbmover.cfg and pwxccl.cfg configuration files into memory. |
| 2 | The PowerExchange Logger uses the entries from the CAPTURE_NODE, CAPTURE_NODE_UID, and CAPTURE_NODE_PWD to connect to the z/OS PowerExchange Listener and request a copy of the z/OS Change Registrations. |
| 3 | The PowerExchange Logger loads all of the Change Registrations into memory. |
| 4 | The PowerExchange Logger retrieves the latest restart tokens file its CDC control files and sends the z/OS PowerExchange Listener a request for CDC data starting from restart tokens contained in the request. |
| 5 | The z/OS PowerExchange Listener sends the requested CDC data to the PowerExchange Logger. |
| 6 | The PowerExchange Logger performs the UOW Cleanser process, writes the CDC data to its condense data files and periodically updates its control files. |
| 7 | After the PowerExchange Logger updates its control files, it writes a process message into the PowerExchange Logger message file.
|
The PowerExchange Logger repeats activities 5 thru 7 until the PowerExchange Logger is stopped.
CDC Data Process Flow from a Linux PowerExchange Logger to a PowerCenter CDC Session
| Activity | Activity Description |
| 1 | The PowerCenter Server CDC workflow retrieves its latest CDC restart tokens and creates a request for CDC data. The request contains the restart tokens, all of the required CDC sources along with the Location, User Name, Password, Map Location, Map Location User, and Map Location Password connection attributes. PowerCenter sends this request to the PowerExchange Client. |
| 2 | The PowerExchange Client uses its dbmover.cfg configuration file to determine what PowerExchange Listener to forward this request to base on the Location value. |
| 3 | The PowerExchange Client then forwards the request to the specified PowerExchange Listener. |
| 4 | The PowerExchange Listener generates an entry in its message log that the PowerCenter CDC workflow has started requesting data. |
| 5 | The PowerExchange Listener uses the Map Location, Map Location User, and Map Location Password to connect to the z/OS PowerExchange Listener and request a copy of the required CDC Extraction Maps. |
| 6 | The z/OS PowerExchange Listener retrieves a copy of the requested CDC Extraction Maps. |
| 7 | The z/OS PowerExchange Listener sends them back to the requesting PowerExchange Listener. The PowerExchange Listener loads the CDC Extraction Maps into memory. |
| 8 | The PowerExchange Listener uses the restart tokens to check the CDC control files to determine which CDC condense data file contains the requested data. The PowerExchange Listener uses this information as a starting point to retrieve the requested CDC data from its CDC condense data files. |
| 9 | The PowerExchange Listener sends the requested CDC data to the requesting PowerExchange Client. |
| 10 | The PowerExchange Client sends the CDC data to the requesting PowerCenter CDC workflow.
|
The PowerExchange Listener repeats activities 8 thru 10 until the requesting PowerCenter CDC workflow is stopped. When the PowerCenter CDC workflow stops, the PowerExchange Listener generates an entry in its message log that the PowerCenter CDC workflow has stopped.
CDC Data Process Flow from a Linux PowerExchange Logger to a IICS CDC Workflow Task
| Activity | Activity Description |
| 1 | The IICS CDC workflow task retrieves its latest CDC restart tokens and creates a request for CDC data. The request contains the restart tokens, for the required CDC sources, along with the Listener Location, User Name, Password, Collection Name, CAPI Connection Name, Map Location, Map Location User, and Map Location Password attributes from the Secure Agent connection being used. |
| 2 | Secure Agent sends this request to the PowerExchange Listener. |
| 3 | The PowerExchange Listener generates an entry in its message log that the IICS CDC workflow task has started requesting data. |
| 4 | The PowerExchange Listener uses the Map Location, Map Location User, and Map Location Password to connect to the z/OS PowerExchange Listener and request a copy of the required CDC Extraction Maps. |
| 5 | The z/OS PowerExchange Listener retrieves a copy of the requested CDC Extraction Maps. |
| 6 | The z/OS PowerExchange Listener sends them back to the requesting PowerExchange Listener. The PowerExchange Listener loads the CDC Extraction Maps into memory. |
| 7 | The PowerExchange Listener uses the restart tokens to check the CDC control files to determine which CDC condense data file contains the requested data. The PowerExchange Listener uses this information as a starting point to retrieve the requested CDC data from its CDC condense data files. |
| 8 | The PowerExchange Listener sends the requested CDC data to the requesting Secure Agent. The Secure Agent sends the requested CDC data to the IICS CDC workflow task.
|
This PowerExchange Listener repeats activities 7 thru 8 until the IICS CDC workflow task is stopped.
CDC Data Process Flow from a Linux PowerExchange Logger to a PowerExchange CDC Publisher Process
| Activity | Activity Description |
| 1 | The PowerExchange CDC Publisher retrieves its latest CDC restart tokens and creates a request for CDC data. The request contains the restart tokens, all of the required CDC sources along with the Extract.pwxNodeLocation, Extract.pwxNodeUserId, Extract.pwxNodePwd, Extract.pwxXmapLocation, Extract.pwxXmapUserId, and Extract.pwxXmapPwd properties in the cdcPublisherPowerExchange configuration file. The PowerExchange CDC Publisher sends this request to the PowerExchange Listener/Client. |
| 2 | The PowerExchange Listener generates an entry in its message log that the PowerExchange CDC Publisher process has started requesting data. |
| 3 | The PowerExchange Listener uses the Extract.pwxXmapLocation, Extract.pwxXmapUserId, and Extract.pwxXmapPwd properties to connect to the z/OS PowerExchange Listener and request a copy of the required CDC Extraction Maps. |
| 4 | The z/OS PowerExchange Listener retrieves a copy of the requested CDC Extraction Maps. |
| 5 | The z/OS PowerExchange Listener sends them back to the requesting PowerExchange Listener. The PowerExchange Listener loads the CDC Extraction Maps into memory. |
| 6 | The PowerExchange Listener uses the restart tokens to check in the CDC control files to determine which CDC condense data file contains the requested data. The PowerExchange Listener uses this information as a starting point to retrieve the requested CDC data from its CDC condense data files. |
| 7 | The PowerExchange Listener/Client sends the requested CDC data to the requesting PowerExchange CDC Publisher task.
|
The Remote PowerExchange Listener/Client repeats activities 6 thru 7 until the PowerExchange CDC Publisher process is stopped. When the PowerExchange CDC Publisher stops, the PowerExchange Listener generates an entry in its message log that the PowerExchange CDC Publisher has stopped.