New Features in Data Engineering Streaming 10.5.1 are as follows:

• Support for the following transformations in Databricks: Address Validator, Classifier, Parser, Standardizer
• Support for intelligent structure models on Databricks: You can incorporate an intelligent structure model in a Confluent Kafka, Kafka, Azure Event Hubs, and Amazon Kinesis Stream data object for streaming mappings that run on the Databricks Spark engine.

Click here to read more about the new features in Data Engineering Streaming 10.5.1

Informatica DEI/DEQ/DES What's New and Changed (Release Guide)

Informatica - 10.5.1- What's New and Changed

Changes in Intelligent Structure Models 10.5.1:

• The intelligent structure models might transform dates and times to output formats different from the output formats in previous versions.
• For more information about intelligent structure models, see the Data Engineering Integration 10.5.1 User Guide.

Click here to read more about the changes in Intelligent Structure Models in Informatica 10.5.1

Product Availability Matrix

Informatica's products rely upon and interact with an extensive range of products supplied by third-party vendors. Examples include database systems, ERP applications, web and application servers, browsers, and operating systems. Support for major releases of third-party products is determined and published in Informatica's Product Availability Matrix (PAM).

The PAM states which third-party product release is supported with a specified version of an Informatica product.

Refer to the Informatica 10.5.1 Product Availability Matrix

Product Name Change

Effective in version 10.4.0, the Informatica Big Data product family is renamed Data Engineering.

Click here to learn more.

Prerequisites

Complete the following steps before the upgrade:

• Back-Up the Data Transformation Files
• Back-Up Model Repository
• Prepare the Data Integration Service
• Prepare the Reference Data Directories
• Back-Up Databases
• Prepare the Domain

Click here to read in detail about the prerequisites for the upgrade.

You can directly upgrade to 10.5.1 from Informatica 10.1.1, 10.2.0, 10.2.1, 10.2.2, 10.4.0, and 10.4.1.

• 10.1.1  -> 10.5.1 
• 10.2.0  -> 10.5.1 
• 10.2.1  -> 10.5.1 
• 10.2.2  -> 10.5.1 
• 10.4.0  -> 10.5.1 
• 10.4.1  -> 10.5.1

IMPORTANT NOTE: If Data Engineering is in the same domain as any other product supported by the Informatica installer, like Enterprise Data Catalog, Enterprise Data Preparation, and Data Privacy Management, verify that all products are upgraded to the same supported upgrade version before you upgrade to 10.5.

If the product version that is currently installed cannot be upgraded to Informatica 10.5, you must first upgrade to a supported version (say, 10.4.1) and then move to the latest version.

Click here for complete information on the steps to upgrade from the older versions (Docs Home > Shared Content for Data Engineering > 10.4.1 > List of Guides > Installation and Upgrade).

Upgrade from 10.1.1

Part 1: How to upgrade DEI from 10.1.1 to 10.4.0/10.4.1

Part 2: How to upgrade DEI from 10.1.1 to 10.4.0/10.4.1

Note: Though the Video KBs explain upgrading to 10.4.0, the same process would also apply for the 10.5 upgrades.

You can upgrade to 10.5.1 from the following versions.

• Upgrade from 10.1.1 to 10.5.1
• Upgrade from 10.2.0 to 10.5.1
• Upgrade from 10.2.1 to 10.5.1
• Upgrade from 10.2.2 to 10.5.1
• Upgrade from 10.4.0 to 10.5.1
• Upgrade from 10.4.1 to 10.5.1

From 10.1.1 => 10.5.1 

Pre-Steps

• Change Hive Execution Engine to 'Spark'/'Blaze'. For a bulk update, refer to the following 'INFA Command Reference Guide' (or) the following KB article:

Informatica Command Reference Guide - enableMappingValidationEnvironment

HOW TO: Update the mapping engine to Spark/Blaze using infacmd command

Post-Steps

• Create 'Metadata Access Service' (MAS) to import Hive/HDFS objects. For more information, refer to the following video KB:

Introduction to Metadata Access Service

• Create 'Cluster Configuration Object` (CCO) for the Hadoop Cluster. For more information, refer to the following guide (or) KB article:

Informatica Administrator Guide - Create Cluster Configuration

HOW TO: Create CCO using 'Import from Archive File' option in Informatica DEI

• Associate existing 'Hadoop'/'Hive'/'HDFS' connections to the newly created CCO. 

CCO details could be updated using the 'infacmd isp UpdateConnection' command, like below:

${infa_domain_home}/isp/bin/infacmd.sh isp updateConnection -dn ${infa_domain_name} -un ${infacmd_user_name} -sdn ${infacmd_user_security_domain} -cn ${hive_or_hdfs_or_hadoop_conn_name} -o "clusterConfigId='${new_cco_id}'"

For more information, refer the following documents:

infacmd cluster listAssociatedConnections, infacmd ispupdateConnection, infacmd ispListConnectionOptions (to view the connection attributes)

• Migrate the existing Hadoop pushdown job configurations from the 'hadoopEnv.properties'file to the 'Hadoop' Connection. For more information, refer to the following document:

hadoopEnv.properties

• If the 'INFA_HOME' path has been changed during the upgrade, perform the following action:

Copy the existing jar files from '$OLD_INFA_HOME/externaljdbcjars'  location into '$NEW_INFA_HOME/externaljdbcjars'  folder. 

• Convert 'Native' type Physical Data Objects (PDO) -> 'Sqoop-enabled' JDBC object(s), in case of converting mappings to Spark mode. 

• If Analyst Service is used to interact with Hadoop cluster services like Hive, then update the JVM Argument '-DHADOOP_DIST_DIR'  to the closest available version if the exact version is not available. For more information, refer to the following documents:

Informatica Application Services Guide - Analyst Service - Update Advanced Properties 

ERROR: "java.lang.RuntimeException: [Cannot find the distribution file: CDH_5.13/infa_resources.xml]..." while testing the hive connection in the Analyst tool

From 10.2.0 => 10.5.1 

Pre-Steps

• Change Hive Execution Engine to 'Spark'/'Blaze'. For a bulk update, refer to the following 'INFA Command Reference Guide' (or) the following KB article:

Informatica Command Reference Guide - enableMappingValidationEnvironment

HOW TO: Update the mapping engine to Spark/Blaze using infacmd command

Post-Steps  

• Create 'Metadata Access Service' (MAS) to import Hive/HDFS objects. For more information, refer to the following Video KB:

Introduction to Metadata Access Service

• Migrate the existing Hadoop pushdown job configurations from the 'hadoopEnv.properties' file to the 'Hadoop' Connection. For more information, refer to the following  document:
  hadoopEnv.properties

• Ensure that 'Distribution type' and 'Distribution Version' in the CCO match the Cluster type (CDH/HDP/EMR/HDInsights) & its closest available version if the exact version is not available. For more information, refer to the following KB article:

FAQ: Can the same DIS be used for connecting multiple Hadoop clusters from Informatica 10.2.1?

• Refresh 'Cluster Configuration Object' (CCO) for the Hadoop Cluster. For more information, refer to the following guide (or) KB article:

Informatica Administrator Guide - Refresh Cluster Configuration

HOW TO: Refresh CCO using 'Import from Archive File' option in Informatica DEI

• If the 'INFA_HOME' path has been changed during the upgrade, perform the following action: 

Copy the existing jar files '$OLD_INFA_HOME/externaljdbcjars' location into '$NEW_INFA_HOME/externaljdbcjars' folder.

Copy the Hadoop 'site-xml' files, if present, from '$OLD_INFA_HOME/services/shared/hadoop/[distribution]/conf' folder into corresponding location of upgraded new INFA_HOME location.

• Convert 'Native' type Physical Data Objects (PDO) -> 'Sqoop-enabled' JDBC object(s), in case of converting mappings to Spark mode.
• If Analyst Service is used to interact with Hadoop cluster services like Hive, then update the JVM Argument '-DHADOOP_DIST_DIR' to the closest available version exact version is not available. For more information, refer to the following documents:

Informatica Application Services Guide - Analyst Service - Update Advanced Properties

ERROR: "java.lang.RuntimeException: [Cannot find the distribution file: CDH_5.13/infa_resources.xml]..." while testing the hive connection in the Analyst tool

From 10.2.1 => 10.5.1

Pre-Steps

• Change Hive Execution Engine to 'Spark'/'Blaze'. For a bulk update, refer to the following 'INFA Command Reference Guide' (or) the following KB article:

Informatica Command Reference Guide - enableMappingValidationEnvironment

HOW TO: Update the mapping engine to Spark/Blaze using infacmd command

Post-Steps

• Migrate the existing Hadoop pushdown job configurations from the 'hadoopEnv.properties' file to the 'Hadoop' Connection. For more information, refer to the following document: 
  hadoopEnv.properties

• Ensure that 'Distribution type' and 'Distribution Version' in the CCO match the Cluster type (CDH/HDP/EMR/HDInsights) & its closest available version if the exact version is not available. For more information, refer to the following KB article:

FAQ: Can the same DIS be used for connecting multiple Hadoop clusters from Informatica 10.2.1?

• If the 'INFA_HOME' path has been changed during the upgrade, perform the following action:

Copy the existing jar files from '$OLD_INFA_HOME/externaljdbcjars'  location into '$NEW_INFA_HOME/externaljdbcjars'  folder.

Copy the Hadoop 'site-xml' files, if present, from '$OLD_INFA_HOME/services/shared/hadoop/[distribution]/conf' folder into corresponding location of upgraded new INFA_HOME location.

• Convert 'Native' type Physical Data Objects (PDO) -> 'Sqoop-enabled' JDBC object(s), in case of converting mappings to Spark mode.
• If Analyst Service is used to interact with Hadoop cluster services like Hive, then update the JVM Argument '-DHADOOP_DIST_DIR' to the closest available version exact version is not available. For more information, refer to the following documents:

Informatica Application Services Guide - Analyst Service - Update Advanced Properties

ERROR: "java.lang.RuntimeException: [Cannot find the distribution file: CDH_5.13/infa_resources.xml]..." while testing the hive connection in the Analyst tool

From 10.2.2 => 10.5.1

Post-Steps

• Ensure that 'Distribution type' and 'Distribution Version' in the CCO match the Cluster type (CDH/HDP/EMR/HDInsights) & its closest available version if the exact version is not available. For more information, refer to the following KB article:

FAQ: Can the same DIS be used for connecting multiple Hadoop clusters from Informatica 10.2.1?

• Convert 'Native' type Physical Data Objects (PDO) -> 'Sqoop-enabled' JDBC object(s), in case of converting mappings to Spark mode.
• If Analyst Service is used to interact with Hadoop cluster services like Hive, then update the JVM Argument '-DHADOOP_DIST_DIR' to the closest available version exact version is not available. For more information, refer to the following documents:

Informatica Application Services Guide - Analyst Service - Update Advanced Properties

ERROR: "java.lang.RuntimeException: [Cannot find the distribution file: CDH_5.13/infa_resources.xml]..." while testing the hive connection in the Analyst tool

From 10.4.0 => 10.5.1

Post-Steps 

• Ensure that 'Distribution type' and 'Distribution Version' in the CCO matches the Cluster type (CDH/HDP/EMR/HDInsights) & its closest available version if the exact version is not available. For more information, refer to the following KB article:

FAQ: Can the same DIS be used for connecting multiple Hadoop clusters from Informatica 10.2.1?

• Convert 'Native' type Physical Data Objects (PDO) -> 'Sqoop-enabled' JDBC object(s), in case of converting mappings to Spark mode. 
• If Analyst Service is used to interact with Hadoop cluster services like Hive, then update the JVM Argument '-DHADOOP_DIST_DIR' to the closest available version exact version is not available. For more information, refer to the following documents:

Informatica Application Services Guide - Analyst Service - Update Advanced Properties

ERROR: "java.lang.RuntimeException: [Cannot find the distribution file: CDH_5.13/infa_resources.xml]..." while testing the hive connection in the Analyst tool

From 10.4.1 => 10.5.1

Post-Steps

• Ensure that 'Distribution type' and 'Distribution Version' in the CCO match the Cluster type (CDH/HDP/EMR/HDInsights) & its closest available version if the exact version is not available. For more information, refer to the following KB article:

FAQ: Can the same DIS be used for connecting multiple Hadoop clusters from Informatica 10.2.1?

• Convert 'Native' type Physical Data Objects (PDO) -> 'Sqoop-enabled' JDBC object(s), in case of converting mappings to Spark mode. 
• If Analyst Service is used to interact with Hadoop cluster services like Hive, then update the JVM Argument '-DHADOOP_DIST_DIR' to the closest available version exact version is not available. For more information, refer to the following documents:

Informatica Application Services Guide - Analyst Service - Update Advanced Properties

ERROR: "java.lang.RuntimeException: [Cannot find the distribution file: CDH_5.13/infa_resources.xml]..." while testing the hive connection in the Analyst tool

Upgrade Checklist

The upgrade checklist summarizes the tasks that you must perform to complete an upgrade. If you upgrade the Informatica product on more than one machine, complete the first upgrade using the detailed instructions in this guide. You can use this checklist to perform subsequent upgrades.

Click here to go through the Upgrade Checklist

Additional Resources

Click here to review all the known issues in the 10.5.1 release.

Data Engineering Integration Known Issues (10.5.1)

Data Engineering Streaming Known Issues (10.5.1)

Third Party Known Issues (10.5.1)

Deploy the Informatica DEI Solution on the AWS Cloud Marketplace (10.4.1)

Deploy the Informatica DEI Solution on the Microsoft Azure Marketplace (10.4.1)

How to Install DEI on Kubernetes with the Container Utility (10.4.0 - 10.4.1)

How to Install DEI on Docker with the Container Utility (10.4.0 - 10.4.1)

Q1. Any options are available for a Bulk update of the mapping execution engine from 'Hive' to 'Spark'/'Blaze.'

A.Yes, 'infacmd mrs enableMappingValidationEnvironment'could be used to modify the execution Engine of the existing mapping. 

Following is the example command that would help to update the Hadoop Mapping execution engine: 

$INFA_HOME/isp/bin/infacmd.sh mrs enableMappingValidationEnvironment -dn D_Scorpius -sn MRS_Scorpius -un ths -pd ***** -pn xyz -cn HDP -mnf Mapping -ve spark -eef hadoop 

       Examining project [xyz]. 

       For project [xyz]: 

       Found [5] mappings. [1] mappings 

       matched the filter(s). Modified[1] mappings. 

       Command ran successfully 

Note: In the example, 'infacmd' is used to update the mappings that are run in 'Hadoop' mode. Hadoop execution mode mappings, whose name starts with the prefix 'Mapping' & present in the project - 'xyz' - would be switched to 'Spark' mode. 

For more information on the usage of the command, refer to the following 'INFA Command Reference Guide' (or) the following KB article:

• Informatica Command Reference Guide - enableMappingValidationEnvironment
• HOW TO: Update the mapping engine to Spark/Blaze using the infacmd command

Q2. How to proceed when 'infacmd enableMappingValidationEnvironment' is not updating all the required mappings?

A. The issue could occur if the MRS Search Index issue is not up to date. To resolve the same, it would be required to perform a 'Re-Index' operation for the MRS search index. Following is the link to the related KB on the issue:

• ERROR: "Found [0] objects. [0] objects matched the filter(s)" while running 'infacmd enableMappingValidationEnvironment' command in Informatica DEI

If the issue still occurs, even after re-indexing the MRS search index, as mentioned in the KB, then raise a new case with Informatica Global Customer Support, along with the MRS Backup file & MRS log. 

Q3. Hadoop Cluster has been upgraded recently to the HDP 3.1.x (or) CDP 7.1.x version. Are any specific details I should be aware of before running pushdown mappings from Informatica?

A. For details on the HDP 3.1.x related changes, feature support, refer to the following KB article:

• FAQ: What Should Data Engineering (Big Data) Customers Know About Upgrading to Hortonworks HDP 3.1?

For 'Cloudera Data Platform' (CDP) integration with Informatica DEI, perform the integration tasks mentioned in the following guide: 

•  Cloudera CDP Integration Tasks

Q4. 'Cluster Configuration Object' (CCO) using the 'Import from Cluster' option is not working as expected for the CDP Hadoop cluster.

A. In specific scenarios, say when Compute Cluster is used, creating CCO for CDP using the 'Import from Cluster' option might not work as expected (or) some properties might be missing in the 'Hadoop site-xml' files, part of the newly created CCO. 

Create/Refresh CCO using the 'Import from Archive' option in the Informatica Administrator console to resolve the issue. For more information, refer to the following KB articles, which details the steps to be followed for creating/refreshing 'CCO' using 'site-xml' files downloaded from the Hadoop cluster:

• HOW TO: Create CCO using 'Import from Archive File' option in Informatica DEI
• HOW TO: Refresh CCO using 'Import from Archive File' option in Informatica DEI

Q5. HDP Cluster has been upgraded to CDP 7.1.x cluster. Any post-integration steps are needed in Informatica before running the mapping?

A. After in-place upgrade of HDP cluster to CDP, perform the following steps in Informatica Domain:

• Create a new 'Cluster Configuration Object' (CCO) for CDP cluster - Use 'Import from Cluster' (or) 'Import from Archive' option in Admin console. 
• Once CCO for the CDP cluster is created, modify the 'Cluster Configuration' attribute of the existing 'Hadoop'/'Hive'/'HDFS' connections to use the newly created CCO corresponding to the CDP cluster. 

To automate the CCO update for the connections, following ‘infacmd’ commands could be used: 

infacmd cluster listAssociatedConnections

infacmd isp UpdateConnection

'infacmd cluster listAssociatedConnections' could be used to get all the 'Hive’/’HDFS’/’Hadoop' type connections associated with a given HDP Cluster CCO. 

${infa_domain_home}/isp/bin/infacmd.sh cluster listAssociatedConnections -dn ${infa_domain_name} -un ${infacmd_user_name} -pd ${infacmd_user_password} -sdn ${infacmd_user_security_domain} -cn ${hdp_cco_name} 

Once the details of connections are obtained, CCO details could be updated using the 'infacmd isp UpdateConnection' command.

${infa_domain_home}/isp/bin/infacmd.sh isp updateConnection -dn ${infa_domain_name} -un ${infacmd_user_name} -sdn ${infacmd_user_security_domain} -cn ${hive_or_hdfs_or_hadoop_conn_name} -o "clusterConfigId='${new_cdp_cco_id}'"

For more information, refer: infacmd cluster listAssociatedConnections, infacmd isp updateConnection, infacmd isp ListConnectionOptions (to view the connection attributes)