Defining Development Environments

Challenge

The basic concept of isolating developers from testers, and both from the production system, is critical to development success in most integration projects. As Data Integration becomes more centralized within organizations often multiple integration projects, developers and maintenance personnel share the same platform. A poorly designed development environment can negatively impact the development process as well as production break-fix cycles.

Description

The task of defining the development environment is important to ensure separation and smooth development and enhancement work across competing integration efforts and projects. Because of the wide variance in corporate technical environments, standards, and objectives, there is no "optimal" development environment. Rather, there are key areas of consideration and decisions that must be made with respect to them to ensure the best fit for an organization.

After the development environment has been defined, it is important to document its configuration, including the information and procedures the developers need to use and follow within the environments. For example, developers need to understand what systems they are logging into, what databases they are accessing, what Org they are accessing, and where sources and targets reside. An important component of any development environment is to configure it as close to the test and production environments as possible given time and budget. This can significantly ease the development and integration efforts downstream and will ultimately save time and cost during the testing phases.

Considerations

The development environment for any data integration solution must consider many of the same issues as a "traditional" development project. Here are some best practices for implementing a development environment in Intelligent Data Management Cloud (IDMC):

• Separate organization for each environment: Running different organizations is the best way to separate development code from whatever is running in production. You can apply a different security policy to objects in production to lock them down. More importantly, a formal test and release management process can be implemented where code is promoted through separate environments using Informatica's object export functionality. This requires some discipline to keep in place, but the level of confidence and control it provides is worth it
• Leveraging Project & Folder structure. For users who wish to maintain asset versions within IDMC itself (as opposed to an external version control system), customers can leverage the folder structure to manually maintain and manage asset versions. This requires incorporating the version information into the folder names such that each folder represents a unique version. Such a versioning practice may work when the number of assets under the folder structure is small, and the frequency of changes is slow and far in-between. However, over time such a versioning practice becomes hard to manage and does not scale to enterprise needs. We recommend using your existing source control systems and adopting a well-defined, uniform team based SDLC (System Development Life Cycle) practice across your organization.
• Enable Source Control management: Source control is a feature by which changes made to IDMC objects such as projects, folders, and assets can be managed and tracked. IDMC does support native integration with Github & Azure DevOps.
• Secure Agents Security: Don’t default into assigning the secure agent full user or admin rights. For least privilege on Windows, the secure agent should run under the Network Service account - assuming it will need to consume any network resources. "Modify" file permissions to the following locations will also need to be granted:
  • The secure agent installation folder, e.g., C:\Program Files\Informatica Cloud Secure Agent.
  • Any folders that hold configuration files or data that will be used by tasks and mappings.
• Adopt some naming conventions: This is a shared development environment, so some consistency over naming should be asserted. This is particularly important given that views are the only means to organize objects. Whatever system is adopted, object names should clearly identify the project or application they are developed for, the type of connection, and the purpose.

IDMC Best Practices on naming conventions can be referred to for more details.

• Security Management. IDMC has built-in security features that allow an administrative user to define the access rights of all other users to objects in the IDMC. The organization of security groups should be carefully planned and implemented prior to the start of development. As an additional option, SSO can be used to assist in simplifying the organization of users and permissions.
• Plan memory size: If large integration jobs are run on a default installation of Informatica Cloud they may start failing with the following error:

[ERROR] java.lang.OutOfMemoryError: Java heap space

The secure agent runs in a Java virtual machine which means that you a chunk of memory is assigned to it. For some connectors this dictates the capacity of integration jobs as they load the entire data set into the heap. This can be increased by tweaking the start-up parameters for the virtual machine. The default installed size is 256MB which doesn’t go far – some organizations  expand this to as much as 8GB. This implies that secure agents can’t be thrown just anywhere – it is necessary to plan for the requirements and ensure they have enough resources allocated to them.