Complex Transforms. CoSort Supports Custom, Field-Level Transformation Functions.

Home » Solutions » Data Transformation » Custom Transforms

Custom Transforms

Complex Transformations at the Field Level through Custom Routines

Challenges:

Many ETL systems need complex transformations. Their custom coding often requires special script components, SQL procedures, or assemblies. In addition to learning and maintenance curves, these objects may not operate at the field level, in the same I/O pass as other transformations, or leverage the APIs of best-of-breed software.

You may have many data reengineering requirements that result from data profiling. But there may not be enough time or expertise available to write enough custom scanning, cleansing, transformation, security, translation or standardization routines to address these needs.

You may also need customized data protection that is either based on specific security circumstances, or requiring complex security functions beyond the encryption, masking, de-identification or filtering options available with your application.

Solutions:

The CoSort package's SortCL tool integrates and processes large file-based data sources in the staging area. While the tool simultaneously transforms, reports, and secures data at the field level, the opportunity for custom transforms is also at hand. SortCL's field-level function support allows you to plug in complex transforms and run them with your other data processing, presentation, and protection tasks.

You can specify custom transforms on the same field twice in one job script; i.e. during inrec (before sort, join, or report declaration) and outfile (e.g. after sorting). This allows you to transform each field in at least two different ways in that one I/O pass.

Note that the SortCL tool leverages its own field-level transform capability out of the box. In addition to default encryption and de-identification routines, there is a third-party library example. A Melissa Data address standardization library function call is documented in the SortCL manual. You can invoke similar transformation routines from Trillium, other vendors, or your own libraries using the C interface template.

From a data security standpoint, both SortCL and IRI's new FieldShield software allow field-level calls for built-in or alternative encryption libraries, custom data profiling, and your own protection functions (beyond what our language allows).

Bespoke development services are also available! If you do not have the time or expertise in API level programming to generate custom transformation routines, consider using IRI software engineers to design, test, and deliver field-level data transformation functions that meet your specific requirements.

IRI, The CoSort Company