SortCL's Multi-Dimensional Table Lookup Function for Flat Files

Home » Solutions » Data Transformation » Table Lookups

Table Lookups

CoSort's Multi-Dimensional Lookup Functionality Substitutes Values Fast

Challenges:

Complex computations -- including the pseudonymization of field data, multi-table joins, and discrete solutions based on business logic -- can be expensive and slow. Where a simple lookup can replace a runtime computation, the performance gain can be significant since retrieving a value in memory can be faster than computing that value.

Lookups typically occur in ETL tools and databases. Lookup transformations must be specially configured, and extra steps may be required to improve their performance in volume.

Meanwhile, analogous functionality has not been available in the file system where more speed, resources, and simultaneous functionality may be available.

Solutions:

Multi-dimensional lookup table functionality is now available in external, flat file environments. Both IRI's FieldShield and CoSort (SortCL) software support lookup transformations during the inrec and output phases of a job script. Value substitutions can be made through several columns in a delimited set file.

This approach bypasses database tuning and integrity issues to deliver:

• discrete solutions
• a pseudonymization method for data security
• an alternative to joins across many tables

By running lookups in the file system through set files and an explicit job script, you can spend less time preparing for, and getting results from, your data substituion sources.

In addition, you can:

• combine lookups with other processes in the same I/O pass
• process resulting values in the same SortCL script
• immediately encrypt or otherwise protect the lookup results
• simultaneously format the lookup results in custom reports

IRI, The CoSort Company