Awesome

Notice

To better serve Wise business and customer needs, the PipelineWise codebase needs to shrink. We have made the difficult decision that, going forward many components of PipelineWise will be removed or incorporated in the main repo. The last version before this decision is v0.64.1

We thank all in the open-source community, that over the past 6 years, have helped to make PipelineWise a robust product for heterogeneous replication of many many Terabytes, daily

pipelinewise-transform-field

Transformation component between Singer taps and targets.

This is a PipelineWise compatible component.

How to use it

The recommended method of running this component is to use it from PipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files, and most of things are automated. Please check the related documentation at Transformations

If you want to run this Singer compatible component independently please read further.

Install

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a virtualenv:

  python3 -m venv venv
  pip install pipelinewise-transform-field

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip setuptools
  pip install .

To validate transformations

transform-field --validate --config [config.json] --catalog [catalog.json]

To run

Put it between a tap and a target with simple unix pipes:

some-singer-tap | transform-field --config [config.json] | some-singer-target

It's reading incoming messages from STDIN and using config.json to transform incoming RECORD messages.

Note: To avoid version conflicts run tap, transform and targets in separate virtual environments.

Transformation types

The following are the transformation types supported by pipelinewise-transform-field:

SET-NULL: Transforms any input to NULL
HASH: Transforms string input to hash
HASH-SKIP-FIRST-n: Transforms string input to hash skipping first n characters, e.g. HASH-SKIP-FIRST-2
MASK-DATE: Replaces the months and day parts of date columns to be always 1st of Jan
MASK-NUMBER: Transforms any numeric value to zero
MASK-HIDDEN: Transforms any string to 'hidden'
MASK-STRING-SKIP-ENDS-n: Transforms string input to masked version skipping first and last n characters, e.g. MASK-STRING-SKIP-ENDS-3

PS: 1 =< n =< 9

Conditional transformations

It is possible to transform a record's property based on some given condition(s), the transformation will only take place when all conditions are met.

A condition is a combination of:

column [required]: the field to look up to
operation [required]: the comparison type to use, the supported ones are equals and regex_match.
value [required]: the column value to look for in records.

An equality condition on a column

{
  "column": "<some column name>",
  "equals": <some important value>
}

A regex condition on a column

{
  "column": "<some column name>",
  "regex_match": "<some regex pattern>"
}

A condition on a property within a JSON-type column

{
  "column": "<some column name>",
  "field_path": "<xpath to property within 'column' object>",
  "equals": <some important value>
}

Configuration

You need to define which columns have to be transformed by which method and in which condition the transformation needs to be applied.

Basic transformation

A basic transformation is where a field in all a stream records will be transformed can be achieved with:

{
  "tap_stream_name": "<stream ID>",
  "field_id": "<Name of the field to transform in the record>",
  "type": "<Transformation type>"
}

Transformation within JSON

In order to transform property(ies) within a JSON type field, you can make use of field_paths property:

{
  "tap_stream_name": "<stream ID>",
  "field_id": "<Name of the field to transform in the record>",
  "field_paths": ["xpath to property 1", "xpath to property 2"],
  "type": "<Transformation type>"
}

Conditional Transformation

To apply transformation conditionally, you can make use of the property when which can have one or many conditions:

{
  "tap_stream_name": "<stream ID>",
  "field_id": "<Name of the field to transform in the record>",
  "type": "<Transformation type>",
  "when": [
    {"column": "string_col_1", "equals": "some value"},
    {"column": "string_col_2", "regex_match": ".*PII.*"},
    {"column": "numeric_col_1", "equals": 33},
    {"column": "json_column", "field_path": "metadata/comment", "regex_match": "sensitive"}
  ]
}

Sample config config.json

(Tip: PipelineWise generating this for you from a more readable YAML format)

To check code style:

Install python dependencies in a virtual env

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip setuptools
  pip install .[test]

Run pylint

pylint transform_field

To run tests:

Install python dependencies in a virtual env and run unit and integration tests

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip setuptools
  pip install .[test]

Run tests:

Unit tests

  pytest -v tests/unit

Integration tests

  pytest -v tests/integration

All tests

  pytest -v tests

License

Apache License Version 2.0

See LICENSE to see the full text.