/
Performance Evaluation

Performance Evaluation

Owned by Robin Rielley
Jul 14, 2020
1 min read

Setup

Hardware used for measuring the performance:

  • 2.9 GHz Intel Core i5

  • 16 GB 2133 MHz LPDDR3

  • Java 7

Light Data Transformation DMD

These are the high-level transformations being performed on the data:

  • Parsing of CSV

  • Drop columns

  • Setting defaults on column

  • Changing case

  • Masking data

  • Filtering rows based on an expression

Directives

parse-as-csv demo , true drop demo drop demo_12 fill-null-or-empty demo_11 N/A uppercase demo_17 mask-number demo_18 xxx### drop demo_6 drop demo_7 fill-null-or-empty demo_5 N/A uppercase demo_3 filter-row-if-true demo_9 =~ "CA" mask-number demo_10 xxx## mask-shuffle demo_4

Experiments

These two experiments were run: the first with 13M records, and the second with 80M records.

Experiment #1

  • Number of records: 13,499,973

  • Number of bytes: 4,499,534,313 (~ 4GB)

  • Number of columns: 18

Performance Numbers