Performance Evaluation

Owned by Robin Rielley
Jul 14, 2020
1 min read

Setup

Hardware used for measuring the performance:

  • 2.9 GHz Intel Core i5

  • 16 GB 2133 MHz LPDDR3

  • Java 7

Light Data Transformation DMD

These are the high-level transformations being performed on the data:

  • Parsing of CSV

  • Drop columns

  • Setting defaults on column

  • Changing case

  • Masking data

  • Filtering rows based on an expression

Directives

parse-as-csv demo , true drop demo drop demo_12 fill-null-or-empty demo_11 N/A uppercase demo_17 mask-number demo_18 xxx### drop demo_6 drop demo_7 fill-null-or-empty demo_5 N/A uppercase demo_3 filter-row-if-true demo_9 =~ "CA" mask-number demo_10 xxx## mask-shuffle demo_4

Experiments

These two experiments were run: the first with 13M records, and the second with 80M records.

Experiment #1

  • Number of records: 13,499,973

  • Number of bytes: 4,499,534,313 (~ 4GB)

  • Number of columns: 18

Performance Numbers

count = 13,376,053 mean rate = 64998.50 records/second 1-minute rate = 64921.29 records/second 5-minute rate = 46866.70 records/second 15-minute rate = 36149.86 records/second

Experiment #2

  • Number of records: 80,999,838 (80M)

  • Number of bytes: 26,997,205,878 (~ 26GB)

  • Number of columns: 18

  • Total time: 1294 seconds (21.5 minutes)

Performance Numbers

count = 80,944,061 mean rate = 62465.93 records/second 1-minute rate = 62706.39 records/second 5-minute rate = 60755.41 records/second 15-minute rate = 56673.32 records/second

 

Created in 2020 by Google Inc.