Add deduplication option

[mikeperello-scopely]

Description of change

It may occur for some taps, such as the DynamoDB one (using DynamoDB Streams) that duplicates are generated. To solve this, based on an attribute (Typically a timestamp), a query will run before the loading part to BQ, removing all duplicates.

Manual QA steps

In order to run the deduplication, we need to specify as environment variables, the following attributes:

• deduplication_property: we need to specify the attribute that we want to deduplicate on (Tipically a timestamp).
  In case of having a duplicate, the query is going to keep the one, by default, with the bigger deduplication_property or a random element between those who have the bigger one.
  But we can also modify the order, if for example we want to keep the smaller one.
  • deduplication_order:
    • DESC: Default option. Keeps the one with the bigger deduplication_property or a random element between those who have the bigger one.
    • ASC: Keeps the one with the smaller deduplication_property or a random element between those who have the bigger one.

Additional info

For example, let's say we have the following data coming from the tap:

#_id, name, email, city, date
#1, Mike, NULL, NULL, 2022-07-19 09:00:00
#1, Mike, mike@mike.com, NULL, 2022-07-19 09:10:00
#2, Scopely, scopely@scopely.com, NULL, 2022-07-19 09:05:00
#1, Mike, mike@mike.com, Barcelona, 2022-07-19 09:15:00

In this case, being date as deduplication_property and deduplication_order by default (DESC) we would have:

#_id, name, email, city, date
#1, Mike, mike@mike.com, Barcelona, 2022-07-19 09:15:00
#2, Scopely, scopely@scopely.com, NULL, 2022-07-19 09:05:00

If we set the deduplication_order to ASC, we would have:

#_id, name, email, city, date
#1, Mike, NULL, NULL, 2022-07-19 09:00:00
#2, Scopely, scopely@scopely.com, NULL, 2022-07-19 09:05:00