Add a data generation module

[Zejnilovic]

Background

Together with the testing of data, we could use data generation.

Feature

• Data generation
• random data
• data by schema

[Zejnilovic]

Comment by @GeorgiChochov at Enceladus

---

The generator can be a spark job which accepts a spark schema and generates an appropriate dataset.

The test data should be defined by:

format (xml, csv, parquet, etc.)
data types (string, int... be exhaustive)
number of records
number of columns
levels of nesting (structs and arrays)
specific and random data (we can use the schema's metadata to specify this)
This might involve the creation of a schema generation tool (there might be something out there), which can be done in a separate task if necessary.