When the primary and secondary DataFrames get union'd together, the output DataFrame's rows are grouped together by class label.
Before you build any models you want to make sure the rows get randomly shuffled so as to not introduce any artificial bias into the model.
You can address this in the DownsamplingModel by doing an orderBy function after the union operation, like so:
primary.union(secondary).orderBy(rand(42))
When the primary and secondary DataFrames get union'd together, the output DataFrame's rows are grouped together by class label.
Before you build any models you want to make sure the rows get randomly shuffled so as to not introduce any artificial bias into the model.
You can address this in the DownsamplingModel by doing an orderBy function after the union operation, like so:
primary.union(secondary).orderBy(rand(42))