* Dpfs/change date format (#2) * Add basic changeDateFormatter, which supports DatePatternEnum as source and target pattern * Rulebased prototype script * Add possibility to start without source date, using WIP format rules * Add serializable everywhere because it was needed. Cannot add to ExecutionContext * Fix by using DatePattern.DateEnum everywhere * Add test in scala and java, whatever works first * Added pattern * BLah * Fix scala test and delete java test * Revert "Add serializable everywhere because it was needed. Cannot add to ExecutionContext" This reverts commit 66b4155. * Adds regex but breaks logic somehow * Change method order to be executable as script * Adds comments * Extract unambiguous patterns and use for parsing * Sort by value not key * Remove scala script * Update author and remove prints * Rename to AdaptiveChangeDateFormat * Add description * Move Test to package sub folder * Change testName * Fixes errors due to API changes * Adds Test Case for Applicability + Minor fix * Adds first semi-working score implementation * Convert all rows to strings beforehand. Weird: ID column is not relayed to callApllicability * Add one more field * Extract map from Row, get column, then convert to string * Adds new file with dates for testing * Typo in filename * Only input one column into the applicability * Check if preparator was already applied in metadata * calApplicability: Check 10% of dataset first, before going through every row * Set filename in child classes * Extend PreparatorScalaTest and add explaination to tests * Formatting * Add test again, was executed twice due to pipeline.getRawData.show() * Reworks splitting and rule application * Add null for unimplemented SchemaMapping * Reaname variable * Reorganize imports * Add separators to pattern generation * Add separators to pattern and use month text, if present, for pattern generation * Refactoring * Blah * First similarity check * Add functions to analyze text fields of date clusters * Finishs clustering implementation * Added clustering code to preparator and extracted SimpleDate * WIP: Big refactoring Dates are grouped by similarity criteria, extracted for each date Then finds the pattern of a cluster for at least one date. This is then the pattern of the cluster. Builds a map from SimilarityCriteria -> Pattern When actually formatting: Extract similarity criteria of date. Get the pattern from the Map using it. Convert with pattern. * Pass detected locale through to SimpleDateFormat. Order: Locale.US, user locale, all others * Only use length of number date parts as similarity criteria Strings tend to be not fixed length for the same thing, e.g. different length of month names * Pad single digit dates before building similarity criteria * Filter out text date parts before applying number rules * Fixes calApplicability and minor refactoring * Applicability score is now calculated correctly * Fixes DoW resolution * Fixes test file path * Remove legacy test * Delete playground * Override testFileName * Cleanup, address todos * Remove preamble (#34) * new pom.xml * split file classes * conversion helper method * Define methods * Add TODO * Add cases for different separators * trying to add a test * Add logic to find speparator in file and split * Change return type and logic * Create two DataFrames where to put in the splitted file content * added a test. doesnt work yet * Fix adding something to dataset * two tests that can be executed * basic file split implemented. Tests seem to error but still run through * fixed bug occuring with RandomAccessLists and added a test for files with multiple columns * method for finding the separator. * method for finding the separator - now with confidence scoring * Add method for lower number of columns * Add forgotten return * Refactor method * Remove unused variables and fix test logic * Fix test * Add new method and test for the case of new values * Renaming variables and delete unused code * added write to file, metadata check and bound helper methods into implementation * fixed code of previous group * added method for multiple columns * added testsuite for easier debugging * consectutive empty lines now found as possible preamble * reoccuring chars implemented * changed to double, refactoring * integrated into main method * Add method to check if consecutive lines has the same first character * merge, fixed bug * refactoring of chars in each line method * started working on clustering * Add function to calculate the median * Try to go over each cluster * Try to get scoring of each cluster and use this to calculate median * clustering method now removes lines with unusually low/ high number of separators * word2vec clustering of one column * Extract method and create method for median * extending w2v to mutiple columns * realization, that you might need countvectorizer instead of word2vec * started with types of chars as characterization * types of chars clustering now has high recall, average precision * Filter median cluster to dataset * change to spark 2.4 and clustering now works * Refactor median method and add test case * updated scoring metrix * now all lines with the most often occurring non-alpha-numeric char are removed * unexpected high numbers of separators are now found and removed * basic read file test * refactoring * refactoring - clustering and respective tests * voting for remove based on type * clustering of char types now on fleek * Create datasets for testing * Change dataset and test * Change file path * Add new tast case * New test cases * Add test cases for separator * scoring function fine tuning, fixed bug with separator detection * started on metric for separator skew * fixed a bug in separator skew with double conversion * character type skew is now calculated. added sep skew and char type skew to scoring function * implemented execute Logic and refactored calApplicability * refactoring, improved algorithms, added pipelinetest * tested with all available datasets. advanced algs are only applied if no preamble char was detected * Remove Preamble - Justus Theresia (#20) * new pom.xml * split file classes * conversion helper method * Define methods * Add TODO * Add cases for different separators * trying to add a test * Add logic to find speparator in file and split * Change return type and logic * Create two DataFrames where to put in the splitted file content * added a test. doesnt work yet * Fix adding something to dataset * two tests that can be executed * basic file split implemented. Tests seem to error but still run through * fixed bug occuring with RandomAccessLists and added a test for files with multiple columns * method for finding the separator. * method for finding the separator - now with confidence scoring * Add method for lower number of columns * Add forgotten return * Refactor method * Remove unused variables and fix test logic * Fix test * Add new method and test for the case of new values * Renaming variables and delete unused code * added write to file, metadata check and bound helper methods into implementation * fixed code of previous group * added method for multiple columns * added testsuite for easier debugging * consectutive empty lines now found as possible preamble * reoccuring chars implemented * changed to double, refactoring * integrated into main method * Add method to check if consecutive lines has the same first character * merge, fixed bug * refactoring of chars in each line method * started working on clustering * Add function to calculate the median * Try to go over each cluster * Try to get scoring of each cluster and use this to calculate median * clustering method now removes lines with unusually low/ high number of separators * word2vec clustering of one column * Extract method and create method for median * extending w2v to mutiple columns * realization, that you might need countvectorizer instead of word2vec * started with types of chars as characterization * types of chars clustering now has high recall, average precision * Filter median cluster to dataset * change to spark 2.4 and clustering now works * Refactor median method and add test case * updated scoring metrix * now all lines with the most often occurring non-alpha-numeric char are removed * unexpected high numbers of separators are now found and removed * basic read file test * refactoring * refactoring - clustering and respective tests * voting for remove based on type * clustering of char types now on fleek * Create datasets for testing * Change dataset and test * Change file path * Add new tast case * New test cases * Add test cases for separator * scoring function fine tuning, fixed bug with separator detection * started on metric for separator skew * fixed a bug in separator skew with double conversion * character type skew is now calculated. added sep skew and char type skew to scoring function * implemented execute Logic and refactored calApplicability * refactoring, improved algorithms, added pipelinetest * tested with all available datasets. advanced algs are only applied if no preamble char was detected * find a proper path for the split file preparator output * fixed weird error that occurs if no non-alpha-numeric occur in the dataset * adapted tests for changed project structure * refactored split file tests to be more in line with our current understanding of the framework * resolve conflicts in ConversionHelper * stash: ignore failed test cases * split attribute create nullable new columns * change the action from first to foreach in DefaultChangeFilesEncodingImpl, because first may not do the action for all clusters (since spark 2.4) * disallow printing dataframe to the console

Suggestion: Merge some modules into dataprep-common or dataprep-core #2

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions