At the moment there's a lot of copying and duplicate iteration done, for instance the sanitization iterates through all rows before even starting, along with the newly added trimming of white lines:
source: list[str] = list(itertools.dropwhile(lambda s: s.strip() == "", source))
source: Generator[str, Any, None] = (_preprocess_line(line) for line in source)
This can easily be optimized away by iterating through the lines, though it creates more complex code, it will speed things up.
Two things to do:
[] Create a performance test
[] Implement new flow using single loop
At the moment there's a lot of copying and duplicate iteration done, for instance the sanitization iterates through all rows before even starting, along with the newly added trimming of white lines:
This can easily be optimized away by iterating through the lines, though it creates more complex code, it will speed things up.
Two things to do:
[] Create a performance test
[] Implement new flow using single loop