Report Generation error - pandas.errors.ParserError: Error tokenizing data. C error: Expected 21 fields in line 2099, saw 22 [DEX-401]

[ihsandemir]

When running report generation we see an error like this:

INFO  14:46:08  Loading full_run/A72_dstat.csv
Traceback (most recent call last):
  File "/home/thomas.ball/appdisk/git/hazelcast-simulator/bin/../src/perftest_cli.py", line 63, in <module>
    PerftestCli()
  File "/home/thomas.ball/appdisk/git/hazelcast-simulator/bin/../src/perftest_cli.py", line 37, in __init__
    getattr(self, args.command)()
  File "/home/thomas.ball/appdisk/git/hazelcast-simulator/bin/../src/perftest_cli.py", line 58, in report
    PerfTestReportCli(sys.argv[2:])
  File "/appdisk/thomas.ball/git/hazelcast-simulator/src/simulator/perftest_report.py", line 257, in __init__
    df = analyze(config)
         ^^^^^^^^^^^^^^^
  File "/appdisk/thomas.ball/git/hazelcast-simulator/src/simulator/perftest_report.py", line 33, in analyze
    run_data = analyze_run(config, run_dir, run_label)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/appdisk/thomas.ball/git/hazelcast-simulator/src/simulator/perftest_report.py", line 64, in analyze_run
    analyze_dstat(run_dir, attributes)
  File "/appdisk/thomas.ball/git/hazelcast-simulator/src/simulator/perftest_report_dstat.py", line 22, in analyze_dstat
    df = pd.read_csv(csv_path, skiprows=5)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas.ball/.venvs/simulator/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas.ball/.venvs/simulator/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 626, in _read
    return parser.read(nrows)
           ^^^^^^^^^^^^^^^^^^
  File "/home/thomas.ball/.venvs/simulator/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1923, in read
    ) = self._engine.read(  # type: ignore[attr-defined]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas.ball/.venvs/simulator/lib/python3.11/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 234, in read
    chunks = self._reader.read_low_memory(nrows)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory
  File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows
  File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows
  File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status
  File "parsers.pyx", line 2061, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 21 fields in line 2099, saw 22

When looking at the file full_run/A72_dstat.csv, it really has an extra column:

1745588449,54058840,6093136,56460,4078960,0.096,0.623,1.572,97.413,0,0,131069.597,0,0,0,0,28323,46043,11.310,5.470,9.500,"missed 4 ticks"

dstat add the extra field: missed 4 ticks and this causes parsing error.

I attached an example run log and the csv file for reference.

A72_dstat.csv
report_generation_log.txt

Possible solutions:

1. Configure dstat not to add this extra field in case of such missing ticks.
2. Let pandas ignore this field during parsing.