Normalize relative paths in compile_commands.json

[gnail]

Oftentimes the script location is different to compile_commands.json and relative paths don't make sense once the .lnt file has been generated. We can get around this by normalizing the paths as part of processing.

However, this does break the current unit tests. Specifically, since os.path is platform dependent it only works if the paths match the executing platform, so we can't easily satisfy unit tests for both Windows and Linux path styles. What do you think we should do?

[ssproessig]

@gnail sorry for not getting back to you earlier - been a bit busy...

checking out your branch and running the test locally leads to failing tests now

Testing started at 16:49 ...
C:\_\py36\python.exe C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pydev\pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 51006 --file C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\_jb_nosetest_runner.py --path G:/_customer/xyz/pc_lint_json_db/test_lint4jsondb.py
pydev debugger: process 7424 is connecting

Connected to pydev debugger (build 191.7479.30)
Launching Nosetest with arguments C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\_jb_nosetest_runner.py G:/_customer/xyz/pc_lint_json_db/test_lint4jsondb.py in G:\_customer\xyz\pc_lint_json_db
......

F:/VS2017/working/mocs_compilation.cpp != F:\VS2017\working\mocs_compilation.cpp

Expected :F:\VS2017\working\mocs_compilation.cpp
Actual   :F:/VS2017/working/mocs_compilation.cpp
<Click to see difference>

Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 1194, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: 'F:\\VS2017\\working\\mocs_compilation.cpp' != 'F:/VS2017/working/mocs_compilation.cpp'
- F:\VS2017\working\mocs_compilation.cpp
?   ^      ^       ^
+ F:/VS2017/working/mocs_compilation.cpp
?   ^      ^       ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\_\py36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\_\py36\lib\unittest\case.py", line 601, in run
    testMethod()
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 89, in test_03a_source_trail_style_json_db
    "F:/VS2017/working/mocs_compilation.cpp")



/home/user/code/login/LoginDialog.cpp != \home\user\code\login\LoginDialog.cpp

Expected :\home\user\code\login\LoginDialog.cpp
Actual   :/home/user/code/login/LoginDialog.cpp
<Click to see difference>

Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 1194, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: '\\home\\user\\code\\login\\LoginDialog.cpp' != '/home/user/code/login/LoginDialog.cpp'
- \home\user\code\login\LoginDialog.cpp
? ^    ^    ^    ^     ^
+ /home/user/code/login/LoginDialog.cpp
? ^    ^    ^    ^     ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\_\py36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\_\py36\lib\unittest\case.py", line 601, in run
    testMethod()
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 117, in test_03b_cmake_style_json_db
    "/home/user/code/login/LoginDialog.cpp")

F

G:/qzipreader/main.cpp != G:\qzipreader\main.cpp

Expected :G:\qzipreader\main.cpp
Actual   :G:/qzipreader/main.cpp
<Click to see difference>

Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 1194, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: 'G:\\qzipreader\\main.cpp' != 'G:/qzipreader/main.cpp'
- G:\qzipreader\main.cpp
?   ^          ^
+ G:/qzipreader/main.cpp
?   ^          ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\_\py36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\_\py36\lib\unittest\case.py", line 601, in run
    testMethod()
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 152, in test_03c_qbs_created_for_msvc_json_db
    "G:/qzipreader/main.cpp")


Failure
Traceback (most recent call last):
  File "C:\_\py36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\_\py36\lib\unittest\case.py", line 601, in run
    testMethod()
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 188, in test_03d_cmake_ninja_windows_json_db
    'C:\\Qt\\Qt5.9.2\\5.9.2\\msvc2015_64\\include'
  File "C:\_\py36\lib\unittest\case.py", line 1019, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "C:\_\py36\lib\unittest\case.py", line 1001, in assertSequenceEqual
    self.fail(msg)
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: ['D:\\Code\\build_cu\\QtApplication', 'C:\\[96 chars]ets'] != ['QtApplication', 'C:\\Qt\\Qt5.9.2\\5.9.2\\[76 chars]ets']

First differing element 0:
'D:\\Code\\build_cu\\QtApplication'
'QtApplication'

- ['D:\\Code\\build_cu\\QtApplication',
+ ['QtApplication',
   'C:\\Qt\\Qt5.9.2\\5.9.2\\msvc2015_64\\include',
   'C:\\Qt\\Qt5.9.2\\5.9.2\\msvc2015_64\\include\\QtWidgets']

F.F

2 != 0

Expected :0
Actual   :2
<Click to see difference>

Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 0 != 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\_\py36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\_\py36\lib\unittest\case.py", line 601, in run
    testMethod()
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 213, in test_05a_include_only
    self.assertEqual(len(db.items), 2)



2 != 4

Expected :4
Actual   :2
<Click to see difference>

Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 4 != 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\_\py36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\_\py36\lib\unittest\case.py", line 601, in run
    testMethod()
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 221, in test_05b_exclude_all
    self.assertEqual(len(db.items), 2)

FF

a2.cpp != <dir>\a2.cpp

Expected :<dir>\a2.cpp
Actual   :a2.cpp
<Click to see difference>

Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 1194, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: '<dir>\\a2.cpp' != 'a2.cpp'
- <dir>\a2.cpp
+ a2.cpp


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\_\py36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\_\py36\lib\unittest\case.py", line 601, in run
    testMethod()
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 232, in test_05c_mix_include_and_exclude
    self.assertEqual(db.items[0].file, "a2.cpp")


Assertion failed
FF.
======================================================================
FAIL: test_03a_source_trail_style_json_db (test_lint4jsondb.Lint4JsonCompilationDbUnitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 1194, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: 'F:\\VS2017\\working\\mocs_compilation.cpp' != 'F:/VS2017/working/mocs_compilation.cpp'
- F:\VS2017\working\mocs_compilation.cpp
?   ^      ^       ^
+ F:/VS2017/working/mocs_compilation.cpp
?   ^      ^       ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 89, in test_03a_source_trail_style_json_db
    "F:/VS2017/working/mocs_compilation.cpp")
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 38, in _patched_equals
    raise error
teamcity.diff_tools.EqualsAssertionError:  :: F:\VS2017\working\mocs_compilation.cpp != F:/VS2017/working/mocs_compilation.cpp

======================================================================
FAIL: test_03b_cmake_style_json_db (test_lint4jsondb.Lint4JsonCompilationDbUnitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 1194, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: '\\home\\user\\code\\login\\LoginDialog.cpp' != '/home/user/code/login/LoginDialog.cpp'
- \home\user\code\login\LoginDialog.cpp
? ^    ^    ^    ^     ^
+ /home/user/code/login/LoginDialog.cpp
? ^    ^    ^    ^     ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 117, in test_03b_cmake_style_json_db
    "/home/user/code/login/LoginDialog.cpp")
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 38, in _patched_equals
    raise error
teamcity.diff_tools.EqualsAssertionError:  :: \home\user\code\login\LoginDialog.cpp != /home/user/code/login/LoginDialog.cpp

======================================================================
FAIL: test_03c_qbs_created_for_msvc_json_db (test_lint4jsondb.Lint4JsonCompilationDbUnitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 1194, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: 'G:\\qzipreader\\main.cpp' != 'G:/qzipreader/main.cpp'
- G:\qzipreader\main.cpp
?   ^          ^
+ G:/qzipreader/main.cpp
?   ^          ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 152, in test_03c_qbs_created_for_msvc_json_db
    "G:/qzipreader/main.cpp")
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 38, in _patched_equals
    raise error
teamcity.diff_tools.EqualsAssertionError:  :: G:\qzipreader\main.cpp != G:/qzipreader/main.cpp

======================================================================
FAIL: test_03d_cmake_ninja_windows_json_db (test_lint4jsondb.Lint4JsonCompilationDbUnitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 188, in test_03d_cmake_ninja_windows_json_db
    'C:\\Qt\\Qt5.9.2\\5.9.2\\msvc2015_64\\include'
AssertionError: Lists differ: ['D:\\Code\\build_cu\\QtApplication', 'C:\\[96 chars]ets'] != ['QtApplication', 'C:\\Qt\\Qt5.9.2\\5.9.2\\[76 chars]ets']

First differing element 0:
'D:\\Code\\build_cu\\QtApplication'
'QtApplication'

- ['D:\\Code\\build_cu\\QtApplication',
+ ['QtApplication',
   'C:\\Qt\\Qt5.9.2\\5.9.2\\msvc2015_64\\include',
   'C:\\Qt\\Qt5.9.2\\5.9.2\\msvc2015_64\\include\\QtWidgets']

======================================================================
FAIL: test_05a_include_only (test_lint4jsondb.Lint4JsonCompilationDbUnitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 0 != 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 213, in test_05a_include_only
    self.assertEqual(len(db.items), 2)
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 38, in _patched_equals
    raise error
teamcity.diff_tools.EqualsAssertionError:  :: 0 != 2

======================================================================
FAIL: test_05b_exclude_all (test_lint4jsondb.Lint4JsonCompilationDbUnitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 4 != 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 221, in test_05b_exclude_all
    self.assertEqual(len(db.items), 2)
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 38, in _patched_equals
    raise error
teamcity.diff_tools.EqualsAssertionError:  :: 4 != 2

======================================================================
FAIL: test_05c_mix_include_and_exclude (test_lint4jsondb.Lint4JsonCompilationDbUnitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 32, in _patched_equals
    old(self, first, second, msg)
  File "C:\_\py36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\_\py36\lib\unittest\case.py", line 1194, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\_\py36\lib\unittest\case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: '<dir>\\a2.cpp' != 'a2.cpp'
- <dir>\a2.cpp
+ a2.cpp


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:\_customer\xyz\pc_lint_json_db\test_lint4jsondb.py", line 232, in test_05c_mix_include_and_exclude
    self.assertEqual(db.items[0].file, "a2.cpp")
  File "C:\jetbrains\apps\PyCharm-P\ch-0\191.7479.30\helpers\pycharm\teamcity\diff_tools.py", line 38, in _patched_equals
    raise error
teamcity.diff_tools.EqualsAssertionError:  :: <dir>\a2.cpp != a2.cpp

----------------------------------------------------------------------
Ran 15 tests in 0.092s

FAILED (failures=7)

Process finished with exit code 1
<MagicMock name='Popen().communicate().__getitem__().decode()' id='66165648'>

Assertion failed

Assertion failed

[gnail]

Yea I've talked about this in the original post. IMO the current unit tests are not sustainable as it tests both Windows & POSIX paths but os.path is platform-dependent so at least half the tests will fail. We can explicitly use posixpath and ntpath for the unit tests but it will require a fair amount of change in the unit test. We could also use the new pathlib but that would require Python 3.4+. What do you think?