Skip to content

Fix race conditions in threaded code - #2

Merged
Theelx merged 5 commits into
Theelx:mainfrom
noperator-zz:threading-4
Nov 2, 2022
Merged

Fix race conditions in threaded code#2
Theelx merged 5 commits into
Theelx:mainfrom
noperator-zz:threading-4

Conversation

@noperator-zz

@noperator-zz noperator-zz commented Oct 5, 2022

Copy link
Copy Markdown

Hi and thanks for your work on line-profiler. I realize this is a fork, but it looks like it will be merged in soon (pyutils#165) so I'm taking the chance to apply a fix here instead of on V3.x

Currently, line-profiler is inconsistent when used on threaded code. The lines of some functions do not show up in the output at all.
This stems from two problems:

  • Tracing must be enabled/disabled per-thread. Therefore, enable_count must be a thread-local variable, or different threads will step on each other toes. This was discovered in Enable tracing for all threads. rkern/line_profiler#10, but never merged, so I have re-implemented here
  • Similarly, _c_last_time must be thread-local as it is used by disable. I don't know enough Cython to implement this with threading.local(), so instead I changed _c_last_time to be a mapping between the thread id and the map of LastTime's

I'm not sure how much this will impact performance; I have not measured it. Please let me know if there is a better solution.
Finally, here's a simple test case to show the issue:

from line_profiler import LineProfiler
import time
import threading

prof = LineProfiler()

@prof
def f():
    time.sleep(0.1)  # This line won't show up in the output file

@prof
def f2():
    t = threading.Thread(target=f)
    t.start()
    t.join()  # Or sometimes this one

f2()

@noperator-zz

Copy link
Copy Markdown
Author

I made one more change in a5d0e54 to fix some behavior that was different compared to 3.5.x.
Basically, the for line in for loops ends up with two entries in the get_stats output, whereas on 3.5.x it would only have one entry. This is due to using the hash of the current line when adding an entry for the old line (self._c_code_map[code_hash][key] = ...). An alternative way to fix this is to use the hash of the old line, but I'm not sure about the implications. I have another branch implementing this fix if you want to take a look: noperator-zz@2569cbc#diff-06943e1df2f9908c2330ae5f03985e808db297dbe0440f36c1f720e6474dcb7fR375

@Theelx

Theelx commented Nov 2, 2022

Copy link
Copy Markdown
Owner

I made one more change in a5d0e54 to fix some behavior that was different compared to 3.5.x. Basically, the for line in for loops ends up with two entries in the get_stats output, whereas on 3.5.x it would only have one entry. This is due to using the hash of the current line when adding an entry for the old line (self._c_code_map[code_hash][key] = ...). An alternative way to fix this is to use the hash of the old line, but I'm not sure about the implications. I have another branch implementing this fix if you want to take a look: noperator-zz@2569cbc#diff-06943e1df2f9908c2330ae5f03985e808db297dbe0440f36c1f720e6474dcb7fR375

Thanks for finding this! I wasn't sure whether 3.5.x was wrong or whether my version was wrong when I first encountered this, but now I know my version was wrong.

@Theelx
Theelx merged commit abcf021 into Theelx:main Nov 2, 2022
Theelx pushed a commit that referenced this pull request Aug 6, 2025
Theelx pushed a commit that referenced this pull request Aug 6, 2025
kernprof.py
    _restore
        - Refactored from `_restore_list`
        - New class methods `.sequence()`, `.mapping()`, and
          `.instance_dict()` for restoring various objects
    _remove()
        New utility function for deleting files and directories
    main()
        "Developer mode" options (note: all of the below should be
        considered implementational details)
        - Refactored internal methods for printing messages and
          diagnostics to use `line_profiler._diagnostics.log`
        - Now outputting diagnostics regardless of verbosity level if
          `line_profiler._diagnostics.DEBUG`
        - Now keeping the temporary files if
          `line_profiler._diagnostics.KEEP_TEMPDIRS`
        - Now not executing any source file (setup, pre-imports,
          profiled code) nor writing profile output if
          `line_profiler._diagnostics.NO_EXEC`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants