Add Thread-Safe Statistics Exporter and Expanded Examples#13
Add Thread-Safe Statistics Exporter and Expanded Examples#13appraneethreddy wants to merge 2 commits into
Conversation
vigsun19
left a comment
vigsun19
left a comment
There was a problem hiding this comment.
Looks good overall! But have some minor issues
- Lock logic creates a lot of .lock files that needs to be cleaned up
- Specify encoding on file open
- Add filelock lib to requirements.txt (filelock~=3.18.0)
- Clean up all the .csv/.json files from this MR , so these are not on the code repo.
| lock_path = f"{file_path}.lock" | ||
| try: | ||
| with FileLock(lock_path, timeout=10): | ||
| if format == 'json': |
There was a problem hiding this comment.
This is creating a lot of unnecessary/empty .lock files. We would need an additional step to cleanup/delete those .lock files
if os.path.exists(lock_path):
os.remove(lock_path)
There was a problem hiding this comment.
Thanks for pointing this out! I've fixed the issue by wrapping the file lock operation in a try...finally block. The .lock file is now reliably removed in the finally block, which guarantees cleanup even if errors occur during the export. This change prevents any leftover lock files from cluttering the directory.
| def _export_to_json(self, file_path: str): | ||
| """Private helper method to export stats to a JSON file.""" | ||
| try: | ||
| with open(file_path, 'w') as f: |
There was a problem hiding this comment.
Add encoding to avoid portability issues across platforms
with open(file_path, 'w', encoding='utf-8') as f:
There was a problem hiding this comment.
Good catch! I've updated the _export_to_json and _export_to_csv methods to open files with encoding='utf-8'. This will ensure the export feature works consistently across different operating systems and avoids any potential portability issues.
There was a problem hiding this comment.
Looks good overall! But have some minor issues
- Lock logic creates a lot of .lock files that needs to be cleaned up
- Specify encoding on file open
- Add filelock lib to requirements.txt (filelock~=3.18.0)
- Clean up all the .csv/.json files from this MR , so these are not on the code repo.
2 participants
This PR introduces a major new feature to smartprofiler: the ability to export profiling statistics to JSON and CSV files. This update also includes expanded examples to showcase the new functionality and important bug fixes for cross-platform compatibility.
Key Features & Enhancements:
New Statistics Exporter:
The
BaseProfiler
now includes a new
export_stats
method, allowing profiling data to be saved in either JSON or CSV format.
Built for Concurrency: The exporter is designed to be thread-safe and process-safe from the ground up. It uses the filelock library to prevent file corruption when used in concurrent applications.
Robust Error Handling: File I/O operations are wrapped in try...except blocks to gracefully handle potential errors.
Expanded Examples:
The
examples_general_usage.py
c:\Users\appra\smartprofiler_clone\examples\examples_general_usage.py
script has been updated to demonstrate the new export functionality for all profilers (CPU, Disk, Function, Memory, and Network).
Bug Fixes:
Resolved a FileNotFoundError on Windows by replacing hardcoded /tmp paths with the cross-platform tempfile.gettempdir().