From 055fed97ac9ae57de159a8599bc05b855ea9bc04 Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Fri, 6 Mar 2026 08:39:48 +0530 Subject: [PATCH 1/9] Create backupLogs --- backupLogs | 396 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 396 insertions(+) create mode 100644 backupLogs diff --git a/backupLogs b/backupLogs new file mode 100644 index 000000000..944537b7c --- /dev/null +++ b/backupLogs @@ -0,0 +1,396 @@ +# High-Level Design: backup_logs.sh Migration to C + +## 1. Overview + +### 1.1 Purpose +This document outlines the High-Level Design (HLD) for migrating the existing `backup_logs.sh` shell script to a C implementation. The migration aims to improve performance, reduce memory footprint, and enhance reliability for embedded RDK systems. + +### 1.2 Scope +- Migration of all functionality from `backup_logs.sh` to C code +- Support for both HDD-enabled and HDD-disabled devices +- Maintain compatibility with existing systemd integration +- Preserve log backup and rotation functionality + +### 1.3 Constraints +- Target embedded systems with limited memory (few KBs to few MBs) +- CPU resources are constrained with low clock speeds +- Must be platform-neutral and portable across multiple architectures +- Minimize dynamic memory allocation +- Avoid floating-point arithmetic where possible +- Thread-safe implementation required + +## 2. System Architecture + +### 2.1 Architecture Overview +The C implementation will follow a modular design with the following key components: + +``` +backup_logs (main executable) +├── Configuration Manager +├── Directory Manager +├── Log Backup Engine +├── File Operations Manager +├── Disk Threshold Monitor +├── System Integration Module +└── Error Handler & Logger +``` + +### 2.2 Component Description + +#### 2.2.1 Configuration Manager +- **Purpose**: Load and parse system configuration files +- **Responsibilities**: + - Parse `/etc/include.properties` + - Parse `/etc/device.properties` + - Parse `/etc/env_setup.sh` if available + - Validate configuration parameters + - Provide configuration data to other modules + +#### 2.2.2 Directory Manager +- **Purpose**: Handle directory creation and validation +- **Responsibilities**: + - Create log workspace directories + - Validate directory permissions + - Manage directory path resolution + - Handle directory cleanup operations + +#### 2.2.3 Log Backup Engine +- **Purpose**: Core backup logic implementation +- **Responsibilities**: + - Implement HDD-enabled device backup strategy + - Implement HDD-disabled device backup strategy with rotation + - Handle log file identification and filtering + - Execute backup operations based on device type + +#### 2.2.4 File Operations Manager +- **Purpose**: Low-level file operations +- **Responsibilities**: + - File moving and copying operations + - File existence checking + - Pattern-based file finding + - Timestamp generation and management + +#### 2.2.5 Disk Threshold Monitor +- **Purpose**: Monitor disk usage and trigger cleanup +- **Responsibilities**: + - Check disk usage percentages + - Trigger cleanup scripts when thresholds exceed + - Integration with existing disk_threshold_check.sh + +#### 2.2.6 System Integration Module +- **Purpose**: System-level integrations +- **Responsibilities**: + - Systemd notification handling + - Integration with external scripts + - Process status reporting + +#### 2.2.7 Error Handler & Logger +- **Purpose**: Centralized error handling and logging +- **Responsibilities**: + - Structured error reporting + - Log message formatting with timestamps + - Error code standardization + +## 3. Data Structures + +### 3.1 Core Data Structures + +```c +typedef struct { + char log_path[PATH_MAX]; + char prev_log_path[PATH_MAX]; + char prev_log_backup_path[PATH_MAX]; + char persistent_path[PATH_MAX]; + bool hdd_enabled; +} backup_config_t; + +typedef struct { + char source_path[PATH_MAX]; + char dest_path[PATH_MAX]; + backup_operation_t operation; + char source_extension[32]; + char dest_extension[32]; +} backup_operation_t; + +typedef enum { + BACKUP_OP_MOVE, + BACKUP_OP_COPY, + BACKUP_OP_DELETE +} backup_operation_type_t; + +typedef struct { + int error_code; + char error_message[256]; + const char* function_name; + int line_number; +} error_info_t; +``` + +### 3.2 Memory Management Strategy +- Use fixed-size buffers to avoid dynamic allocation +- Implement memory pools for temporary operations +- Stack-based allocation for small, short-lived data +- Pre-allocated arrays for file lists and paths + +## 4. Module Interfaces + +### 4.1 Configuration Manager Interface +```c +int config_load(backup_config_t* config); +int config_validate(const backup_config_t* config); +const char* config_get_log_path(void); +bool config_is_hdd_enabled(void); +``` + +### 4.2 Directory Manager Interface +```c +int dir_create_workspace(const char* path); +int dir_create_if_not_exists(const char* path); +int dir_cleanup(const char* path, const char* pattern); +bool dir_exists(const char* path); +``` + +### 4.3 Log Backup Engine Interface +```c +int backup_execute_hdd_enabled_strategy(const backup_config_t* config); +int backup_execute_hdd_disabled_strategy(const backup_config_t* config); +int backup_and_recover_logs(const char* source, const char* dest, + backup_operation_type_t op, const char* s_ext, + const char* d_ext); +``` + +### 4.4 File Operations Interface +```c +int file_move(const char* source, const char* dest); +int file_copy(const char* source, const char* dest); +int file_find_pattern(const char* dir, const char* pattern, + char results[][PATH_MAX], int max_results); +bool file_exists(const char* path); +int file_touch(const char* path); +``` + +## 5. Data Flow + +### 5.1 Main Execution Flow +1. **Initialization Phase** + - Load system configuration + - Initialize logging subsystem + - Validate runtime environment + +2. **Preparation Phase** + - Create required directories + - Check disk thresholds + - Determine backup strategy based on HDD status + +3. **Backup Execution Phase** + - Execute appropriate backup strategy + - Handle log rotation (HDD-disabled devices) + - Move/copy log files based on strategy + +4. **Cleanup Phase** + - Clean up old log files + - Copy system version files + - Send systemd notification + +5. **Termination Phase** + - Release resources + - Report final status + +### 5.2 Error Handling Flow +- Centralized error handling through error_info_t structure +- Error propagation through return codes +- Logging of all error conditions with context +- Graceful degradation on non-critical failures + +### 5.3 Visual Flow Representation + +The system workflow is represented through detailed diagrams stored in the [diagrams directory](diagrams/backup_logs_flowcharts.md). This includes both Mermaid-format diagrams and text-based alternatives for environments with limited diagram rendering capabilities. + +#### 5.3.1 Main Backup Process Flow +The primary execution flow showing the complete backup process from initialization to completion, including decision points for HDD-enabled vs HDD-disabled devices. Available in both Mermaid and text formats. + +#### 5.3.2 Component Interaction Sequence +A detailed sequence diagram illustrating the interactions between all system components, showing the order of operations and data flow between modules. Includes timing annotations and error paths. + +#### 5.3.3 HDD Disabled Strategy Detail +A specialized flowchart focusing on the complex log rotation logic for HDD-disabled devices, showing the 4-level backup rotation mechanism with all decision points clearly marked. + +#### 5.3.4 Error Handling and Recovery Flow +A comprehensive error handling flowchart showing how different types of errors are categorized, handled, and recovered from in the system. Includes recovery strategies and escalation paths. + +## 6. Key Algorithms + +### 6.1 HDD-Disabled Backup Algorithm +``` +1. Check for existing backup levels (messages.txt, bak1_messages.txt, etc.) +2. If no existing backups: + - Move all logs to PreviousLogs +3. If backup level 1 exists but not level 2: + - Move current logs to PreviousLogs with bak1_ prefix +4. If backup levels 1-2 exist but not level 3: + - Move current logs to PreviousLogs with bak2_ prefix +5. If all backup levels exist: + - Rotate: bak1->root, bak2->bak1, bak3->bak2, current->bak3 +6. Create last_reboot marker file +``` + +### 6.2 HDD-Enabled Backup Algorithm +``` +1. If no messages.txt in PreviousLogs: + - Move all logs to PreviousLogs + - Create last_reboot marker +2. If messages.txt exists: + - Remove existing last_reboot markers + - Create timestamped backup directory + - Move current logs to timestamped directory + - Create last_reboot marker in timestamped directory +``` + +### 6.3 File Pattern Matching Algorithm +- Use POSIX-compliant pattern matching +- Support for wildcard patterns (*.txt, *.log, etc.) +- Efficient directory traversal with depth control +- Filter by file type (regular files vs. symbolic links) + +## 7. Threading and Concurrency + +### 7.1 Threading Strategy +- Single-threaded design for simplicity and reliability +- Thread-safe utility functions for potential future extensions +- Use of atomic operations for shared state (if any) + +### 7.2 Synchronization +- File locking for critical operations +- Mutex protection for shared resources (if threading is added later) +- Process-level coordination through lockfiles + +## 8. Performance Considerations + +### 8.1 Memory Optimization +- Fixed-size buffers with compile-time sizing +- Stack allocation preference over heap allocation +- Minimal memory fragmentation through planned allocation patterns +- Efficient string handling with bounded operations + +### 8.2 I/O Optimization +- Batch file operations where possible +- Minimize system calls through buffered operations +- Efficient directory traversal algorithms +- Streaming operations for large files + +### 8.3 CPU Optimization +- Avoid expensive operations in loops +- Use bit operations for flags and states +- Minimize string operations and use const strings where possible +- Efficient pattern matching algorithms + +## 9. Integration Points + +### 9.1 System Integration +- **Systemd Integration**: Maintain compatibility with existing service files +- **Configuration Files**: Parse existing shell-format configuration files +- **External Scripts**: Integration with `disk_threshold_check.sh` +- **File System**: Interaction with various mount points and file systems + +### 9.2 Backward Compatibility +- Maintain existing directory structure and naming conventions +- Preserve log file formats and timestamps +- Keep existing environment variable usage +- Maintain compatibility with log analysis tools + +## 10. Error Handling Strategy + +### 10.1 Error Categories +- **Fatal Errors**: Configuration failures, permission issues +- **Recoverable Errors**: Individual file operation failures +- **Warnings**: Non-critical issues that don't prevent execution + +### 10.2 Error Reporting +- Structured error codes for programmatic handling +- Human-readable error messages for debugging +- Integration with existing logging infrastructure +- Syslog integration for system-level error reporting + +## 11. Testing Strategy + +### 11.1 Unit Testing +- Test individual modules in isolation +- Mock external dependencies (file system, system calls) +- Comprehensive error condition testing +- Memory leak detection and prevention + +### 11.2 Integration Testing +- Test complete backup scenarios +- Verify compatibility with existing system +- Performance benchmarking against shell script +- Multi-platform validation + +### 11.3 System Testing +- End-to-end functionality verification +- Stress testing with large log volumes +- Resource constraint testing +- Recovery testing after various failure scenarios + +## 12. Deployment Considerations + +### 12.1 Build System +- Integration with existing autotools configuration +- Cross-compilation support for multiple architectures +- Compiler optimization flags for embedded targets +- Static linking considerations for deployment + +### 12.2 Installation +- Backward-compatible installation process +- Service file updates for systemd integration +- Configuration migration support +- Rollback capability + +### 12.3 Monitoring +- Health check mechanisms +- Performance metrics collection +- Resource usage monitoring +- Integration with existing monitoring infrastructure + +## 13. Future Enhancements + +### 13.1 Planned Features +- Configuration hot-reloading capability +- Enhanced compression for archived logs +- Remote log backup capability +- Advanced filtering and retention policies + +### 13.2 Extensibility +- Plugin architecture for custom backup strategies +- Configurable backup policies +- API for external tools integration +- Event-driven architecture support + +## 14. Risk Analysis + +### 14.1 Technical Risks +- **Memory Management**: Risk of memory leaks in embedded environment +- **File System Operations**: Race conditions with concurrent access +- **Configuration Parsing**: Compatibility issues with shell variable expansion +- **Performance**: Potential performance regression compared to shell script + +### 14.2 Mitigation Strategies +- Comprehensive testing with memory analysis tools +- File locking and atomic operations for critical sections +- Robust configuration parsing with validation +- Performance benchmarking and optimization + +## 15. Success Criteria + +### 15.1 Functional Requirements +- ✅ Complete feature parity with existing shell script +- ✅ Support for both HDD-enabled and HDD-disabled devices +- ✅ Proper log rotation and backup functionality +- ✅ Integration with systemd and existing infrastructure + +### 15.2 Non-Functional Requirements +- ✅ Memory usage reduction of at least 20% compared to shell process +- ✅ Startup time improvement of at least 30% +- ✅ CPU usage reduction during backup operations +- ✅ Cross-platform compatibility across target embedded systems + +This HLD provides a comprehensive roadmap for migrating the backup_logs.sh script to a robust, efficient C implementation suitable for embedded RDK environments. From 05421cd9607165709a23dec063b3da340d41196f Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:41:27 +0530 Subject: [PATCH 2/9] Delete backupLogs --- backupLogs | 396 ----------------------------------------------------- 1 file changed, 396 deletions(-) delete mode 100644 backupLogs diff --git a/backupLogs b/backupLogs deleted file mode 100644 index 944537b7c..000000000 --- a/backupLogs +++ /dev/null @@ -1,396 +0,0 @@ -# High-Level Design: backup_logs.sh Migration to C - -## 1. Overview - -### 1.1 Purpose -This document outlines the High-Level Design (HLD) for migrating the existing `backup_logs.sh` shell script to a C implementation. The migration aims to improve performance, reduce memory footprint, and enhance reliability for embedded RDK systems. - -### 1.2 Scope -- Migration of all functionality from `backup_logs.sh` to C code -- Support for both HDD-enabled and HDD-disabled devices -- Maintain compatibility with existing systemd integration -- Preserve log backup and rotation functionality - -### 1.3 Constraints -- Target embedded systems with limited memory (few KBs to few MBs) -- CPU resources are constrained with low clock speeds -- Must be platform-neutral and portable across multiple architectures -- Minimize dynamic memory allocation -- Avoid floating-point arithmetic where possible -- Thread-safe implementation required - -## 2. System Architecture - -### 2.1 Architecture Overview -The C implementation will follow a modular design with the following key components: - -``` -backup_logs (main executable) -├── Configuration Manager -├── Directory Manager -├── Log Backup Engine -├── File Operations Manager -├── Disk Threshold Monitor -├── System Integration Module -└── Error Handler & Logger -``` - -### 2.2 Component Description - -#### 2.2.1 Configuration Manager -- **Purpose**: Load and parse system configuration files -- **Responsibilities**: - - Parse `/etc/include.properties` - - Parse `/etc/device.properties` - - Parse `/etc/env_setup.sh` if available - - Validate configuration parameters - - Provide configuration data to other modules - -#### 2.2.2 Directory Manager -- **Purpose**: Handle directory creation and validation -- **Responsibilities**: - - Create log workspace directories - - Validate directory permissions - - Manage directory path resolution - - Handle directory cleanup operations - -#### 2.2.3 Log Backup Engine -- **Purpose**: Core backup logic implementation -- **Responsibilities**: - - Implement HDD-enabled device backup strategy - - Implement HDD-disabled device backup strategy with rotation - - Handle log file identification and filtering - - Execute backup operations based on device type - -#### 2.2.4 File Operations Manager -- **Purpose**: Low-level file operations -- **Responsibilities**: - - File moving and copying operations - - File existence checking - - Pattern-based file finding - - Timestamp generation and management - -#### 2.2.5 Disk Threshold Monitor -- **Purpose**: Monitor disk usage and trigger cleanup -- **Responsibilities**: - - Check disk usage percentages - - Trigger cleanup scripts when thresholds exceed - - Integration with existing disk_threshold_check.sh - -#### 2.2.6 System Integration Module -- **Purpose**: System-level integrations -- **Responsibilities**: - - Systemd notification handling - - Integration with external scripts - - Process status reporting - -#### 2.2.7 Error Handler & Logger -- **Purpose**: Centralized error handling and logging -- **Responsibilities**: - - Structured error reporting - - Log message formatting with timestamps - - Error code standardization - -## 3. Data Structures - -### 3.1 Core Data Structures - -```c -typedef struct { - char log_path[PATH_MAX]; - char prev_log_path[PATH_MAX]; - char prev_log_backup_path[PATH_MAX]; - char persistent_path[PATH_MAX]; - bool hdd_enabled; -} backup_config_t; - -typedef struct { - char source_path[PATH_MAX]; - char dest_path[PATH_MAX]; - backup_operation_t operation; - char source_extension[32]; - char dest_extension[32]; -} backup_operation_t; - -typedef enum { - BACKUP_OP_MOVE, - BACKUP_OP_COPY, - BACKUP_OP_DELETE -} backup_operation_type_t; - -typedef struct { - int error_code; - char error_message[256]; - const char* function_name; - int line_number; -} error_info_t; -``` - -### 3.2 Memory Management Strategy -- Use fixed-size buffers to avoid dynamic allocation -- Implement memory pools for temporary operations -- Stack-based allocation for small, short-lived data -- Pre-allocated arrays for file lists and paths - -## 4. Module Interfaces - -### 4.1 Configuration Manager Interface -```c -int config_load(backup_config_t* config); -int config_validate(const backup_config_t* config); -const char* config_get_log_path(void); -bool config_is_hdd_enabled(void); -``` - -### 4.2 Directory Manager Interface -```c -int dir_create_workspace(const char* path); -int dir_create_if_not_exists(const char* path); -int dir_cleanup(const char* path, const char* pattern); -bool dir_exists(const char* path); -``` - -### 4.3 Log Backup Engine Interface -```c -int backup_execute_hdd_enabled_strategy(const backup_config_t* config); -int backup_execute_hdd_disabled_strategy(const backup_config_t* config); -int backup_and_recover_logs(const char* source, const char* dest, - backup_operation_type_t op, const char* s_ext, - const char* d_ext); -``` - -### 4.4 File Operations Interface -```c -int file_move(const char* source, const char* dest); -int file_copy(const char* source, const char* dest); -int file_find_pattern(const char* dir, const char* pattern, - char results[][PATH_MAX], int max_results); -bool file_exists(const char* path); -int file_touch(const char* path); -``` - -## 5. Data Flow - -### 5.1 Main Execution Flow -1. **Initialization Phase** - - Load system configuration - - Initialize logging subsystem - - Validate runtime environment - -2. **Preparation Phase** - - Create required directories - - Check disk thresholds - - Determine backup strategy based on HDD status - -3. **Backup Execution Phase** - - Execute appropriate backup strategy - - Handle log rotation (HDD-disabled devices) - - Move/copy log files based on strategy - -4. **Cleanup Phase** - - Clean up old log files - - Copy system version files - - Send systemd notification - -5. **Termination Phase** - - Release resources - - Report final status - -### 5.2 Error Handling Flow -- Centralized error handling through error_info_t structure -- Error propagation through return codes -- Logging of all error conditions with context -- Graceful degradation on non-critical failures - -### 5.3 Visual Flow Representation - -The system workflow is represented through detailed diagrams stored in the [diagrams directory](diagrams/backup_logs_flowcharts.md). This includes both Mermaid-format diagrams and text-based alternatives for environments with limited diagram rendering capabilities. - -#### 5.3.1 Main Backup Process Flow -The primary execution flow showing the complete backup process from initialization to completion, including decision points for HDD-enabled vs HDD-disabled devices. Available in both Mermaid and text formats. - -#### 5.3.2 Component Interaction Sequence -A detailed sequence diagram illustrating the interactions between all system components, showing the order of operations and data flow between modules. Includes timing annotations and error paths. - -#### 5.3.3 HDD Disabled Strategy Detail -A specialized flowchart focusing on the complex log rotation logic for HDD-disabled devices, showing the 4-level backup rotation mechanism with all decision points clearly marked. - -#### 5.3.4 Error Handling and Recovery Flow -A comprehensive error handling flowchart showing how different types of errors are categorized, handled, and recovered from in the system. Includes recovery strategies and escalation paths. - -## 6. Key Algorithms - -### 6.1 HDD-Disabled Backup Algorithm -``` -1. Check for existing backup levels (messages.txt, bak1_messages.txt, etc.) -2. If no existing backups: - - Move all logs to PreviousLogs -3. If backup level 1 exists but not level 2: - - Move current logs to PreviousLogs with bak1_ prefix -4. If backup levels 1-2 exist but not level 3: - - Move current logs to PreviousLogs with bak2_ prefix -5. If all backup levels exist: - - Rotate: bak1->root, bak2->bak1, bak3->bak2, current->bak3 -6. Create last_reboot marker file -``` - -### 6.2 HDD-Enabled Backup Algorithm -``` -1. If no messages.txt in PreviousLogs: - - Move all logs to PreviousLogs - - Create last_reboot marker -2. If messages.txt exists: - - Remove existing last_reboot markers - - Create timestamped backup directory - - Move current logs to timestamped directory - - Create last_reboot marker in timestamped directory -``` - -### 6.3 File Pattern Matching Algorithm -- Use POSIX-compliant pattern matching -- Support for wildcard patterns (*.txt, *.log, etc.) -- Efficient directory traversal with depth control -- Filter by file type (regular files vs. symbolic links) - -## 7. Threading and Concurrency - -### 7.1 Threading Strategy -- Single-threaded design for simplicity and reliability -- Thread-safe utility functions for potential future extensions -- Use of atomic operations for shared state (if any) - -### 7.2 Synchronization -- File locking for critical operations -- Mutex protection for shared resources (if threading is added later) -- Process-level coordination through lockfiles - -## 8. Performance Considerations - -### 8.1 Memory Optimization -- Fixed-size buffers with compile-time sizing -- Stack allocation preference over heap allocation -- Minimal memory fragmentation through planned allocation patterns -- Efficient string handling with bounded operations - -### 8.2 I/O Optimization -- Batch file operations where possible -- Minimize system calls through buffered operations -- Efficient directory traversal algorithms -- Streaming operations for large files - -### 8.3 CPU Optimization -- Avoid expensive operations in loops -- Use bit operations for flags and states -- Minimize string operations and use const strings where possible -- Efficient pattern matching algorithms - -## 9. Integration Points - -### 9.1 System Integration -- **Systemd Integration**: Maintain compatibility with existing service files -- **Configuration Files**: Parse existing shell-format configuration files -- **External Scripts**: Integration with `disk_threshold_check.sh` -- **File System**: Interaction with various mount points and file systems - -### 9.2 Backward Compatibility -- Maintain existing directory structure and naming conventions -- Preserve log file formats and timestamps -- Keep existing environment variable usage -- Maintain compatibility with log analysis tools - -## 10. Error Handling Strategy - -### 10.1 Error Categories -- **Fatal Errors**: Configuration failures, permission issues -- **Recoverable Errors**: Individual file operation failures -- **Warnings**: Non-critical issues that don't prevent execution - -### 10.2 Error Reporting -- Structured error codes for programmatic handling -- Human-readable error messages for debugging -- Integration with existing logging infrastructure -- Syslog integration for system-level error reporting - -## 11. Testing Strategy - -### 11.1 Unit Testing -- Test individual modules in isolation -- Mock external dependencies (file system, system calls) -- Comprehensive error condition testing -- Memory leak detection and prevention - -### 11.2 Integration Testing -- Test complete backup scenarios -- Verify compatibility with existing system -- Performance benchmarking against shell script -- Multi-platform validation - -### 11.3 System Testing -- End-to-end functionality verification -- Stress testing with large log volumes -- Resource constraint testing -- Recovery testing after various failure scenarios - -## 12. Deployment Considerations - -### 12.1 Build System -- Integration with existing autotools configuration -- Cross-compilation support for multiple architectures -- Compiler optimization flags for embedded targets -- Static linking considerations for deployment - -### 12.2 Installation -- Backward-compatible installation process -- Service file updates for systemd integration -- Configuration migration support -- Rollback capability - -### 12.3 Monitoring -- Health check mechanisms -- Performance metrics collection -- Resource usage monitoring -- Integration with existing monitoring infrastructure - -## 13. Future Enhancements - -### 13.1 Planned Features -- Configuration hot-reloading capability -- Enhanced compression for archived logs -- Remote log backup capability -- Advanced filtering and retention policies - -### 13.2 Extensibility -- Plugin architecture for custom backup strategies -- Configurable backup policies -- API for external tools integration -- Event-driven architecture support - -## 14. Risk Analysis - -### 14.1 Technical Risks -- **Memory Management**: Risk of memory leaks in embedded environment -- **File System Operations**: Race conditions with concurrent access -- **Configuration Parsing**: Compatibility issues with shell variable expansion -- **Performance**: Potential performance regression compared to shell script - -### 14.2 Mitigation Strategies -- Comprehensive testing with memory analysis tools -- File locking and atomic operations for critical sections -- Robust configuration parsing with validation -- Performance benchmarking and optimization - -## 15. Success Criteria - -### 15.1 Functional Requirements -- ✅ Complete feature parity with existing shell script -- ✅ Support for both HDD-enabled and HDD-disabled devices -- ✅ Proper log rotation and backup functionality -- ✅ Integration with systemd and existing infrastructure - -### 15.2 Non-Functional Requirements -- ✅ Memory usage reduction of at least 20% compared to shell process -- ✅ Startup time improvement of at least 30% -- ✅ CPU usage reduction during backup operations -- ✅ Cross-platform compatibility across target embedded systems - -This HLD provides a comprehensive roadmap for migrating the backup_logs.sh script to a robust, efficient C implementation suitable for embedded RDK environments. From 8bef6a9f64722bd9fa84271c59a9dd1e185b7998 Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:42:16 +0530 Subject: [PATCH 3/9] Create backup_logs_migration_HLD.md --- backup_logs_migration_HLD.md | 396 +++++++++++++++++++++++++++++++++++ 1 file changed, 396 insertions(+) create mode 100644 backup_logs_migration_HLD.md diff --git a/backup_logs_migration_HLD.md b/backup_logs_migration_HLD.md new file mode 100644 index 000000000..944537b7c --- /dev/null +++ b/backup_logs_migration_HLD.md @@ -0,0 +1,396 @@ +# High-Level Design: backup_logs.sh Migration to C + +## 1. Overview + +### 1.1 Purpose +This document outlines the High-Level Design (HLD) for migrating the existing `backup_logs.sh` shell script to a C implementation. The migration aims to improve performance, reduce memory footprint, and enhance reliability for embedded RDK systems. + +### 1.2 Scope +- Migration of all functionality from `backup_logs.sh` to C code +- Support for both HDD-enabled and HDD-disabled devices +- Maintain compatibility with existing systemd integration +- Preserve log backup and rotation functionality + +### 1.3 Constraints +- Target embedded systems with limited memory (few KBs to few MBs) +- CPU resources are constrained with low clock speeds +- Must be platform-neutral and portable across multiple architectures +- Minimize dynamic memory allocation +- Avoid floating-point arithmetic where possible +- Thread-safe implementation required + +## 2. System Architecture + +### 2.1 Architecture Overview +The C implementation will follow a modular design with the following key components: + +``` +backup_logs (main executable) +├── Configuration Manager +├── Directory Manager +├── Log Backup Engine +├── File Operations Manager +├── Disk Threshold Monitor +├── System Integration Module +└── Error Handler & Logger +``` + +### 2.2 Component Description + +#### 2.2.1 Configuration Manager +- **Purpose**: Load and parse system configuration files +- **Responsibilities**: + - Parse `/etc/include.properties` + - Parse `/etc/device.properties` + - Parse `/etc/env_setup.sh` if available + - Validate configuration parameters + - Provide configuration data to other modules + +#### 2.2.2 Directory Manager +- **Purpose**: Handle directory creation and validation +- **Responsibilities**: + - Create log workspace directories + - Validate directory permissions + - Manage directory path resolution + - Handle directory cleanup operations + +#### 2.2.3 Log Backup Engine +- **Purpose**: Core backup logic implementation +- **Responsibilities**: + - Implement HDD-enabled device backup strategy + - Implement HDD-disabled device backup strategy with rotation + - Handle log file identification and filtering + - Execute backup operations based on device type + +#### 2.2.4 File Operations Manager +- **Purpose**: Low-level file operations +- **Responsibilities**: + - File moving and copying operations + - File existence checking + - Pattern-based file finding + - Timestamp generation and management + +#### 2.2.5 Disk Threshold Monitor +- **Purpose**: Monitor disk usage and trigger cleanup +- **Responsibilities**: + - Check disk usage percentages + - Trigger cleanup scripts when thresholds exceed + - Integration with existing disk_threshold_check.sh + +#### 2.2.6 System Integration Module +- **Purpose**: System-level integrations +- **Responsibilities**: + - Systemd notification handling + - Integration with external scripts + - Process status reporting + +#### 2.2.7 Error Handler & Logger +- **Purpose**: Centralized error handling and logging +- **Responsibilities**: + - Structured error reporting + - Log message formatting with timestamps + - Error code standardization + +## 3. Data Structures + +### 3.1 Core Data Structures + +```c +typedef struct { + char log_path[PATH_MAX]; + char prev_log_path[PATH_MAX]; + char prev_log_backup_path[PATH_MAX]; + char persistent_path[PATH_MAX]; + bool hdd_enabled; +} backup_config_t; + +typedef struct { + char source_path[PATH_MAX]; + char dest_path[PATH_MAX]; + backup_operation_t operation; + char source_extension[32]; + char dest_extension[32]; +} backup_operation_t; + +typedef enum { + BACKUP_OP_MOVE, + BACKUP_OP_COPY, + BACKUP_OP_DELETE +} backup_operation_type_t; + +typedef struct { + int error_code; + char error_message[256]; + const char* function_name; + int line_number; +} error_info_t; +``` + +### 3.2 Memory Management Strategy +- Use fixed-size buffers to avoid dynamic allocation +- Implement memory pools for temporary operations +- Stack-based allocation for small, short-lived data +- Pre-allocated arrays for file lists and paths + +## 4. Module Interfaces + +### 4.1 Configuration Manager Interface +```c +int config_load(backup_config_t* config); +int config_validate(const backup_config_t* config); +const char* config_get_log_path(void); +bool config_is_hdd_enabled(void); +``` + +### 4.2 Directory Manager Interface +```c +int dir_create_workspace(const char* path); +int dir_create_if_not_exists(const char* path); +int dir_cleanup(const char* path, const char* pattern); +bool dir_exists(const char* path); +``` + +### 4.3 Log Backup Engine Interface +```c +int backup_execute_hdd_enabled_strategy(const backup_config_t* config); +int backup_execute_hdd_disabled_strategy(const backup_config_t* config); +int backup_and_recover_logs(const char* source, const char* dest, + backup_operation_type_t op, const char* s_ext, + const char* d_ext); +``` + +### 4.4 File Operations Interface +```c +int file_move(const char* source, const char* dest); +int file_copy(const char* source, const char* dest); +int file_find_pattern(const char* dir, const char* pattern, + char results[][PATH_MAX], int max_results); +bool file_exists(const char* path); +int file_touch(const char* path); +``` + +## 5. Data Flow + +### 5.1 Main Execution Flow +1. **Initialization Phase** + - Load system configuration + - Initialize logging subsystem + - Validate runtime environment + +2. **Preparation Phase** + - Create required directories + - Check disk thresholds + - Determine backup strategy based on HDD status + +3. **Backup Execution Phase** + - Execute appropriate backup strategy + - Handle log rotation (HDD-disabled devices) + - Move/copy log files based on strategy + +4. **Cleanup Phase** + - Clean up old log files + - Copy system version files + - Send systemd notification + +5. **Termination Phase** + - Release resources + - Report final status + +### 5.2 Error Handling Flow +- Centralized error handling through error_info_t structure +- Error propagation through return codes +- Logging of all error conditions with context +- Graceful degradation on non-critical failures + +### 5.3 Visual Flow Representation + +The system workflow is represented through detailed diagrams stored in the [diagrams directory](diagrams/backup_logs_flowcharts.md). This includes both Mermaid-format diagrams and text-based alternatives for environments with limited diagram rendering capabilities. + +#### 5.3.1 Main Backup Process Flow +The primary execution flow showing the complete backup process from initialization to completion, including decision points for HDD-enabled vs HDD-disabled devices. Available in both Mermaid and text formats. + +#### 5.3.2 Component Interaction Sequence +A detailed sequence diagram illustrating the interactions between all system components, showing the order of operations and data flow between modules. Includes timing annotations and error paths. + +#### 5.3.3 HDD Disabled Strategy Detail +A specialized flowchart focusing on the complex log rotation logic for HDD-disabled devices, showing the 4-level backup rotation mechanism with all decision points clearly marked. + +#### 5.3.4 Error Handling and Recovery Flow +A comprehensive error handling flowchart showing how different types of errors are categorized, handled, and recovered from in the system. Includes recovery strategies and escalation paths. + +## 6. Key Algorithms + +### 6.1 HDD-Disabled Backup Algorithm +``` +1. Check for existing backup levels (messages.txt, bak1_messages.txt, etc.) +2. If no existing backups: + - Move all logs to PreviousLogs +3. If backup level 1 exists but not level 2: + - Move current logs to PreviousLogs with bak1_ prefix +4. If backup levels 1-2 exist but not level 3: + - Move current logs to PreviousLogs with bak2_ prefix +5. If all backup levels exist: + - Rotate: bak1->root, bak2->bak1, bak3->bak2, current->bak3 +6. Create last_reboot marker file +``` + +### 6.2 HDD-Enabled Backup Algorithm +``` +1. If no messages.txt in PreviousLogs: + - Move all logs to PreviousLogs + - Create last_reboot marker +2. If messages.txt exists: + - Remove existing last_reboot markers + - Create timestamped backup directory + - Move current logs to timestamped directory + - Create last_reboot marker in timestamped directory +``` + +### 6.3 File Pattern Matching Algorithm +- Use POSIX-compliant pattern matching +- Support for wildcard patterns (*.txt, *.log, etc.) +- Efficient directory traversal with depth control +- Filter by file type (regular files vs. symbolic links) + +## 7. Threading and Concurrency + +### 7.1 Threading Strategy +- Single-threaded design for simplicity and reliability +- Thread-safe utility functions for potential future extensions +- Use of atomic operations for shared state (if any) + +### 7.2 Synchronization +- File locking for critical operations +- Mutex protection for shared resources (if threading is added later) +- Process-level coordination through lockfiles + +## 8. Performance Considerations + +### 8.1 Memory Optimization +- Fixed-size buffers with compile-time sizing +- Stack allocation preference over heap allocation +- Minimal memory fragmentation through planned allocation patterns +- Efficient string handling with bounded operations + +### 8.2 I/O Optimization +- Batch file operations where possible +- Minimize system calls through buffered operations +- Efficient directory traversal algorithms +- Streaming operations for large files + +### 8.3 CPU Optimization +- Avoid expensive operations in loops +- Use bit operations for flags and states +- Minimize string operations and use const strings where possible +- Efficient pattern matching algorithms + +## 9. Integration Points + +### 9.1 System Integration +- **Systemd Integration**: Maintain compatibility with existing service files +- **Configuration Files**: Parse existing shell-format configuration files +- **External Scripts**: Integration with `disk_threshold_check.sh` +- **File System**: Interaction with various mount points and file systems + +### 9.2 Backward Compatibility +- Maintain existing directory structure and naming conventions +- Preserve log file formats and timestamps +- Keep existing environment variable usage +- Maintain compatibility with log analysis tools + +## 10. Error Handling Strategy + +### 10.1 Error Categories +- **Fatal Errors**: Configuration failures, permission issues +- **Recoverable Errors**: Individual file operation failures +- **Warnings**: Non-critical issues that don't prevent execution + +### 10.2 Error Reporting +- Structured error codes for programmatic handling +- Human-readable error messages for debugging +- Integration with existing logging infrastructure +- Syslog integration for system-level error reporting + +## 11. Testing Strategy + +### 11.1 Unit Testing +- Test individual modules in isolation +- Mock external dependencies (file system, system calls) +- Comprehensive error condition testing +- Memory leak detection and prevention + +### 11.2 Integration Testing +- Test complete backup scenarios +- Verify compatibility with existing system +- Performance benchmarking against shell script +- Multi-platform validation + +### 11.3 System Testing +- End-to-end functionality verification +- Stress testing with large log volumes +- Resource constraint testing +- Recovery testing after various failure scenarios + +## 12. Deployment Considerations + +### 12.1 Build System +- Integration with existing autotools configuration +- Cross-compilation support for multiple architectures +- Compiler optimization flags for embedded targets +- Static linking considerations for deployment + +### 12.2 Installation +- Backward-compatible installation process +- Service file updates for systemd integration +- Configuration migration support +- Rollback capability + +### 12.3 Monitoring +- Health check mechanisms +- Performance metrics collection +- Resource usage monitoring +- Integration with existing monitoring infrastructure + +## 13. Future Enhancements + +### 13.1 Planned Features +- Configuration hot-reloading capability +- Enhanced compression for archived logs +- Remote log backup capability +- Advanced filtering and retention policies + +### 13.2 Extensibility +- Plugin architecture for custom backup strategies +- Configurable backup policies +- API for external tools integration +- Event-driven architecture support + +## 14. Risk Analysis + +### 14.1 Technical Risks +- **Memory Management**: Risk of memory leaks in embedded environment +- **File System Operations**: Race conditions with concurrent access +- **Configuration Parsing**: Compatibility issues with shell variable expansion +- **Performance**: Potential performance regression compared to shell script + +### 14.2 Mitigation Strategies +- Comprehensive testing with memory analysis tools +- File locking and atomic operations for critical sections +- Robust configuration parsing with validation +- Performance benchmarking and optimization + +## 15. Success Criteria + +### 15.1 Functional Requirements +- ✅ Complete feature parity with existing shell script +- ✅ Support for both HDD-enabled and HDD-disabled devices +- ✅ Proper log rotation and backup functionality +- ✅ Integration with systemd and existing infrastructure + +### 15.2 Non-Functional Requirements +- ✅ Memory usage reduction of at least 20% compared to shell process +- ✅ Startup time improvement of at least 30% +- ✅ CPU usage reduction during backup operations +- ✅ Cross-platform compatibility across target embedded systems + +This HLD provides a comprehensive roadmap for migrating the backup_logs.sh script to a robust, efficient C implementation suitable for embedded RDK environments. From 9fb20bc8ecfbdfd226ac441bcf09aa6a4e7d4a08 Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:44:40 +0530 Subject: [PATCH 4/9] Delete backup_logs_migration_HLD.md --- backup_logs_migration_HLD.md | 396 ----------------------------------- 1 file changed, 396 deletions(-) delete mode 100644 backup_logs_migration_HLD.md diff --git a/backup_logs_migration_HLD.md b/backup_logs_migration_HLD.md deleted file mode 100644 index 944537b7c..000000000 --- a/backup_logs_migration_HLD.md +++ /dev/null @@ -1,396 +0,0 @@ -# High-Level Design: backup_logs.sh Migration to C - -## 1. Overview - -### 1.1 Purpose -This document outlines the High-Level Design (HLD) for migrating the existing `backup_logs.sh` shell script to a C implementation. The migration aims to improve performance, reduce memory footprint, and enhance reliability for embedded RDK systems. - -### 1.2 Scope -- Migration of all functionality from `backup_logs.sh` to C code -- Support for both HDD-enabled and HDD-disabled devices -- Maintain compatibility with existing systemd integration -- Preserve log backup and rotation functionality - -### 1.3 Constraints -- Target embedded systems with limited memory (few KBs to few MBs) -- CPU resources are constrained with low clock speeds -- Must be platform-neutral and portable across multiple architectures -- Minimize dynamic memory allocation -- Avoid floating-point arithmetic where possible -- Thread-safe implementation required - -## 2. System Architecture - -### 2.1 Architecture Overview -The C implementation will follow a modular design with the following key components: - -``` -backup_logs (main executable) -├── Configuration Manager -├── Directory Manager -├── Log Backup Engine -├── File Operations Manager -├── Disk Threshold Monitor -├── System Integration Module -└── Error Handler & Logger -``` - -### 2.2 Component Description - -#### 2.2.1 Configuration Manager -- **Purpose**: Load and parse system configuration files -- **Responsibilities**: - - Parse `/etc/include.properties` - - Parse `/etc/device.properties` - - Parse `/etc/env_setup.sh` if available - - Validate configuration parameters - - Provide configuration data to other modules - -#### 2.2.2 Directory Manager -- **Purpose**: Handle directory creation and validation -- **Responsibilities**: - - Create log workspace directories - - Validate directory permissions - - Manage directory path resolution - - Handle directory cleanup operations - -#### 2.2.3 Log Backup Engine -- **Purpose**: Core backup logic implementation -- **Responsibilities**: - - Implement HDD-enabled device backup strategy - - Implement HDD-disabled device backup strategy with rotation - - Handle log file identification and filtering - - Execute backup operations based on device type - -#### 2.2.4 File Operations Manager -- **Purpose**: Low-level file operations -- **Responsibilities**: - - File moving and copying operations - - File existence checking - - Pattern-based file finding - - Timestamp generation and management - -#### 2.2.5 Disk Threshold Monitor -- **Purpose**: Monitor disk usage and trigger cleanup -- **Responsibilities**: - - Check disk usage percentages - - Trigger cleanup scripts when thresholds exceed - - Integration with existing disk_threshold_check.sh - -#### 2.2.6 System Integration Module -- **Purpose**: System-level integrations -- **Responsibilities**: - - Systemd notification handling - - Integration with external scripts - - Process status reporting - -#### 2.2.7 Error Handler & Logger -- **Purpose**: Centralized error handling and logging -- **Responsibilities**: - - Structured error reporting - - Log message formatting with timestamps - - Error code standardization - -## 3. Data Structures - -### 3.1 Core Data Structures - -```c -typedef struct { - char log_path[PATH_MAX]; - char prev_log_path[PATH_MAX]; - char prev_log_backup_path[PATH_MAX]; - char persistent_path[PATH_MAX]; - bool hdd_enabled; -} backup_config_t; - -typedef struct { - char source_path[PATH_MAX]; - char dest_path[PATH_MAX]; - backup_operation_t operation; - char source_extension[32]; - char dest_extension[32]; -} backup_operation_t; - -typedef enum { - BACKUP_OP_MOVE, - BACKUP_OP_COPY, - BACKUP_OP_DELETE -} backup_operation_type_t; - -typedef struct { - int error_code; - char error_message[256]; - const char* function_name; - int line_number; -} error_info_t; -``` - -### 3.2 Memory Management Strategy -- Use fixed-size buffers to avoid dynamic allocation -- Implement memory pools for temporary operations -- Stack-based allocation for small, short-lived data -- Pre-allocated arrays for file lists and paths - -## 4. Module Interfaces - -### 4.1 Configuration Manager Interface -```c -int config_load(backup_config_t* config); -int config_validate(const backup_config_t* config); -const char* config_get_log_path(void); -bool config_is_hdd_enabled(void); -``` - -### 4.2 Directory Manager Interface -```c -int dir_create_workspace(const char* path); -int dir_create_if_not_exists(const char* path); -int dir_cleanup(const char* path, const char* pattern); -bool dir_exists(const char* path); -``` - -### 4.3 Log Backup Engine Interface -```c -int backup_execute_hdd_enabled_strategy(const backup_config_t* config); -int backup_execute_hdd_disabled_strategy(const backup_config_t* config); -int backup_and_recover_logs(const char* source, const char* dest, - backup_operation_type_t op, const char* s_ext, - const char* d_ext); -``` - -### 4.4 File Operations Interface -```c -int file_move(const char* source, const char* dest); -int file_copy(const char* source, const char* dest); -int file_find_pattern(const char* dir, const char* pattern, - char results[][PATH_MAX], int max_results); -bool file_exists(const char* path); -int file_touch(const char* path); -``` - -## 5. Data Flow - -### 5.1 Main Execution Flow -1. **Initialization Phase** - - Load system configuration - - Initialize logging subsystem - - Validate runtime environment - -2. **Preparation Phase** - - Create required directories - - Check disk thresholds - - Determine backup strategy based on HDD status - -3. **Backup Execution Phase** - - Execute appropriate backup strategy - - Handle log rotation (HDD-disabled devices) - - Move/copy log files based on strategy - -4. **Cleanup Phase** - - Clean up old log files - - Copy system version files - - Send systemd notification - -5. **Termination Phase** - - Release resources - - Report final status - -### 5.2 Error Handling Flow -- Centralized error handling through error_info_t structure -- Error propagation through return codes -- Logging of all error conditions with context -- Graceful degradation on non-critical failures - -### 5.3 Visual Flow Representation - -The system workflow is represented through detailed diagrams stored in the [diagrams directory](diagrams/backup_logs_flowcharts.md). This includes both Mermaid-format diagrams and text-based alternatives for environments with limited diagram rendering capabilities. - -#### 5.3.1 Main Backup Process Flow -The primary execution flow showing the complete backup process from initialization to completion, including decision points for HDD-enabled vs HDD-disabled devices. Available in both Mermaid and text formats. - -#### 5.3.2 Component Interaction Sequence -A detailed sequence diagram illustrating the interactions between all system components, showing the order of operations and data flow between modules. Includes timing annotations and error paths. - -#### 5.3.3 HDD Disabled Strategy Detail -A specialized flowchart focusing on the complex log rotation logic for HDD-disabled devices, showing the 4-level backup rotation mechanism with all decision points clearly marked. - -#### 5.3.4 Error Handling and Recovery Flow -A comprehensive error handling flowchart showing how different types of errors are categorized, handled, and recovered from in the system. Includes recovery strategies and escalation paths. - -## 6. Key Algorithms - -### 6.1 HDD-Disabled Backup Algorithm -``` -1. Check for existing backup levels (messages.txt, bak1_messages.txt, etc.) -2. If no existing backups: - - Move all logs to PreviousLogs -3. If backup level 1 exists but not level 2: - - Move current logs to PreviousLogs with bak1_ prefix -4. If backup levels 1-2 exist but not level 3: - - Move current logs to PreviousLogs with bak2_ prefix -5. If all backup levels exist: - - Rotate: bak1->root, bak2->bak1, bak3->bak2, current->bak3 -6. Create last_reboot marker file -``` - -### 6.2 HDD-Enabled Backup Algorithm -``` -1. If no messages.txt in PreviousLogs: - - Move all logs to PreviousLogs - - Create last_reboot marker -2. If messages.txt exists: - - Remove existing last_reboot markers - - Create timestamped backup directory - - Move current logs to timestamped directory - - Create last_reboot marker in timestamped directory -``` - -### 6.3 File Pattern Matching Algorithm -- Use POSIX-compliant pattern matching -- Support for wildcard patterns (*.txt, *.log, etc.) -- Efficient directory traversal with depth control -- Filter by file type (regular files vs. symbolic links) - -## 7. Threading and Concurrency - -### 7.1 Threading Strategy -- Single-threaded design for simplicity and reliability -- Thread-safe utility functions for potential future extensions -- Use of atomic operations for shared state (if any) - -### 7.2 Synchronization -- File locking for critical operations -- Mutex protection for shared resources (if threading is added later) -- Process-level coordination through lockfiles - -## 8. Performance Considerations - -### 8.1 Memory Optimization -- Fixed-size buffers with compile-time sizing -- Stack allocation preference over heap allocation -- Minimal memory fragmentation through planned allocation patterns -- Efficient string handling with bounded operations - -### 8.2 I/O Optimization -- Batch file operations where possible -- Minimize system calls through buffered operations -- Efficient directory traversal algorithms -- Streaming operations for large files - -### 8.3 CPU Optimization -- Avoid expensive operations in loops -- Use bit operations for flags and states -- Minimize string operations and use const strings where possible -- Efficient pattern matching algorithms - -## 9. Integration Points - -### 9.1 System Integration -- **Systemd Integration**: Maintain compatibility with existing service files -- **Configuration Files**: Parse existing shell-format configuration files -- **External Scripts**: Integration with `disk_threshold_check.sh` -- **File System**: Interaction with various mount points and file systems - -### 9.2 Backward Compatibility -- Maintain existing directory structure and naming conventions -- Preserve log file formats and timestamps -- Keep existing environment variable usage -- Maintain compatibility with log analysis tools - -## 10. Error Handling Strategy - -### 10.1 Error Categories -- **Fatal Errors**: Configuration failures, permission issues -- **Recoverable Errors**: Individual file operation failures -- **Warnings**: Non-critical issues that don't prevent execution - -### 10.2 Error Reporting -- Structured error codes for programmatic handling -- Human-readable error messages for debugging -- Integration with existing logging infrastructure -- Syslog integration for system-level error reporting - -## 11. Testing Strategy - -### 11.1 Unit Testing -- Test individual modules in isolation -- Mock external dependencies (file system, system calls) -- Comprehensive error condition testing -- Memory leak detection and prevention - -### 11.2 Integration Testing -- Test complete backup scenarios -- Verify compatibility with existing system -- Performance benchmarking against shell script -- Multi-platform validation - -### 11.3 System Testing -- End-to-end functionality verification -- Stress testing with large log volumes -- Resource constraint testing -- Recovery testing after various failure scenarios - -## 12. Deployment Considerations - -### 12.1 Build System -- Integration with existing autotools configuration -- Cross-compilation support for multiple architectures -- Compiler optimization flags for embedded targets -- Static linking considerations for deployment - -### 12.2 Installation -- Backward-compatible installation process -- Service file updates for systemd integration -- Configuration migration support -- Rollback capability - -### 12.3 Monitoring -- Health check mechanisms -- Performance metrics collection -- Resource usage monitoring -- Integration with existing monitoring infrastructure - -## 13. Future Enhancements - -### 13.1 Planned Features -- Configuration hot-reloading capability -- Enhanced compression for archived logs -- Remote log backup capability -- Advanced filtering and retention policies - -### 13.2 Extensibility -- Plugin architecture for custom backup strategies -- Configurable backup policies -- API for external tools integration -- Event-driven architecture support - -## 14. Risk Analysis - -### 14.1 Technical Risks -- **Memory Management**: Risk of memory leaks in embedded environment -- **File System Operations**: Race conditions with concurrent access -- **Configuration Parsing**: Compatibility issues with shell variable expansion -- **Performance**: Potential performance regression compared to shell script - -### 14.2 Mitigation Strategies -- Comprehensive testing with memory analysis tools -- File locking and atomic operations for critical sections -- Robust configuration parsing with validation -- Performance benchmarking and optimization - -## 15. Success Criteria - -### 15.1 Functional Requirements -- ✅ Complete feature parity with existing shell script -- ✅ Support for both HDD-enabled and HDD-disabled devices -- ✅ Proper log rotation and backup functionality -- ✅ Integration with systemd and existing infrastructure - -### 15.2 Non-Functional Requirements -- ✅ Memory usage reduction of at least 20% compared to shell process -- ✅ Startup time improvement of at least 30% -- ✅ CPU usage reduction during backup operations -- ✅ Cross-platform compatibility across target embedded systems - -This HLD provides a comprehensive roadmap for migrating the backup_logs.sh script to a robust, efficient C implementation suitable for embedded RDK environments. From 9291578840111a112285dc430368f7debc208533 Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:45:34 +0530 Subject: [PATCH 5/9] Create backup_logs_migration_HLD.md --- backup_logs/backup_logs_migration_HLD.md | 396 +++++++++++++++++++++++ 1 file changed, 396 insertions(+) create mode 100644 backup_logs/backup_logs_migration_HLD.md diff --git a/backup_logs/backup_logs_migration_HLD.md b/backup_logs/backup_logs_migration_HLD.md new file mode 100644 index 000000000..944537b7c --- /dev/null +++ b/backup_logs/backup_logs_migration_HLD.md @@ -0,0 +1,396 @@ +# High-Level Design: backup_logs.sh Migration to C + +## 1. Overview + +### 1.1 Purpose +This document outlines the High-Level Design (HLD) for migrating the existing `backup_logs.sh` shell script to a C implementation. The migration aims to improve performance, reduce memory footprint, and enhance reliability for embedded RDK systems. + +### 1.2 Scope +- Migration of all functionality from `backup_logs.sh` to C code +- Support for both HDD-enabled and HDD-disabled devices +- Maintain compatibility with existing systemd integration +- Preserve log backup and rotation functionality + +### 1.3 Constraints +- Target embedded systems with limited memory (few KBs to few MBs) +- CPU resources are constrained with low clock speeds +- Must be platform-neutral and portable across multiple architectures +- Minimize dynamic memory allocation +- Avoid floating-point arithmetic where possible +- Thread-safe implementation required + +## 2. System Architecture + +### 2.1 Architecture Overview +The C implementation will follow a modular design with the following key components: + +``` +backup_logs (main executable) +├── Configuration Manager +├── Directory Manager +├── Log Backup Engine +├── File Operations Manager +├── Disk Threshold Monitor +├── System Integration Module +└── Error Handler & Logger +``` + +### 2.2 Component Description + +#### 2.2.1 Configuration Manager +- **Purpose**: Load and parse system configuration files +- **Responsibilities**: + - Parse `/etc/include.properties` + - Parse `/etc/device.properties` + - Parse `/etc/env_setup.sh` if available + - Validate configuration parameters + - Provide configuration data to other modules + +#### 2.2.2 Directory Manager +- **Purpose**: Handle directory creation and validation +- **Responsibilities**: + - Create log workspace directories + - Validate directory permissions + - Manage directory path resolution + - Handle directory cleanup operations + +#### 2.2.3 Log Backup Engine +- **Purpose**: Core backup logic implementation +- **Responsibilities**: + - Implement HDD-enabled device backup strategy + - Implement HDD-disabled device backup strategy with rotation + - Handle log file identification and filtering + - Execute backup operations based on device type + +#### 2.2.4 File Operations Manager +- **Purpose**: Low-level file operations +- **Responsibilities**: + - File moving and copying operations + - File existence checking + - Pattern-based file finding + - Timestamp generation and management + +#### 2.2.5 Disk Threshold Monitor +- **Purpose**: Monitor disk usage and trigger cleanup +- **Responsibilities**: + - Check disk usage percentages + - Trigger cleanup scripts when thresholds exceed + - Integration with existing disk_threshold_check.sh + +#### 2.2.6 System Integration Module +- **Purpose**: System-level integrations +- **Responsibilities**: + - Systemd notification handling + - Integration with external scripts + - Process status reporting + +#### 2.2.7 Error Handler & Logger +- **Purpose**: Centralized error handling and logging +- **Responsibilities**: + - Structured error reporting + - Log message formatting with timestamps + - Error code standardization + +## 3. Data Structures + +### 3.1 Core Data Structures + +```c +typedef struct { + char log_path[PATH_MAX]; + char prev_log_path[PATH_MAX]; + char prev_log_backup_path[PATH_MAX]; + char persistent_path[PATH_MAX]; + bool hdd_enabled; +} backup_config_t; + +typedef struct { + char source_path[PATH_MAX]; + char dest_path[PATH_MAX]; + backup_operation_t operation; + char source_extension[32]; + char dest_extension[32]; +} backup_operation_t; + +typedef enum { + BACKUP_OP_MOVE, + BACKUP_OP_COPY, + BACKUP_OP_DELETE +} backup_operation_type_t; + +typedef struct { + int error_code; + char error_message[256]; + const char* function_name; + int line_number; +} error_info_t; +``` + +### 3.2 Memory Management Strategy +- Use fixed-size buffers to avoid dynamic allocation +- Implement memory pools for temporary operations +- Stack-based allocation for small, short-lived data +- Pre-allocated arrays for file lists and paths + +## 4. Module Interfaces + +### 4.1 Configuration Manager Interface +```c +int config_load(backup_config_t* config); +int config_validate(const backup_config_t* config); +const char* config_get_log_path(void); +bool config_is_hdd_enabled(void); +``` + +### 4.2 Directory Manager Interface +```c +int dir_create_workspace(const char* path); +int dir_create_if_not_exists(const char* path); +int dir_cleanup(const char* path, const char* pattern); +bool dir_exists(const char* path); +``` + +### 4.3 Log Backup Engine Interface +```c +int backup_execute_hdd_enabled_strategy(const backup_config_t* config); +int backup_execute_hdd_disabled_strategy(const backup_config_t* config); +int backup_and_recover_logs(const char* source, const char* dest, + backup_operation_type_t op, const char* s_ext, + const char* d_ext); +``` + +### 4.4 File Operations Interface +```c +int file_move(const char* source, const char* dest); +int file_copy(const char* source, const char* dest); +int file_find_pattern(const char* dir, const char* pattern, + char results[][PATH_MAX], int max_results); +bool file_exists(const char* path); +int file_touch(const char* path); +``` + +## 5. Data Flow + +### 5.1 Main Execution Flow +1. **Initialization Phase** + - Load system configuration + - Initialize logging subsystem + - Validate runtime environment + +2. **Preparation Phase** + - Create required directories + - Check disk thresholds + - Determine backup strategy based on HDD status + +3. **Backup Execution Phase** + - Execute appropriate backup strategy + - Handle log rotation (HDD-disabled devices) + - Move/copy log files based on strategy + +4. **Cleanup Phase** + - Clean up old log files + - Copy system version files + - Send systemd notification + +5. **Termination Phase** + - Release resources + - Report final status + +### 5.2 Error Handling Flow +- Centralized error handling through error_info_t structure +- Error propagation through return codes +- Logging of all error conditions with context +- Graceful degradation on non-critical failures + +### 5.3 Visual Flow Representation + +The system workflow is represented through detailed diagrams stored in the [diagrams directory](diagrams/backup_logs_flowcharts.md). This includes both Mermaid-format diagrams and text-based alternatives for environments with limited diagram rendering capabilities. + +#### 5.3.1 Main Backup Process Flow +The primary execution flow showing the complete backup process from initialization to completion, including decision points for HDD-enabled vs HDD-disabled devices. Available in both Mermaid and text formats. + +#### 5.3.2 Component Interaction Sequence +A detailed sequence diagram illustrating the interactions between all system components, showing the order of operations and data flow between modules. Includes timing annotations and error paths. + +#### 5.3.3 HDD Disabled Strategy Detail +A specialized flowchart focusing on the complex log rotation logic for HDD-disabled devices, showing the 4-level backup rotation mechanism with all decision points clearly marked. + +#### 5.3.4 Error Handling and Recovery Flow +A comprehensive error handling flowchart showing how different types of errors are categorized, handled, and recovered from in the system. Includes recovery strategies and escalation paths. + +## 6. Key Algorithms + +### 6.1 HDD-Disabled Backup Algorithm +``` +1. Check for existing backup levels (messages.txt, bak1_messages.txt, etc.) +2. If no existing backups: + - Move all logs to PreviousLogs +3. If backup level 1 exists but not level 2: + - Move current logs to PreviousLogs with bak1_ prefix +4. If backup levels 1-2 exist but not level 3: + - Move current logs to PreviousLogs with bak2_ prefix +5. If all backup levels exist: + - Rotate: bak1->root, bak2->bak1, bak3->bak2, current->bak3 +6. Create last_reboot marker file +``` + +### 6.2 HDD-Enabled Backup Algorithm +``` +1. If no messages.txt in PreviousLogs: + - Move all logs to PreviousLogs + - Create last_reboot marker +2. If messages.txt exists: + - Remove existing last_reboot markers + - Create timestamped backup directory + - Move current logs to timestamped directory + - Create last_reboot marker in timestamped directory +``` + +### 6.3 File Pattern Matching Algorithm +- Use POSIX-compliant pattern matching +- Support for wildcard patterns (*.txt, *.log, etc.) +- Efficient directory traversal with depth control +- Filter by file type (regular files vs. symbolic links) + +## 7. Threading and Concurrency + +### 7.1 Threading Strategy +- Single-threaded design for simplicity and reliability +- Thread-safe utility functions for potential future extensions +- Use of atomic operations for shared state (if any) + +### 7.2 Synchronization +- File locking for critical operations +- Mutex protection for shared resources (if threading is added later) +- Process-level coordination through lockfiles + +## 8. Performance Considerations + +### 8.1 Memory Optimization +- Fixed-size buffers with compile-time sizing +- Stack allocation preference over heap allocation +- Minimal memory fragmentation through planned allocation patterns +- Efficient string handling with bounded operations + +### 8.2 I/O Optimization +- Batch file operations where possible +- Minimize system calls through buffered operations +- Efficient directory traversal algorithms +- Streaming operations for large files + +### 8.3 CPU Optimization +- Avoid expensive operations in loops +- Use bit operations for flags and states +- Minimize string operations and use const strings where possible +- Efficient pattern matching algorithms + +## 9. Integration Points + +### 9.1 System Integration +- **Systemd Integration**: Maintain compatibility with existing service files +- **Configuration Files**: Parse existing shell-format configuration files +- **External Scripts**: Integration with `disk_threshold_check.sh` +- **File System**: Interaction with various mount points and file systems + +### 9.2 Backward Compatibility +- Maintain existing directory structure and naming conventions +- Preserve log file formats and timestamps +- Keep existing environment variable usage +- Maintain compatibility with log analysis tools + +## 10. Error Handling Strategy + +### 10.1 Error Categories +- **Fatal Errors**: Configuration failures, permission issues +- **Recoverable Errors**: Individual file operation failures +- **Warnings**: Non-critical issues that don't prevent execution + +### 10.2 Error Reporting +- Structured error codes for programmatic handling +- Human-readable error messages for debugging +- Integration with existing logging infrastructure +- Syslog integration for system-level error reporting + +## 11. Testing Strategy + +### 11.1 Unit Testing +- Test individual modules in isolation +- Mock external dependencies (file system, system calls) +- Comprehensive error condition testing +- Memory leak detection and prevention + +### 11.2 Integration Testing +- Test complete backup scenarios +- Verify compatibility with existing system +- Performance benchmarking against shell script +- Multi-platform validation + +### 11.3 System Testing +- End-to-end functionality verification +- Stress testing with large log volumes +- Resource constraint testing +- Recovery testing after various failure scenarios + +## 12. Deployment Considerations + +### 12.1 Build System +- Integration with existing autotools configuration +- Cross-compilation support for multiple architectures +- Compiler optimization flags for embedded targets +- Static linking considerations for deployment + +### 12.2 Installation +- Backward-compatible installation process +- Service file updates for systemd integration +- Configuration migration support +- Rollback capability + +### 12.3 Monitoring +- Health check mechanisms +- Performance metrics collection +- Resource usage monitoring +- Integration with existing monitoring infrastructure + +## 13. Future Enhancements + +### 13.1 Planned Features +- Configuration hot-reloading capability +- Enhanced compression for archived logs +- Remote log backup capability +- Advanced filtering and retention policies + +### 13.2 Extensibility +- Plugin architecture for custom backup strategies +- Configurable backup policies +- API for external tools integration +- Event-driven architecture support + +## 14. Risk Analysis + +### 14.1 Technical Risks +- **Memory Management**: Risk of memory leaks in embedded environment +- **File System Operations**: Race conditions with concurrent access +- **Configuration Parsing**: Compatibility issues with shell variable expansion +- **Performance**: Potential performance regression compared to shell script + +### 14.2 Mitigation Strategies +- Comprehensive testing with memory analysis tools +- File locking and atomic operations for critical sections +- Robust configuration parsing with validation +- Performance benchmarking and optimization + +## 15. Success Criteria + +### 15.1 Functional Requirements +- ✅ Complete feature parity with existing shell script +- ✅ Support for both HDD-enabled and HDD-disabled devices +- ✅ Proper log rotation and backup functionality +- ✅ Integration with systemd and existing infrastructure + +### 15.2 Non-Functional Requirements +- ✅ Memory usage reduction of at least 20% compared to shell process +- ✅ Startup time improvement of at least 30% +- ✅ CPU usage reduction during backup operations +- ✅ Cross-platform compatibility across target embedded systems + +This HLD provides a comprehensive roadmap for migrating the backup_logs.sh script to a robust, efficient C implementation suitable for embedded RDK environments. From a358ddd51d307db10445fd24d99e2d8b76dc5de0 Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:46:49 +0530 Subject: [PATCH 6/9] Create backup_logs_requirements.md --- backup_logs/backup_logs_requirements.md | 274 ++++++++++++++++++++++++ 1 file changed, 274 insertions(+) create mode 100644 backup_logs/backup_logs_requirements.md diff --git a/backup_logs/backup_logs_requirements.md b/backup_logs/backup_logs_requirements.md new file mode 100644 index 000000000..df2f2686e --- /dev/null +++ b/backup_logs/backup_logs_requirements.md @@ -0,0 +1,274 @@ +# Functional Requirements: backup_logs.sh Migration + +## 1. Overview + +This document outlines the detailed functional requirements for migrating the `backup_logs.sh` shell script to a C implementation for embedded RDK systems. + +## 2. Functional Requirements + +### 2.1 Configuration Management (REQ-001) +**Description**: The system must load and parse configuration from multiple sources +**Requirements**: +- Parse `/etc/include.properties` for log path configuration +- Parse `/etc/device.properties` for device-specific settings +- Parse `/etc/env_setup.sh` if available for environment variables +- Extract `LOG_PATH`, `HDD_ENABLED`, and `APP_PERSISTENT_PATH` variables +- Validate all configuration parameters before proceeding +- Handle missing or malformed configuration files gracefully + +**Input**: Configuration files in shell variable format +**Output**: Structured configuration data +**Error Handling**: Log configuration errors and exit with appropriate error code + +### 2.2 Directory Management (REQ-002) +**Description**: Create and manage required log directory structures +**Requirements**: +- Create `$LOG_PATH` directory if it doesn't exist +- Create `$LOG_PATH/PreviousLogs` directory structure +- Create `$LOG_PATH/PreviousLogs_backup` directory structure +- Clean existing backup directory contents before use +- Set appropriate permissions on created directories +- Handle directory creation failures gracefully + +**Input**: Configuration paths +**Output**: Created directory structures +**Constraints**: Must work with various filesystem types and permissions + +### 2.3 Disk Threshold Monitoring (REQ-003) +**Description**: Monitor disk usage and trigger cleanup when necessary +**Requirements**: +- Execute disk threshold check if `/lib/rdk/disk_threshold_check.sh` exists +- Pass parameter `0` to the disk check script +- Handle script execution failures without stopping backup process +- Log disk check results for monitoring + +**Input**: Disk check script path +**Output**: Disk status information +**Dependencies**: External `disk_threshold_check.sh` script + +### 2.4 HDD-Disabled Device Backup Strategy (REQ-004) +**Description**: Implement 4-level log rotation for devices without HDD +**Requirements**: +- Support up to 4 backup levels: base, bak1_, bak2_, bak3_ +- Move files based on existing backup level: + - Level 0: Move current logs to PreviousLogs + - Level 1: Move current logs with `bak1_` prefix + - Level 2: Move current logs with `bak2_` prefix + - Level 3: Rotate all levels (bak1→base, bak2→bak1, bak3→bak2, current→bak3) +- Create `last_reboot` marker file after each backup +- Clean current log directory after backup completion + +**Input**: Current log files and existing backup state +**Output**: Rotated backup files with appropriate naming +**File Patterns**: `*.txt*`, `*.log*`, `*.bin*`, `bootlog` files + +### 2.5 HDD-Enabled Device Backup Strategy (REQ-005) +**Description**: Implement timestamped backup for devices with HDD +**Requirements**: +- Check for existing `messages.txt` in PreviousLogs directory +- If no existing backup: Move all logs to PreviousLogs directory +- If backup exists: Create timestamped backup directory (`logbackup-MM-DD-YY-HH-MM-SSAM`) +- Move current logs to timestamped directory +- Remove any existing `last_reboot` markers before creating new one +- Create `last_reboot` marker in appropriate location + +**Input**: Current log files and existing backup state +**Output**: Timestamped backup directories with organized log files +**File Patterns**: `*.txt*`, `*.log*`, `bootlog` files (no `.bin*` files) + +### 2.6 File Operations (REQ-006) +**Description**: Perform reliable file and directory operations +**Requirements**: +- Move files with error handling and validation +- Support pattern-based file finding (find with depth and type constraints) +- Handle both regular files and symbolic links +- Implement atomic file operations where possible +- Validate file operations and report failures +- Support large numbers of files efficiently + +**Input**: Source and destination paths, file patterns +**Output**: Moved/copied files with status reporting +**Constraints**: Must handle filesystem limitations and permissions + +### 2.7 Version File Management (REQ-007) +**Description**: Copy system version information to log directory +**Requirements**: +- Copy `/version.txt` to current log directory +- Copy `/etc/skyversion.txt` to current log directory as `skyversion.txt` +- Copy `/etc/rippleversion.txt` to current log directory as `rippleversion.txt` +- Handle missing version files gracefully (non-fatal errors) +- Preserve file timestamps and permissions where possible + +**Input**: System version files +**Output**: Version files in log directory +**Error Handling**: Log warnings for missing files but continue execution + +### 2.8 Special Log File Handling (REQ-008) +**Description**: Handle temporary and special log files +**Requirements**: +- Move `/tmp/disk_cleanup.log` to current log directory if exists +- Move `/tmp/mount_log.txt` to current log directory if exists +- Move `/tmp/mount-ta_log.txt` to current log directory if exists +- Handle file moves atomically to prevent data loss +- Continue execution if special files are missing + +**Input**: Temporary log files from `/tmp` directory +**Output**: Special log files moved to permanent location +**Timing**: Execute after main backup but before cleanup + +### 2.9 System Integration (REQ-009) +**Description**: Integrate with systemd and system services +**Requirements**: +- Send systemd ready notification upon completion +- Set systemd status message: "Logs Backup Done..!" +- Create persistent marker file at `$PERSISTENT_PATH/logFileBackup` +- Handle systemd notification failures gracefully +- Support operation in non-systemd environments + +**Input**: Completion status +**Output**: System notifications and marker files +**Dependencies**: systemd-notify command availability + +### 2.10 Logging and Monitoring (REQ-010) +**Description**: Provide comprehensive logging and error reporting +**Requirements**: +- Log all major operations with timestamps +- Use format: `timestamp scriptname: message` +- Support different log levels (info, warning, error, debug) +- Log to stdout/stderr for systemd journal integration +- Include function name and line numbers in error logs +- Support structured logging for automated processing + +**Input**: Operation status and error conditions +**Output**: Formatted log messages with timestamps +**Format**: Compatible with existing RDK logging standards + +## 3. Non-Functional Requirements + +### 3.1 Performance Requirements (NFR-001) +- Memory usage must be ≤ 512KB peak during operation +- Startup time must be ≤ 2 seconds on target hardware +- File operations must complete within 30 seconds for typical log volumes +- CPU usage should not exceed 10% during backup operations + +### 3.2 Reliability Requirements (NFR-002) +- System must handle unexpected shutdowns gracefully +- Backup operations must be atomic (complete or rollback) +- Must recover from partial backup states on restart +- Handle filesystem full conditions without data loss + +### 3.3 Portability Requirements (NFR-003) +- Support ARM, MIPS, and x86 architectures +- Compatible with various embedded Linux distributions +- Work with different filesystem types (ext4, JFFS2, UBIFS) +- Support cross-compilation toolchains + +### 3.4 Security Requirements (NFR-004) +- Validate all file paths to prevent directory traversal +- Handle file permissions correctly without privilege escalation +- Sanitize all inputs from configuration files +- Protect against symlink attacks during file operations + +## 4. Input/Output Specifications + +### 4.1 Inputs +- **Configuration Files**: `/etc/include.properties`, `/etc/device.properties`, `/etc/env_setup.sh` +- **Log Files**: Files matching patterns `*.txt*`, `*.log*`, `*.bin*`, `bootlog` +- **Version Files**: `/version.txt`, `/etc/skyversion.txt`, `/etc/rippleversion.txt` +- **Temporary Files**: `/tmp/disk_cleanup.log`, `/tmp/mount_log.txt`, `/tmp/mount-ta_log.txt` + +### 4.2 Outputs +- **Backup Directories**: Organized log file backups with appropriate naming +- **Marker Files**: `last_reboot` markers for tracking backup cycles +- **System Notifications**: systemd ready notifications and status messages +- **Log Messages**: Timestamped operation logs for monitoring + +### 4.3 Error Codes +- **0**: Success - All operations completed successfully +- **1**: Configuration Error - Invalid or missing configuration +- **2**: Filesystem Error - Directory creation or file operation failure +- **3**: Permission Error - Insufficient permissions for required operations +- **4**: Resource Error - Insufficient disk space or memory +- **5**: System Error - External script or system call failure + +## 5. Dependencies + +### 5.1 System Dependencies +- POSIX-compliant filesystem +- Standard C library (libc) +- systemd-notify utility (optional) +- `/bin/timestamp` utility for log formatting +- Access to `/proc` filesystem for system information + +### 5.2 External Scripts +- `/lib/rdk/disk_threshold_check.sh` - Disk usage monitoring +- Configuration parsing utilities for shell variable format + +### 5.3 File System Requirements +- Write access to log directories +- Sufficient disk space for log rotation (minimum 2x current log size) +- Support for atomic file operations (rename) + +## 6. Constraints + +### 6.1 Timing Constraints +- Must complete within systemd service timeout (typically 90 seconds) +- Backup rotation should complete within 10 seconds for typical volumes +- Configuration loading must complete within 1 second + +### 6.2 Memory Constraints +- Peak memory usage limited to 512KB on embedded systems +- No dynamic memory allocation for file lists exceeding 100MB +- Stack usage limited to 64KB maximum depth + +### 6.3 Storage Constraints +- Must work with log directories up to 1GB in size +- Support up to 10,000 individual log files +- Handle filenames up to 255 characters (filesystem limit) + +## 7. Edge Cases and Error Scenarios + +### 7.1 Configuration Edge Cases +- Missing configuration files +- Malformed shell variable syntax +- Invalid path specifications +- Conflicting configuration values +- Unicode characters in paths + +### 7.2 Filesystem Edge Cases +- Disk full during backup operations +- Permission changes during execution +- Network filesystem disconnections +- Corrupted filesystem states +- Very large individual log files (>100MB) + +### 7.3 System Edge Cases +- System shutdown during backup +- Multiple backup processes running simultaneously +- Clock adjustments affecting timestamps +- Filesystem readonly states +- Missing system utilities + +## 8. Acceptance Criteria + +### 8.1 Functional Acceptance +- [ ] All backup strategies work correctly for both HDD configurations +- [ ] Log rotation maintains proper sequence and naming +- [ ] Version file copying works without data loss +- [ ] System integrations (systemd) function properly +- [ ] Error handling provides useful diagnostic information + +### 8.2 Performance Acceptance +- [ ] Memory usage stays within embedded system constraints +- [ ] Startup and completion times meet target requirements +- [ ] File operations scale appropriately with log volume +- [ ] CPU usage remains reasonable during operation + +### 8.3 Reliability Acceptance +- [ ] Operations complete successfully in normal conditions +- [ ] System handles error conditions gracefully +- [ ] Recovery from partial states works correctly +- [ ] No data loss occurs during operations +- [ ] Cross-platform compatibility verified + +This requirements document provides the foundation for implementing a robust, efficient C replacement for the backup_logs.sh script that meets the needs of embedded RDK systems. From 73410575e3c401de99aeefb3eaa52eaafd53d536 Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:48:16 +0530 Subject: [PATCH 7/9] Add Low-Level Design for backup_logs.sh migration This document outlines the low-level design for migrating the backup_logs.sh script to C, detailing data structures, algorithms, and error handling for embedded RDK systems. --- backup_logs/backup_logs_LLD.md | 1026 ++++++++++++++++++++++++++++++++ 1 file changed, 1026 insertions(+) create mode 100644 backup_logs/backup_logs_LLD.md diff --git a/backup_logs/backup_logs_LLD.md b/backup_logs/backup_logs_LLD.md new file mode 100644 index 000000000..32593cf51 --- /dev/null +++ b/backup_logs/backup_logs_LLD.md @@ -0,0 +1,1026 @@ +# Low-Level Design: backup_logs.sh Migration to C + +## 1. Overview + +This Low-Level Design (LLD) document provides detailed implementation specifications for migrating the `backup_logs.sh` shell script to C code for embedded RDK systems. + +## 2. Detailed Data Structures + +### 2.1 Core Configuration Structure +```c +#define MAX_PATH_LEN 4096 +#define MAX_MESSAGE_LEN 256 +#define MAX_EXTENSION_LEN 32 +#define MAX_FILES_PER_DIR 10000 +#define MAX_CONFIG_LINE 512 + +typedef struct backup_config { + char log_path[MAX_PATH_LEN]; + char prev_log_path[MAX_PATH_LEN]; + char prev_log_backup_path[MAX_PATH_LEN]; + char persistent_path[MAX_PATH_LEN]; + bool hdd_enabled; + bool config_loaded; +} backup_config_t; +``` + +### 2.2 File Operation Structures +```c +typedef enum { + BACKUP_OP_MOVE, + BACKUP_OP_COPY, + BACKUP_OP_DELETE, + BACKUP_OP_TOUCH +} backup_operation_type_t; + +typedef struct file_entry { + char filename[MAX_PATH_LEN]; + char source_path[MAX_PATH_LEN]; + char dest_path[MAX_PATH_LEN]; + backup_operation_type_t operation; + time_t timestamp; +} file_entry_t; + +typedef struct file_list { + file_entry_t files[MAX_FILES_PER_DIR]; + int count; + int capacity; +} file_list_t; +``` + +### 2.3 Error Handling Structures +```c +typedef enum { + ERROR_SUCCESS = 0, + ERROR_CONFIG_INVALID = 1, + ERROR_FILESYSTEM = 2, + ERROR_PERMISSION = 3, + ERROR_RESOURCE = 4, + ERROR_SYSTEM = 5 +} error_code_t; + +typedef struct error_context { + error_code_t code; + char message[MAX_MESSAGE_LEN]; + const char* function_name; + int line_number; + time_t timestamp; +} error_context_t; +``` + +### 2.4 Backup Level Tracking +```c +typedef enum { + BACKUP_LEVEL_NONE = -1, + BACKUP_LEVEL_BASE = 0, + BACKUP_LEVEL_BAK1 = 1, + BACKUP_LEVEL_BAK2 = 2, + BACKUP_LEVEL_BAK3 = 3 +} backup_level_t; + +typedef struct backup_state { + backup_level_t current_level; + bool has_existing_backup; + char timestamp_str[32]; // Format: MM-DD-YY-HH-MM-SSAM +} backup_state_t; +``` + +## 3. Module Interface Definitions + +### 3.1 Configuration Manager Module +```c +// config_manager.h + +// Initialize configuration system +int config_init(void); + +// Load configuration from files +int config_load(backup_config_t* config); + +// Validate loaded configuration +int config_validate(const backup_config_t* config); + +// Parse shell-format property file +int config_parse_properties(const char* filepath, backup_config_t* config); + +// Parse environment setup script +int config_parse_env_script(const char* filepath, backup_config_t* config); + +// Get specific configuration values +const char* config_get_log_path(const backup_config_t* config); +bool config_is_hdd_enabled(const backup_config_t* config); +const char* config_get_persistent_path(const backup_config_t* config); + +// Cleanup configuration resources +void config_cleanup(backup_config_t* config); +``` + +### 3.2 Directory Manager Module +```c +// directory_manager.h + +// Create directory with proper permissions +int dir_create(const char* path, mode_t mode); + +// Create directory recursively (like mkdir -p) +int dir_create_recursive(const char* path, mode_t mode); + +// Check if directory exists +bool dir_exists(const char* path); + +// Create all required backup directories +int dir_create_workspace(const backup_config_t* config); + +// Clean directory contents matching pattern +int dir_cleanup(const char* path, const char* pattern); + +// Get directory size (total bytes) +long dir_get_size(const char* path); + +// Validate directory permissions +int dir_check_permissions(const char* path, int required_perms); +``` + +### 3.3 File Operations Module +```c +// file_operations.h + +// Initialize file operations system +int fileops_init(void); + +// Move file atomically +int fileops_move(const char* source, const char* dest); + +// Copy file with verification +int fileops_copy(const char* source, const char* dest); + +// Create empty file (touch) +int fileops_touch(const char* path); + +// Check file existence +bool fileops_exists(const char* path); + +// Find files matching pattern +int fileops_find_pattern(const char* directory, const char* pattern, + file_list_t* results); + +// Get file size +long fileops_get_size(const char* path); + +// Verify file integrity after operation +int fileops_verify_integrity(const char* path, const char* checksum); + +// Remove file safely +int fileops_remove(const char* path); + +// Cleanup file operations resources +void fileops_cleanup(void); +``` + +### 3.4 Backup Engine Module +```c +// backup_engine.h + +// Initialize backup engine +int backup_init(const backup_config_t* config); + +// Determine current backup level for HDD-disabled devices +backup_level_t backup_get_level_hdd_disabled(const backup_config_t* config); + +// Execute HDD-enabled backup strategy +int backup_execute_hdd_enabled(const backup_config_t* config); + +// Execute HDD-disabled backup strategy +int backup_execute_hdd_disabled(const backup_config_t* config); + +// Perform log rotation for HDD-disabled devices +int backup_rotate_logs(const backup_config_t* config, backup_level_t current_level); + +// Create timestamped backup directory +int backup_create_timestamped_dir(const backup_config_t* config, + char* timestamp_dir, size_t dir_size); + +// Create last_reboot marker file +int backup_create_reboot_marker(const char* directory); + +// Remove old reboot markers +int backup_remove_old_markers(const char* directory); + +// Cleanup backup engine resources +void backup_cleanup(void); +``` + +### 3.5 System Integration Module +```c +// system_integration.h + +// Initialize system integration +int sysint_init(void); + +// Send systemd notification +int sysint_notify_systemd(const char* message); + +// Execute external script safely +int sysint_execute_script(const char* script_path, const char* args); + +// Check disk threshold +int sysint_check_disk_threshold(void); + +// Generate timestamp string +int sysint_get_timestamp(char* buffer, size_t buffer_size); + +// Create persistent marker file +int sysint_create_persistent_marker(const char* path); + +// Cleanup system integration resources +void sysint_cleanup(void); +``` + +### 3.6 Error Handler and Logger Module +```c +// error_logger.h + +// Initialize logging system +int logger_init(const char* program_name); + +// Log message with level +int logger_log(int level, const char* function, int line, const char* format, ...); + +// Log error with context +int logger_error(const error_context_t* context); + +// Set error context +void logger_set_error(error_context_t* context, error_code_t code, + const char* function, int line, const char* message); + +// Get last error +const error_context_t* logger_get_last_error(void); + +// Clear error state +void logger_clear_error(void); + +// Cleanup logging resources +void logger_cleanup(void); + +// Convenience macros +#define LOG_ERROR(ctx, code, msg) \ + logger_set_error(ctx, code, __FUNCTION__, __LINE__, msg) + +#define LOG_INFO(msg, ...) \ + logger_log(LOG_LEVEL_INFO, __FUNCTION__, __LINE__, msg, ##__VA_ARGS__) + +#define LOG_WARN(msg, ...) \ + logger_log(LOG_LEVEL_WARN, __FUNCTION__, __LINE__, msg, ##__VA_ARGS__) +``` + +## 4. Detailed Algorithms + +### 4.1 Configuration Parsing Algorithm +```c +int config_parse_shell_variable(const char* line, char* key, char* value) { + // Skip comments and empty lines + if (line[0] == '#' || line[0] == '\0' || line[0] == '\n') { + return 0; + } + + // Find assignment operator + char* equals = strchr(line, '='); + if (!equals) { + return -1; // Invalid format + } + + // Extract key (trim whitespace) + size_t key_len = equals - line; + strncpy(key, line, key_len); + key[key_len] = '\0'; + // Trim trailing whitespace from key + while (key_len > 0 && isspace(key[key_len-1])) { + key[--key_len] = '\0'; + } + + // Extract value (handle quotes and expansion) + char* value_start = equals + 1; + // Skip leading whitespace + while (*value_start && isspace(*value_start)) { + value_start++; + } + + // Handle quoted values + if (*value_start == '"' || *value_start == '\'') { + char quote = *value_start; + value_start++; + char* quote_end = strchr(value_start, quote); + if (quote_end) { + size_t value_len = quote_end - value_start; + strncpy(value, value_start, value_len); + value[value_len] = '\0'; + } else { + return -1; // Unterminated quote + } + } else { + // Unquoted value - take until newline or comment + strcpy(value, value_start); + char* comment = strchr(value, '#'); + if (comment) { + *comment = '\0'; + } + // Trim trailing whitespace + size_t value_len = strlen(value); + while (value_len > 0 && isspace(value[value_len-1])) { + value[--value_len] = '\0'; + } + } + + return 1; // Success +} +``` + +### 4.2 HDD-Disabled Backup Level Detection +```c +backup_level_t backup_detect_level_hdd_disabled(const backup_config_t* config) { + char filepath[MAX_PATH_LEN]; + + // Check for messages.txt (base level) + snprintf(filepath, sizeof(filepath), "%s/messages.txt", config->prev_log_path); + if (!fileops_exists(filepath)) { + return BACKUP_LEVEL_NONE; + } + + // Check for bak1_messages.txt + snprintf(filepath, sizeof(filepath), "%s/bak1_messages.txt", config->prev_log_path); + if (!fileops_exists(filepath)) { + return BACKUP_LEVEL_BASE; + } + + // Check for bak2_messages.txt + snprintf(filepath, sizeof(filepath), "%s/bak2_messages.txt", config->prev_log_path); + if (!fileops_exists(filepath)) { + return BACKUP_LEVEL_BAK1; + } + + // Check for bak3_messages.txt + snprintf(filepath, sizeof(filepath), "%s/bak3_messages.txt", config->prev_log_path); + if (!fileops_exists(filepath)) { + return BACKUP_LEVEL_BAK2; + } + + return BACKUP_LEVEL_BAK3; // All levels exist, need rotation +} +``` + +### 4.3 File Pattern Matching Algorithm +```c +int fileops_find_pattern_impl(const char* directory, const char* pattern, + file_list_t* results) { + DIR* dir = opendir(directory); + if (!dir) { + return -1; + } + + struct dirent* entry; + results->count = 0; + + while ((entry = readdir(dir)) != NULL && results->count < results->capacity) { + // Skip . and .. + if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) { + continue; + } + + // Check if filename matches any of the patterns + bool matches = false; + + // Support multiple patterns: *.txt*, *.log*, *.bin*, bootlog + if (strstr(pattern, "*.txt*") && + (strstr(entry->d_name, ".txt") || strstr(entry->d_name, ".TXT"))) { + matches = true; + } else if (strstr(pattern, "*.log*") && + (strstr(entry->d_name, ".log") || strstr(entry->d_name, ".LOG"))) { + matches = true; + } else if (strstr(pattern, "*.bin*") && + (strstr(entry->d_name, ".bin") || strstr(entry->d_name, ".BIN"))) { + matches = true; + } else if (strstr(pattern, "bootlog") && + strcmp(entry->d_name, "bootlog") == 0) { + matches = true; + } + + if (matches) { + // Build full path + snprintf(results->files[results->count].filename, + sizeof(results->files[results->count].filename), + "%s", entry->d_name); + snprintf(results->files[results->count].source_path, + sizeof(results->files[results->count].source_path), + "%s/%s", directory, entry->d_name); + results->count++; + } + } + + closedir(dir); + return results->count; +} +``` + +### 4.4 Log Rotation Algorithm for HDD-Disabled Devices +```c +int backup_rotate_files_hdd_disabled(const backup_config_t* config) { + char source_path[MAX_PATH_LEN]; + char dest_path[MAX_PATH_LEN]; + file_list_t file_list = {0}; + file_list.capacity = MAX_FILES_PER_DIR; + + // Step 1: Move bak1_ files to base names (bak1_messages.txt -> messages.txt) + if (fileops_find_pattern(config->prev_log_path, "bak1_*", &file_list) > 0) { + for (int i = 0; i < file_list.count; i++) { + // Remove bak1_ prefix + const char* base_name = file_list.files[i].filename + 5; // Skip "bak1_" + snprintf(dest_path, sizeof(dest_path), "%s/%s", + config->prev_log_path, base_name); + + if (fileops_move(file_list.files[i].source_path, dest_path) != 0) { + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, + "Failed to rotate bak1 file to base"); + return -1; + } + } + } + + // Step 2: Move bak2_ files to bak1_ names + file_list.count = 0; // Reset list + if (fileops_find_pattern(config->prev_log_path, "bak2_*", &file_list) > 0) { + for (int i = 0; i < file_list.count; i++) { + // Replace bak2_ with bak1_ + const char* base_name = file_list.files[i].filename + 5; // Skip "bak2_" + snprintf(dest_path, sizeof(dest_path), "%s/bak1_%s", + config->prev_log_path, base_name); + + if (fileops_move(file_list.files[i].source_path, dest_path) != 0) { + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, + "Failed to rotate bak2 file to bak1"); + return -1; + } + } + } + + // Step 3: Move bak3_ files to bak2_ names + file_list.count = 0; // Reset list + if (fileops_find_pattern(config->prev_log_path, "bak3_*", &file_list) > 0) { + for (int i = 0; i < file_list.count; i++) { + // Replace bak3_ with bak2_ + const char* base_name = file_list.files[i].filename + 5; // Skip "bak3_" + snprintf(dest_path, sizeof(dest_path), "%s/bak2_%s", + config->prev_log_path, base_name); + + if (fileops_move(file_list.files[i].source_path, dest_path) != 0) { + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, + "Failed to rotate bak3 file to bak2"); + return -1; + } + } + } + + // Step 4: Move current logs to bak3_ names + file_list.count = 0; // Reset list + char log_patterns[] = "*.txt*,*.log*,*.bin*,bootlog"; + if (fileops_find_pattern(config->log_path, log_patterns, &file_list) > 0) { + for (int i = 0; i < file_list.count; i++) { + snprintf(dest_path, sizeof(dest_path), "%s/bak3_%s", + config->prev_log_path, file_list.files[i].filename); + + if (fileops_move(file_list.files[i].source_path, dest_path) != 0) { + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, + "Failed to move current log to bak3"); + return -1; + } + } + } + + return 0; // Success +} +``` + +### 4.5 Timestamp Generation Algorithm +```c +int sysint_generate_timestamp(char* buffer, size_t buffer_size) { + time_t raw_time; + struct tm* time_info; + + // Get current time + time(&raw_time); + time_info = localtime(&raw_time); + + if (!time_info) { + return -1; + } + + // Format: MM-DD-YY-HH-MM-SSAM (e.g., 03-05-26-02-30-45PM) + char am_pm = (time_info->tm_hour >= 12) ? 'P' : 'A'; + int hour_12 = time_info->tm_hour; + if (hour_12 == 0) { + hour_12 = 12; // 12 AM + } else if (hour_12 > 12) { + hour_12 -= 12; // Convert to 12-hour format + } + + int bytes_written = snprintf(buffer, buffer_size, + "%02d-%02d-%02d-%02d-%02d-%02d%cM", + time_info->tm_mon + 1, // Month (1-12) + time_info->tm_mday, // Day (1-31) + time_info->tm_year % 100, // Year (2-digit) + hour_12, // Hour (1-12) + time_info->tm_min, // Minute (0-59) + time_info->tm_sec, // Second (0-59) + am_pm); // AM/PM + + if (bytes_written < 0 || bytes_written >= buffer_size) { + return -1; // Buffer overflow or formatting error + } + + return 0; // Success +} +``` + +## 5. Error Handling Implementation + +### 5.1 Error Context Management +```c +static error_context_t g_last_error = {0}; + +void logger_set_error(error_context_t* context, error_code_t code, + const char* function, int line, const char* message) { + if (!context) { + context = &g_last_error; + } + + context->code = code; + context->function_name = function; + context->line_number = line; + time(&context->timestamp); + + // Copy message safely + if (message) { + strncpy(context->message, message, sizeof(context->message) - 1); + context->message[sizeof(context->message) - 1] = '\0'; + } else { + context->message[0] = '\0'; + } +} + +int logger_error(const error_context_t* context) { + struct tm* time_info = localtime(&context->timestamp); + char time_str[64]; + + strftime(time_str, sizeof(time_str), "%Y-%m-%d %H:%M:%S", time_info); + + fprintf(stderr, "[%s] ERROR %d in %s:%d: %s\n", + time_str, context->code, context->function_name, + context->line_number, context->message); + + return context->code; +} +``` + +### 5.2 Recovery Strategies +```c +int backup_recover_from_partial_state(const backup_config_t* config) { + // Check for incomplete operations by looking for temporary files + file_list_t temp_files = {0}; + temp_files.capacity = MAX_FILES_PER_DIR; + + // Look for .tmp, .bak, or other temporary extensions + if (fileops_find_pattern(config->log_path, "*.tmp", &temp_files) > 0) { + LOG_WARN("Found %d temporary files, attempting recovery", temp_files.count); + + for (int i = 0; i < temp_files.count; i++) { + // Try to determine original filename + char original_name[MAX_PATH_LEN]; + strncpy(original_name, temp_files.files[i].filename, + strlen(temp_files.files[i].filename) - 4); // Remove .tmp + original_name[strlen(temp_files.files[i].filename) - 4] = '\0'; + + char original_path[MAX_PATH_LEN]; + snprintf(original_path, sizeof(original_path), "%s/%s", + config->log_path, original_name); + + // If original doesn't exist, restore from temp + if (!fileops_exists(original_path)) { + if (fileops_move(temp_files.files[i].source_path, original_path) == 0) { + LOG_INFO("Recovered file: %s", original_name); + } + } else { + // Original exists, remove temp file + fileops_remove(temp_files.files[i].source_path); + } + } + } + + // Check for incomplete backup directories + DIR* dir = opendir(config->prev_log_path); + if (dir) { + struct dirent* entry; + while ((entry = readdir(dir)) != NULL) { + // Look for directories with .incomplete suffix + if (strstr(entry->d_name, ".incomplete")) { + char incomplete_path[MAX_PATH_LEN]; + snprintf(incomplete_path, sizeof(incomplete_path), "%s/%s", + config->prev_log_path, entry->d_name); + + LOG_WARN("Found incomplete backup directory: %s", incomplete_path); + // Remove incomplete backup directory + dir_cleanup(incomplete_path, "*"); + rmdir(incomplete_path); + } + } + closedir(dir); + } + + return 0; +} +``` + +## 6. Memory Management Strategy + +### 6.1 Fixed Buffer Pool Implementation +```c +#define BUFFER_POOL_SIZE 10 +#define BUFFER_SIZE 4096 + +static struct { + char buffers[BUFFER_POOL_SIZE][BUFFER_SIZE]; + bool in_use[BUFFER_POOL_SIZE]; + int allocated_count; +} g_buffer_pool = {0}; + +char* buffer_pool_allocate(void) { + for (int i = 0; i < BUFFER_POOL_SIZE; i++) { + if (!g_buffer_pool.in_use[i]) { + g_buffer_pool.in_use[i] = true; + g_buffer_pool.allocated_count++; + return g_buffer_pool.buffers[i]; + } + } + return NULL; // Pool exhausted +} + +void buffer_pool_free(char* buffer) { + for (int i = 0; i < BUFFER_POOL_SIZE; i++) { + if (g_buffer_pool.buffers[i] == buffer) { + g_buffer_pool.in_use[i] = false; + g_buffer_pool.allocated_count--; + return; + } + } +} + +int buffer_pool_get_usage(void) { + return g_buffer_pool.allocated_count; +} +``` + +### 6.2 Stack-based File Operation +```c +int fileops_move_safe(const char* source, const char* dest) { + char temp_dest[MAX_PATH_LEN]; // Stack allocation + error_context_t error_ctx = {0}; // Stack allocation + + // Create temporary destination name + snprintf(temp_dest, sizeof(temp_dest), "%s.tmp", dest); + + // Step 1: Copy to temporary location + if (fileops_copy(source, temp_dest) != 0) { + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, "Failed to copy to temporary location"); + return -1; + } + + // Step 2: Verify copy integrity + if (fileops_get_size(source) != fileops_get_size(temp_dest)) { + fileops_remove(temp_dest); + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, "File size mismatch after copy"); + return -1; + } + + // Step 3: Atomic rename + if (rename(temp_dest, dest) != 0) { + fileops_remove(temp_dest); + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, "Failed to rename to final destination"); + return -1; + } + + // Step 4: Remove original + if (fileops_remove(source) != 0) { + // Log warning but don't fail the operation + LOG_WARN("Failed to remove source file: %s", source); + } + + return 0; // Success +} +``` + +## 7. Performance Optimization Techniques + +### 7.1 Batch File Operations +```c +int fileops_batch_move(const file_list_t* file_list, const char* dest_dir) { + int success_count = 0; + int total_files = file_list->count; + + // Pre-allocate destination paths to avoid repeated allocations + char dest_paths[MAX_FILES_PER_DIR][MAX_PATH_LEN]; + + // Prepare all destination paths first + for (int i = 0; i < total_files; i++) { + snprintf(dest_paths[i], sizeof(dest_paths[i]), "%s/%s", + dest_dir, file_list->files[i].filename); + } + + // Execute moves in batch with progress tracking + for (int i = 0; i < total_files; i++) { + if (fileops_move_safe(file_list->files[i].source_path, dest_paths[i]) == 0) { + success_count++; + } else { + LOG_WARN("Failed to move file %d of %d: %s", + i + 1, total_files, file_list->files[i].filename); + } + + // Report progress every 100 files for large operations + if (total_files > 100 && (i + 1) % 100 == 0) { + LOG_INFO("Moved %d of %d files (%d%%)", success_count, i + 1, + ((i + 1) * 100) / total_files); + } + } + + LOG_INFO("Batch move completed: %d of %d files successful", + success_count, total_files); + + return (success_count == total_files) ? 0 : -1; +} +``` + +### 7.2 Efficient Directory Traversal +```c +int fileops_find_pattern_optimized(const char* directory, const char* pattern, + file_list_t* results) { + DIR* dir = opendir(directory); + if (!dir) { + return -1; + } + + struct dirent* entry; + results->count = 0; + + // Pre-compile pattern matching criteria for efficiency + bool match_txt = strstr(pattern, "*.txt*") != NULL; + bool match_log = strstr(pattern, "*.log*") != NULL; + bool match_bin = strstr(pattern, "*.bin*") != NULL; + bool match_bootlog = strstr(pattern, "bootlog") != NULL; + + while ((entry = readdir(dir)) != NULL && results->count < results->capacity) { + // Quick checks first (most common rejects) + if (entry->d_name[0] == '.') { + continue; // Skip hidden files and . / .. + } + + bool matches = false; + const char* name = entry->d_name; + size_t name_len = strlen(name); + + // Optimized pattern matching + if (match_bootlog && name_len == 7 && strcmp(name, "bootlog") == 0) { + matches = true; + } else if (name_len >= 4) { // Minimum length for extensions + // Check extensions efficiently + if (match_txt && (strcasestr(name, ".txt") != NULL)) { + matches = true; + } else if (match_log && (strcasestr(name, ".log") != NULL)) { + matches = true; + } else if (match_bin && (strcasestr(name, ".bin") != NULL)) { + matches = true; + } + } + + if (matches) { + // Use pointer arithmetic for efficiency + snprintf(results->files[results->count].filename, MAX_PATH_LEN, "%s", name); + snprintf(results->files[results->count].source_path, MAX_PATH_LEN, + "%s/%s", directory, name); + results->count++; + } + } + + closedir(dir); + return results->count; +} +``` + +## 8. Resource Management + +### 8.1 Resource Cleanup Framework +```c +typedef struct cleanup_handler { + void (*cleanup_func)(void*); + void* resource; + struct cleanup_handler* next; +} cleanup_handler_t; + +static cleanup_handler_t* g_cleanup_list = NULL; + +int register_cleanup(void (*cleanup_func)(void*), void* resource) { + cleanup_handler_t* handler = malloc(sizeof(cleanup_handler_t)); + if (!handler) { + return -1; + } + + handler->cleanup_func = cleanup_func; + handler->resource = resource; + handler->next = g_cleanup_list; + g_cleanup_list = handler; + + return 0; +} + +void execute_all_cleanup(void) { + cleanup_handler_t* current = g_cleanup_list; + while (current) { + if (current->cleanup_func && current->resource) { + current->cleanup_func(current->resource); + } + cleanup_handler_t* next = current->next; + free(current); + current = next; + } + g_cleanup_list = NULL; +} + +// Signal handler for graceful shutdown +void signal_handler(int sig) { + LOG_INFO("Received signal %d, cleaning up resources", sig); + execute_all_cleanup(); + exit(sig); +} +``` + +### 8.2 File Descriptor Management +```c +#define MAX_OPEN_FILES 64 + +static struct { + FILE* handles[MAX_OPEN_FILES]; + char paths[MAX_OPEN_FILES][MAX_PATH_LEN]; + int count; +} g_file_registry = {0}; + +FILE* managed_fopen(const char* path, const char* mode) { + if (g_file_registry.count >= MAX_OPEN_FILES) { + LOG_ERROR(NULL, ERROR_RESOURCE, "Too many open files"); + return NULL; + } + + FILE* fp = fopen(path, mode); + if (fp) { + g_file_registry.handles[g_file_registry.count] = fp; + strncpy(g_file_registry.paths[g_file_registry.count], path, MAX_PATH_LEN - 1); + g_file_registry.count++; + } + + return fp; +} + +void managed_fclose_all(void) { + for (int i = 0; i < g_file_registry.count; i++) { + if (g_file_registry.handles[i]) { + fclose(g_file_registry.handles[i]); + g_file_registry.handles[i] = NULL; + } + } + g_file_registry.count = 0; +} +``` + +## 9. Main Program Structure + +### 9.1 Main Function Implementation +```c +int main(int argc, char* argv[]) { + backup_config_t config = {0}; + error_context_t error_ctx = {0}; + int exit_code = ERROR_SUCCESS; + + // Setup signal handlers for graceful shutdown + signal(SIGINT, signal_handler); + signal(SIGTERM, signal_handler); + + do { + // Initialize all subsystems + if (logger_init("backup_logs") != 0) { + fprintf(stderr, "Failed to initialize logging system\n"); + exit_code = ERROR_SYSTEM; + break; + } + + if (config_init() != 0) { + LOG_ERROR(&error_ctx, ERROR_SYSTEM, "Failed to initialize configuration system"); + exit_code = ERROR_SYSTEM; + break; + } + + // Load and validate configuration + if (config_load(&config) != 0) { + LOG_ERROR(&error_ctx, ERROR_CONFIG_INVALID, "Failed to load configuration"); + exit_code = ERROR_CONFIG_INVALID; + break; + } + + if (config_validate(&config) != 0) { + LOG_ERROR(&error_ctx, ERROR_CONFIG_INVALID, "Configuration validation failed"); + exit_code = ERROR_CONFIG_INVALID; + break; + } + + LOG_INFO("Configuration loaded successfully"); + + // Create workspace directories + if (dir_create_workspace(&config) != 0) { + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, "Failed to create workspace directories"); + exit_code = ERROR_FILESYSTEM; + break; + } + + // Check disk threshold + if (sysint_check_disk_threshold() != 0) { + LOG_WARN("Disk threshold check failed or reported issues"); + // Continue execution - not a fatal error + } + + // Attempt recovery from any partial state + if (backup_recover_from_partial_state(&config) != 0) { + LOG_WARN("Partial state recovery had issues"); + // Continue execution + } + + // Execute appropriate backup strategy + if (config.hdd_enabled) { + LOG_INFO("Executing HDD-enabled backup strategy"); + if (backup_execute_hdd_enabled(&config) != 0) { + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, "HDD-enabled backup failed"); + exit_code = ERROR_FILESYSTEM; + break; + } + } else { + LOG_INFO("Executing HDD-disabled backup strategy"); + if (backup_execute_hdd_disabled(&config) != 0) { + LOG_ERROR(&error_ctx, ERROR_FILESYSTEM, "HDD-disabled backup failed"); + exit_code = ERROR_FILESYSTEM; + break; + } + } + + // Clean current log directory + if (dir_cleanup(config.log_path, "*.txt*,*.log*,*-*-*-*-*M-") != 0) { + LOG_WARN("Failed to clean current log directory"); + // Continue - not fatal + } + + // Copy version files + if (backup_copy_version_files(&config) != 0) { + LOG_WARN("Failed to copy some version files"); + // Continue - not fatal + } + + // Handle special log files + if (backup_handle_special_files(&config) != 0) { + LOG_WARN("Failed to handle some special log files"); + // Continue - not fatal + } + + // Create persistent marker + if (sysint_create_persistent_marker(config.persistent_path) != 0) { + LOG_WARN("Failed to create persistent marker"); + // Continue - not fatal + } + + // Send systemd notification + if (sysint_notify_systemd("Logs Backup Done..!") != 0) { + LOG_WARN("Failed to send systemd notification"); + // Continue - not fatal + } + + LOG_INFO("Backup operation completed successfully"); + + } while (0); // Single execution with break-based error handling + + // Cleanup all resources + execute_all_cleanup(); + managed_fclose_all(); + config_cleanup(&config); + logger_cleanup(); + + // Log final status + if (exit_code != ERROR_SUCCESS) { + logger_error(&error_ctx); + } + + return exit_code; +} +``` + +This LLD provides comprehensive implementation details for migrating the backup_logs.sh script to C, including detailed algorithms, data structures, error handling, and performance optimizations specifically designed for embedded RDK systems. From c652f69ec9d76a70b0a30c8cbecfe213bc56207b Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:50:09 +0530 Subject: [PATCH 8/9] Create backup_logs_flowcharts.md --- .../diagrams/backup_logs_flowcharts.md | 444 ++++++++++++++++++ 1 file changed, 444 insertions(+) create mode 100644 backup_logs/diagrams/backup_logs_flowcharts.md diff --git a/backup_logs/diagrams/backup_logs_flowcharts.md b/backup_logs/diagrams/backup_logs_flowcharts.md new file mode 100644 index 000000000..5f133c203 --- /dev/null +++ b/backup_logs/diagrams/backup_logs_flowcharts.md @@ -0,0 +1,444 @@ +# Backup Logs Migration - Flowcharts and Diagrams + +## Text-Based Flowchart Alternatives + +### 1. Main Backup Process Flow (Text Alternative) + +``` +START backup_logs + | + v +Load Configuration + | + v +Configuration Valid? --> NO --> Log Error & Exit --> END + | + v YES +Initialize Logging + | + v +Create Log Workspace + | + v +Create Previous Log Directories + | + v +Check Disk Threshold + | + v +HDD Enabled? + | + +-- YES --> Execute HDD Enabled Strategy + | | + | v + | Check for Existing Backup + | | + | v + | Backup Exists? + | | + | +-- NO --> Move All Logs to Previous --> Create Last Reboot Marker + | | + | +-- YES --> Create Timestamped Directory + | | + | v + | Move Logs to Timestamped Dir + | | + | v + | Create Last Reboot Marker + | + +-- NO --> Execute HDD Disabled Strategy + | + v + Check Backup Levels + | + v + Which Level? + | + +-- Level 0 --> Move to Previous Logs --> Create Last Reboot Marker + | + +-- Level 1 --> Move with bak1_ prefix --> Create Last Reboot Marker + | + +-- Level 2 --> Move with bak2_ prefix --> Create Last Reboot Marker + | + +-- Level 3 --> Rotate All Backup Levels --> Create Last Reboot Marker + +All paths converge to: + | + v +Clean Current Log Directory + | + v +Copy Version Files + | + v +Handle Special Log Files + | + v +Send Systemd Notification + | + v +END +``` + +### 2. HDD Disabled Strategy Detail (Text Alternative) + +``` +START HDD Disabled Strategy + | + v +Remove existing last_bootfile + | + v +Check for messages.txt in Previous Logs + | + v +messages.txt exists? + | + +-- NO --> Find all *.txt, *.log, *.bin, bootlog files + | | + | v + | Move files from LOG_PATH to PREV_LOG_PATH + | | + | v + | Create last_reboot marker --> END STRATEGY + | + +-- YES --> Check for bak1_messages.txt + | + v + bak1_messages.txt exists? + | + +-- NO --> Move current logs with bak1_ prefix --> Create last_reboot marker --> END STRATEGY + | + +-- YES --> Check for bak2_messages.txt + | + v + bak2_messages.txt exists? + | + +-- NO --> Move current logs with bak2_ prefix --> Create last_reboot marker --> END STRATEGY + | + +-- YES --> Check for bak3_messages.txt + | + v + bak3_messages.txt exists? + | + +-- NO --> Move current logs with bak3_ prefix --> Create last_reboot marker --> END STRATEGY + | + +-- YES --> Start Rotation Process + | + v + Move bak1_ files to root names + | + v + Move bak2_ files to bak1_ names + | + v + Move bak3_ files to bak2_ names + | + v + Move current logs to bak3_ names + | + v + Create last_reboot marker --> END STRATEGY +``` + +### 3. Component Interaction Sequence (Text Alternative) + +``` +Main Process -> Logger: Initialize logging system +Main Process -> Config Manager: Load configuration files +Config Manager -> Config Manager: Parse /etc/include.properties +Config Manager -> Config Manager: Parse /etc/device.properties +Config Manager -> Config Manager: Parse /etc/env_setup.sh +Config Manager --> Main Process: Configuration data + +Main Process -> Directory Manager: Create log workspace directories +Directory Manager -> File Operations: Create directory if not exists +File Operations --> Directory Manager: Directory creation status +Directory Manager --> Main Process: Workspace ready + +Main Process -> Disk Monitor: Check disk threshold +Disk Monitor -> File Operations: Execute disk_threshold_check.sh +File Operations --> Disk Monitor: Threshold check result +Disk Monitor --> Main Process: Disk status + +Main Process -> Backup Engine: Execute backup strategy + +IF HDD Enabled Device: + Backup Engine -> File Operations: Check for existing messages.txt + File Operations --> Backup Engine: File existence result + + IF No existing backup: + Backup Engine -> File Operations: Move all logs to Previous + ELSE IF Existing backup found: + Backup Engine -> Directory Manager: Create timestamped directory + Backup Engine -> File Operations: Move logs to timestamped directory + +ELSE IF HDD Disabled Device: + Backup Engine -> File Operations: Check backup levels + File Operations --> Backup Engine: Current backup level + + IF Level 0-2: + Backup Engine -> File Operations: Move with appropriate prefix + ELSE IF Level 3: + Backup Engine -> File Operations: Rotate all backup levels + +Backup Engine -> File Operations: Create last_reboot marker +File Operations --> Backup Engine: Marker creation status +Backup Engine --> Main Process: Backup operation complete + +Main Process -> File Operations: Clean current log directory +Main Process -> File Operations: Copy version files +Main Process -> File Operations: Handle special log files + +Main Process -> System Integration: Send systemd notification +System Integration --> Main Process: Notification sent + +Main Process -> Logger: Log completion status +Logger --> Main Process: Logging complete +``` + +### 4. Error Handling and Recovery Flow (Text Alternative) + +``` +Function Call + | + v +Operation Successful? + | + +-- YES --> Return Success Code --> End + | + +-- NO --> Capture Error Context + | + v + Determine Error Severity + | + v + Error Type? + | + +-- Fatal --> Log Fatal Error + | | + | v + | Cleanup Resources + | | + | v + | Send Emergency Notification + | | + | v + | Exit Process --> End + | + +-- Recoverable --> Log Warning + | | + | v + | Log Error Details + | | + | v + | Attempt Recovery + | | + | v + | Recovery Successful? + | | + | +-- YES --> Continue Operation --> Return Success Code --> End + | | + | +-- NO --> Escalate to Critical + | | + | v + | Log Critical Error (see below) + | + +-- Critical System --> Log Critical Error + | + v + Log to Syslog + | + v + Notify System Monitor + | + v + Attempt Graceful Shutdown + | + v + Return Error Code --> End +``` + +## Mermaid Diagram Sources + +### Main Backup Process Flow (Mermaid) +```mermaid +flowchart TD + A[Start backup_logs] --> B[Load Configuration] + B --> C{Configuration Valid?} + C -->|No| D[Log Error & Exit] + C -->|Yes| E[Initialize Logging] + + E --> F[Create Log Workspace] + F --> G[Create Previous Log Directories] + G --> H[Check Disk Threshold] + + H --> I{HDD Enabled?} + I -->|Yes| J[Execute HDD Enabled Strategy] + I -->|No| K[Execute HDD Disabled Strategy] + + J --> L[Check for Existing Backup] + L --> M{Backup Exists?} + M -->|No| N[Move All Logs to Previous] + M -->|Yes| O[Create Timestamped Directory] + O --> P[Move Logs to Timestamped Dir] + P --> Q[Create Last Reboot Marker] + + K --> R[Check Backup Levels] + R --> S{Which Level?} + S -->|Level 0| T[Move to Previous Logs] + S -->|Level 1| U[Move with bak1_ prefix] + S -->|Level 2| V[Move with bak2_ prefix] + S -->|Level 3| W[Rotate All Backup Levels] + + T --> X[Create Last Reboot Marker] + U --> X + V --> X + W --> X + N --> Q + Q --> X + + X --> Y[Clean Current Log Directory] + Y --> Z[Copy Version Files] + Z --> AA[Handle Special Log Files] + AA --> BB[Send Systemd Notification] + BB --> CC[End] + + D --> CC +``` + +### HDD Disabled Strategy Detail (Mermaid) +```mermaid +flowchart TD + A[Start HDD Disabled Strategy] --> B[Remove existing last_bootfile] + B --> C[Check for messages.txt in Previous Logs] + + C --> D{messages.txt exists?} + D -->|No| E[Find all *.txt, *.log, *.bin, bootlog files] + E --> F[Move files from LOG_PATH to PREV_LOG_PATH] + F --> G[Create last_reboot marker] + G --> Z[End Strategy] + + D -->|Yes| H[Check for bak1_messages.txt] + H --> I{bak1_messages.txt exists?} + I -->|No| J[Move current logs with bak1_ prefix] + J --> G + + I -->|Yes| K[Check for bak2_messages.txt] + K --> L{bak2_messages.txt exists?} + L -->|No| M[Move current logs with bak2_ prefix] + M --> G + + L -->|Yes| N[Check for bak3_messages.txt] + N --> O{bak3_messages.txt exists?} + O -->|No| P[Move current logs with bak3_ prefix] + P --> G + + O -->|Yes| Q[Start Rotation Process] + Q --> R[Move bak1_ files to root names] + R --> S[Move bak2_ files to bak1_ names] + S --> T[Move bak3_ files to bak2_ names] + T --> U[Move current logs to bak3_ names] + U --> G +``` + +### Component Interaction Sequence (Mermaid) +```mermaid +sequenceDiagram + participant Main as Main Process + participant Config as Configuration Manager + participant Dir as Directory Manager + participant Backup as Log Backup Engine + participant FileOps as File Operations Manager + participant Disk as Disk Threshold Monitor + participant SysInt as System Integration Module + participant Logger as Error Handler & Logger + + Main->>Logger: Initialize logging system + Main->>Config: Load configuration files + Config->>Config: Parse /etc/include.properties + Config->>Config: Parse /etc/device.properties + Config->>Config: Parse /etc/env_setup.sh + Config-->>Main: Configuration data + + Main->>Dir: Create log workspace directories + Dir->>FileOps: Create directory if not exists + FileOps-->>Dir: Directory creation status + Dir-->>Main: Workspace ready + + Main->>Disk: Check disk threshold + Disk->>FileOps: Execute disk_threshold_check.sh + FileOps-->>Disk: Threshold check result + Disk-->>Main: Disk status + + Main->>Backup: Execute backup strategy + + alt HDD Enabled Device + Backup->>FileOps: Check for existing messages.txt + FileOps-->>Backup: File existence result + alt No existing backup + Backup->>FileOps: Move all logs to Previous + else Existing backup found + Backup->>Dir: Create timestamped directory + Backup->>FileOps: Move logs to timestamped directory + end + else HDD Disabled Device + Backup->>FileOps: Check backup levels + FileOps-->>Backup: Current backup level + alt Level 0-2 + Backup->>FileOps: Move with appropriate prefix + else Level 3 + Backup->>FileOps: Rotate all backup levels + end + end + + Backup->>FileOps: Create last_reboot marker + FileOps-->>Backup: Marker creation status + Backup-->>Main: Backup operation complete + + Main->>FileOps: Clean current log directory + Main->>FileOps: Copy version files + Main->>FileOps: Handle special log files + + Main->>SysInt: Send systemd notification + SysInt-->>Main: Notification sent + + Main->>Logger: Log completion status + Logger-->>Main: Logging complete +``` + +### Error Handling and Recovery Flow (Mermaid) +```mermaid +flowchart TD + A[Function Call] --> B{Operation Successful?} + B -->|Yes| C[Return Success Code] + B -->|No| D[Capture Error Context] + + D --> E[Determine Error Severity] + E --> F{Error Type?} + + F -->|Fatal| G[Log Fatal Error] + F -->|Recoverable| H[Log Warning] + F -->|Critical System| I[Log Critical Error] + + G --> J[Cleanup Resources] + J --> K[Send Emergency Notification] + K --> L[Exit Process] + + H --> M[Log Error Details] + M --> N[Attempt Recovery] + N --> O{Recovery Successful?} + O -->|Yes| P[Continue Operation] + O -->|No| Q[Escalate to Critical] + Q --> I + + I --> R[Log to Syslog] + R --> S[Notify System Monitor] + S --> T[Attempt Graceful Shutdown] + T --> U[Return Error Code] + + P --> C + C --> V[End] + L --> V + U --> V +``` From a1dc9da6dec7058abc8b3762c947bf7baf36dd76 Mon Sep 17 00:00:00 2001 From: Abhinavpv28 <162570454+Abhinavpv28@users.noreply.github.com> Date: Wed, 11 Mar 2026 20:12:26 +0530 Subject: [PATCH 9/9] Update backup_logs_migration_HLD.md --- backup_logs/backup_logs_migration_HLD.md | 1 - 1 file changed, 1 deletion(-) diff --git a/backup_logs/backup_logs_migration_HLD.md b/backup_logs/backup_logs_migration_HLD.md index 944537b7c..99df069d2 100644 --- a/backup_logs/backup_logs_migration_HLD.md +++ b/backup_logs/backup_logs_migration_HLD.md @@ -42,7 +42,6 @@ backup_logs (main executable) - **Responsibilities**: - Parse `/etc/include.properties` - Parse `/etc/device.properties` - - Parse `/etc/env_setup.sh` if available - Validate configuration parameters - Provide configuration data to other modules