Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .claude/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
"Bash(docker compose:*)",
"Bash(docker exec:*)",
"Bash(gh run view:*)",
"WebFetch(domain:api.codecov.io)"
"WebFetch(domain:api.codecov.io)",
"WebSearch"
],
"deny": []
}
Expand Down
27 changes: 20 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,17 +88,17 @@ SharpSync is a **pure .NET file synchronization library** with no native depende

1. **Core Interfaces** (`src/SharpSync/Core/`)
- `ISyncEngine` - Main synchronization orchestrator (`SynchronizeAsync`, `PreviewSyncAsync`, `GetSyncPlanAsync`, `GetStatsAsync`, `ResetSyncStateAsync`, plus selective/incremental sync and lifecycle methods)
- `ISyncStorage` - Storage backend abstraction (local, WebDAV, cloud) with `ProgressChanged` event
- `ISyncStorage` - Storage backend abstraction (local, WebDAV, cloud) with `ProgressChanged` event and default methods for `SetLastModifiedAsync`/`SetPermissionsAsync`
- `ISyncDatabase` - Sync state persistence
- `IConflictResolver` - Pluggable conflict resolution strategies
- `ISyncFilter` - File filtering for selective sync
- Domain models: `SyncItem`, `SyncOptions`, `SyncProgress`, `SyncResult`
- Domain models: `SyncItem` (with `IsSymlink` support), `SyncOptions`, `SyncProgress`, `SyncResult`

2. **Storage Implementations** (`src/SharpSync/Storage/`)
- `LocalFileStorage` - Local filesystem operations (fully implemented and tested)
- `LocalFileStorage` - Local filesystem operations with symlink detection, timestamp/permission preservation (fully implemented and tested)
- `WebDavStorage` - WebDAV with OAuth2, chunking, and platform-specific optimizations (fully implemented and tested)
- `SftpStorage` - SFTP with password and key-based authentication (fully implemented and tested)
- `FtpStorage` - FTP/FTPS with secure connections support (fully implemented and tested)
- `SftpStorage` - SFTP with password and key-based authentication, symlink detection, timestamp/permission preservation (fully implemented and tested)
- `FtpStorage` - FTP/FTPS with secure connections support and timestamp preservation (fully implemented and tested)
- `S3Storage` - Amazon S3 and S3-compatible storage (MinIO, LocalStack) with multipart uploads (fully implemented and tested)

Additional public types in `src/SharpSync/Storage/`:
Expand All @@ -119,9 +119,10 @@ SharpSync is a **pure .NET file synchronization library** with no native depende

5. **Synchronization Engine** (`src/SharpSync/Sync/`)
- `SyncEngine` - Production-ready sync implementation with:
- Incremental sync with change detection
- Incremental sync with change detection (timestamp, checksum-only, or size-only modes)
- Parallel processing for large file sets
- Three-phase optimization (directories/small files, large files, deletes/conflicts)
- All `SyncOptions` properties fully wired: `TimeoutSeconds`, `ChecksumOnly`, `SizeOnly`, `UpdateExisting`, `ConflictResolution` override, `ExcludePatterns`, `Verbose`, `FollowSymlinks`, `PreserveTimestamps`, `PreservePermissions`
- `SyncFilter` - Pattern-based file filtering

Internal sync pipeline types (in `Oire.SharpSync.Sync` namespace):
Expand Down Expand Up @@ -151,7 +152,12 @@ SharpSync is a **pure .NET file synchronization library** with no native depende
- **Parallel Processing**: Configurable parallelism with intelligent prioritization
- **Bandwidth Throttling**: Configurable transfer rate limits via `SyncOptions.MaxBytesPerSecond`
- **Virtual File Support**: Callback hook for Windows Cloud Files API placeholder integration
- **Structured Logging**: High-performance logging via `Microsoft.Extensions.Logging`
- **Timestamp Preservation**: Optionally preserves file modification times across sync (`PreserveTimestamps`)
- **Permission Preservation**: Optionally preserves Unix file permissions across sync (`PreservePermissions`)
- **Symlink Awareness**: Detects symlinks and optionally follows or skips them (`FollowSymlinks`)
- **Flexible Change Detection**: Checksum-only or size-only modes for change detection
- **Per-Sync Exclusion**: Runtime exclude patterns via `SyncOptions.ExcludePatterns`
- **Structured Logging**: High-performance logging via `Microsoft.Extensions.Logging` with verbose mode

### Dependencies

Expand Down Expand Up @@ -391,6 +397,10 @@ var deleted = await engine.ClearOperationHistoryAsync(DateTime.UtcNow.AddDays(-3
| Activity history | `GetRecentOperationsAsync()` - query completed operations for activity feed |
| History cleanup | `ClearOperationHistoryAsync()` - purge old operation records |
| Per-file progress | `FileProgressChanged` event on `ISyncEngine` - byte-level progress for individual file transfers |
| SyncOptions wiring | All `SyncOptions` properties are now functional: `TimeoutSeconds`, `ChecksumOnly`, `SizeOnly`, `UpdateExisting`, `ConflictResolution` override, `ExcludePatterns`, `Verbose`, `FollowSymlinks`, `PreserveTimestamps`, `PreservePermissions` |
| Timestamp preservation | `ISyncStorage.SetLastModifiedAsync` default interface method, implemented in Local/SFTP/FTP storage |
| Permission preservation | `ISyncStorage.SetPermissionsAsync` default interface method, implemented in Local/SFTP storage |
| Symlink awareness | `SyncItem.IsSymlink` property, detected in Local/SFTP storage, `FollowSymlinks` option in SyncEngine |

### Required SharpSync API Additions (v1.0)

Expand Down Expand Up @@ -520,3 +530,6 @@ All critical items have been resolved.
- ✅ Examples directory with working samples
- ✅ Code coverage reporting (Coverlet + Codecov with badge in README)
- ✅ Console OAuth2 provider example (`examples/ConsoleOAuth2Example.cs`)
- ✅ All `SyncOptions` properties wired and functional (TimeoutSeconds, ChecksumOnly, SizeOnly, UpdateExisting, ConflictResolution override, ExcludePatterns, Verbose, FollowSymlinks, PreserveTimestamps, PreservePermissions)
- ✅ `ISyncStorage.SetLastModifiedAsync` / `SetPermissionsAsync` default interface methods
- ✅ Symlink detection (`SyncItem.IsSymlink`) in Local and SFTP storage
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,8 +193,7 @@ var storage = new FtpStorage(
host: "ftp.example.com",
username: "user",
password: "password",
useSsl: true,
sslMode: FtpSslMode.Explicit
useFtps: true
);

// Implicit FTPS
Expand All @@ -203,8 +202,8 @@ var storage = new FtpStorage(
port: 990,
username: "user",
password: "password",
useSsl: true,
sslMode: FtpSslMode.Implicit
useFtps: true,
useImplicitFtps: true
);
```

Expand Down
19 changes: 7 additions & 12 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,13 +91,13 @@ export SFTP_TEST_ROOT=/home/testuser/upload
dotnet test --verbosity normal
```

### WebDAV Integration Tests (Future)
### WebDAV Integration Tests

WebDAV integration tests are planned for future releases. They will follow a similar pattern:
WebDAV integration tests verify connectivity, file operations, and server capability detection against a real WebDAV server. The tests are located in `tests/SharpSync.Tests/Storage/WebDavStorageTests.cs`.

```bash
# Using Docker Compose
docker-compose -f docker-compose.test.yml up -d webdav
# Using Docker Compose (starts the WebDAV server along with other test services)
docker-compose -f docker-compose.test.yml up -d

export WEBDAV_TEST_URL=http://localhost:8080/webdav
export WEBDAV_TEST_USER=testuser
Expand All @@ -106,6 +106,8 @@ export WEBDAV_TEST_PASS=testpass
dotnet test --verbosity normal
```

Like other integration tests, WebDAV tests skip automatically when the required environment variables are not set.

## Continuous Integration

The GitHub Actions workflow automatically runs all tests, including SFTP integration tests, using a Docker-based SFTP server. See `.github/workflows/dotnet.yml` for the configuration.
Expand Down Expand Up @@ -220,14 +222,7 @@ public async Task MyNewIntegrationTest() {

## Performance Testing

For performance benchmarks, use BenchmarkDotNet:

```bash
# Run in Release mode
dotnet run -c Release --project benchmarks/SharpSync.Benchmarks
```

(Note: Benchmarks project not yet implemented)
Performance benchmarks using BenchmarkDotNet are planned for a future release.

## Additional Resources

Expand Down
71 changes: 70 additions & 1 deletion examples/BasicSyncExample.cs
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ public static async Task BasicSyncAsync() {
filter.AddExcludePattern("node_modules/**");

// 4. Create conflict resolver
var conflictResolver = new DefaultConflictResolver(ConflictResolution.UseNewer);
var conflictResolver = new DefaultConflictResolver(ConflictResolution.UseRemote);

// 5. Create sync engine
using var syncEngine = new SyncEngine(
Expand Down Expand Up @@ -214,6 +214,75 @@ public static async Task ThrottledSyncAsync(ISyncEngine syncEngine) {
Console.WriteLine($"Throttled sync completed: {result.FilesSynchronized} files");
}

/// <summary>
/// Configure sync options for fine-grained control over synchronization behavior.
/// </summary>
public static async Task SyncWithOptionsAsync(ISyncEngine syncEngine) {
// Checksum-only mode: detect changes by file hash instead of timestamps.
// Useful when timestamps are unreliable (e.g., after restoring from backup).
var checksumOptions = new SyncOptions { ChecksumOnly = true };
await syncEngine.SynchronizeAsync(checksumOptions);

// Size-only mode: detect changes by file size only (fastest, least accurate).
// Good for quick checks when only large content changes matter.
var sizeOptions = new SyncOptions { SizeOnly = true };
await syncEngine.SynchronizeAsync(sizeOptions);

// Preserve timestamps and permissions across sync.
// Timestamps are set on the target after each file transfer.
// Permissions (Unix only) are preserved for Local and SFTP storage.
var preserveOptions = new SyncOptions {
PreserveTimestamps = true,
PreservePermissions = true
};
await syncEngine.SynchronizeAsync(preserveOptions);

// Skip symlink directories during sync.
// When false (default), symlink directories are not followed.
var symlinkOptions = new SyncOptions { FollowSymlinks = true };
await syncEngine.SynchronizeAsync(symlinkOptions);

// Per-sync exclude patterns (applied in addition to the engine-level SyncFilter).
// Useful for one-off syncs that need extra filtering without modifying the filter.
var excludeOptions = new SyncOptions {
ExcludePatterns = new List<string> { "*.bak", "thumbs.db", "*.tmp" }
};
await syncEngine.SynchronizeAsync(excludeOptions);

// Timeout: cancel sync if it exceeds the given number of seconds.
var timeoutOptions = new SyncOptions { TimeoutSeconds = 300 };
await syncEngine.SynchronizeAsync(timeoutOptions);

// UpdateExisting=false: only sync new files, skip modifications to existing files.
var newOnlyOptions = new SyncOptions { UpdateExisting = false };
await syncEngine.SynchronizeAsync(newOnlyOptions);

// Override conflict resolution per-sync via options.
// This takes priority over the IConflictResolver passed to the engine constructor.
// Set to ConflictResolution.Ask to delegate to the resolver instead.
var conflictOptions = new SyncOptions {
ConflictResolution = ConflictResolution.UseLocal
};
await syncEngine.SynchronizeAsync(conflictOptions);

// Verbose logging: emits detailed Debug-level log messages for change detection,
// action processing, and phase completion. Requires an ILogger<SyncEngine> to be
// passed to the SyncEngine constructor.
var verboseOptions = new SyncOptions { Verbose = true };
await syncEngine.SynchronizeAsync(verboseOptions);

// Options can be combined freely.
var combinedOptions = new SyncOptions {
ChecksumOnly = true,
PreserveTimestamps = true,
ExcludePatterns = new List<string> { "*.log" },
TimeoutSeconds = 600,
Verbose = true
};
var result = await syncEngine.SynchronizeAsync(combinedOptions);
Console.WriteLine($"Combined sync completed: {result.FilesSynchronized} files");
}

/// <summary>
/// Smart conflict resolution with UI callback.
/// </summary>
Expand Down
3 changes: 2 additions & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ A comprehensive example showing:
- **Selective sync** - Syncing specific files or folders on demand
- **Pause/Resume** - Controlling long-running sync operations
- **Bandwidth throttling** - Limiting transfer speeds
- **Sync options** - Configuring `ChecksumOnly`, `SizeOnly`, `PreserveTimestamps`, `PreservePermissions`, `FollowSymlinks`, `ExcludePatterns`, `TimeoutSeconds`, `UpdateExisting`, `ConflictResolution` override, and `Verbose` logging
- **Smart conflict resolution** - Handling conflicts with UI prompts

## ConsoleOAuth2Example.cs
Expand Down Expand Up @@ -72,7 +73,7 @@ await database.InitializeAsync();

// Create sync engine
var filter = new SyncFilter();
var resolver = new DefaultConflictResolver(ConflictResolution.UseNewer);
var resolver = new DefaultConflictResolver(ConflictResolution.UseRemote);
using var engine = new SyncEngine(localStorage, remoteStorage, database, filter, resolver);

// Run sync
Expand Down
26 changes: 26 additions & 0 deletions src/SharpSync/Core/ISyncStorage.cs
Original file line number Diff line number Diff line change
Expand Up @@ -79,4 +79,30 @@ public interface ISyncStorage {
/// Tests connection to the storage
/// </summary>
Task<bool> TestConnectionAsync(CancellationToken cancellationToken = default);

/// <summary>
/// Sets the last modified time for a file or directory.
/// </summary>
/// <remarks>
/// Not all storage backends support setting modification times. The default implementation
/// is a no-op. Implementations that support this (e.g., local filesystem, SFTP, FTP)
/// should override this method.
/// </remarks>
/// <param name="path">The relative path to the item</param>
/// <param name="lastModified">The last modified time to set (UTC)</param>
/// <param name="cancellationToken">Cancellation token to cancel the operation</param>
Task SetLastModifiedAsync(string path, DateTime lastModified, CancellationToken cancellationToken = default) => Task.CompletedTask;

/// <summary>
/// Sets file permissions for a file or directory.
/// </summary>
/// <remarks>
/// Not all storage backends or platforms support setting file permissions.
/// The default implementation is a no-op. Implementations that support this
/// (e.g., local filesystem on Unix, SFTP) should override this method.
/// </remarks>
/// <param name="path">The relative path to the item</param>
/// <param name="permissions">The permissions string (e.g., "rwxr-xr-x" or "755")</param>
/// <param name="cancellationToken">Cancellation token to cancel the operation</param>
Task SetPermissionsAsync(string path, string permissions, CancellationToken cancellationToken = default) => Task.CompletedTask;
}
5 changes: 5 additions & 0 deletions src/SharpSync/Core/SyncItem.cs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ public class SyncItem {
/// </summary>
public string? ETag { get; set; }

/// <summary>
/// Gets or sets whether this item is a symbolic link
/// </summary>
public bool IsSymlink { get; set; }

/// <summary>
/// Gets or sets additional metadata
/// </summary>
Expand Down
42 changes: 42 additions & 0 deletions src/SharpSync/Logging/LogMessages.cs
Original file line number Diff line number Diff line change
Expand Up @@ -77,4 +77,46 @@ internal static partial class LogMessages {
Level = LogLevel.Warning,
Message = "Failed to log operation for {Path}")]
public static partial void OperationLoggingError(this ILogger logger, Exception ex, string path);

[LoggerMessage(
EventId = 13,
Level = LogLevel.Debug,
Message = "Detecting changes (options: DryRun={DryRun}, ChecksumOnly={ChecksumOnly}, SizeOnly={SizeOnly})")]
public static partial void DetectChangesStart(this ILogger logger, bool dryRun, bool checksumOnly, bool sizeOnly);

[LoggerMessage(
EventId = 14,
Level = LogLevel.Debug,
Message = "Change detection complete: {Additions} additions, {Modifications} modifications, {Deletions} deletions")]
public static partial void DetectChangesComplete(this ILogger logger, int additions, int modifications, int deletions);

[LoggerMessage(
EventId = 15,
Level = LogLevel.Debug,
Message = "HasChanged {Path}: isLocal={IsLocal}, result={Changed}")]
public static partial void HasChangedResult(this ILogger logger, string path, bool isLocal, bool changed);

[LoggerMessage(
EventId = 16,
Level = LogLevel.Debug,
Message = "Processing action: {ActionType} {Path}")]
public static partial void ProcessingAction(this ILogger logger, Core.SyncActionType actionType, string path);

[LoggerMessage(
EventId = 17,
Level = LogLevel.Debug,
Message = "Phase {Phase} complete: processed {Count} actions")]
public static partial void PhaseComplete(this ILogger logger, int phase, int count);

[LoggerMessage(
EventId = 18,
Level = LogLevel.Warning,
Message = "Failed to preserve timestamps for {Path}")]
public static partial void TimestampPreservationError(this ILogger logger, Exception ex, string path);

[LoggerMessage(
EventId = 19,
Level = LogLevel.Warning,
Message = "Failed to preserve permissions for {Path}")]
public static partial void PermissionPreservationError(this ILogger logger, Exception ex, string path);
}
13 changes: 13 additions & 0 deletions src/SharpSync/Storage/FtpStorage.cs
Original file line number Diff line number Diff line change
Expand Up @@ -473,6 +473,19 @@ public async Task<string> ComputeHashAsync(string path, CancellationToken cancel
return Convert.ToBase64String(hashBytes);
}

/// <summary>
/// Sets the last modified time for a file on the FTP server
/// </summary>
public async Task SetLastModifiedAsync(string path, DateTime lastModified, CancellationToken cancellationToken = default) {
await EnsureConnectedAsync(cancellationToken);
var fullPath = GetFullPath(path);

await ExecuteWithRetry(async () => {
await _client!.SetModifiedTime(fullPath, lastModified, cancellationToken);
return true;
}, cancellationToken);
}

#region Helper Methods

/// <summary>
Expand Down
Loading