Concorde

Concorde is an offline-first sync platform for Rails 8 applications. It provides automatic data synchronization between clients and servers using CRDT (Conflict-free Replicated Data Types) principles, leveraging Rails 8's Solid Stack (SolidCable and SolidQueue) for real-time updates and background processing.

Features

• Offline-First: Full functionality even without internet connection
• Automatic Sync: Changes sync automatically when connection is restored
• Conflict Resolution: Built-in CRDT-based conflict resolution (Last Write Wins by default)
• Real-Time Updates: Uses SolidCable for instant sync notifications
• Easy Integration: Simple setup with Rails models using include Concorde::Syncable
• Client SDK: JavaScript SDK with IndexedDB storage for browser apps
• Multi-Tab Support: Synchronization across browser tabs
• Minimal Dependencies: Uses Rails 8's built-in Solid Stack (no Redis required)

Installation

Add this line to your application's Gemfile:

gem "concorde"

Then execute:

$ bundle install
$ rails generate concorde:install
$ rails db:migrate

The installer will:

• Create an initializer at config/initializers/concorde.rb
• Copy migrations for sync tables
• Mount the engine in your routes
• Configure SolidCable for real-time sync
• Optionally create an example Todo model with sync enabled

Usage

1. Enable Sync on Your Models

Add the Syncable concern to any model you want to sync:

class Task < ApplicationRecord
  include Concorde::Syncable
  
  # Optionally exclude certain attributes from sync
  sync_exclude :internal_notes, :admin_only_field
end

2. Initialize the Client SDK

In your JavaScript:

// Initialize Concorde
const concorde = new Concorde({
  apiUrl: '/concorde/sync',
  clientId: localStorage.getItem('concorde_client_id') || undefined,
  userId: currentUserId
});

// Save client ID for future sessions
localStorage.setItem('concorde_client_id', concorde.config.clientId);

// Start syncing
await concorde.start();

3. Use the Offline-First API

// Create a record (works offline)
const task = await concorde.create('Task', {
  title: 'Buy milk',
  completed: false
});

// Update a record
await concorde.update('Task', task.id, { completed: true });

// Delete a record
await concorde.delete('Task', task.id);

// Query records
const allTasks = await concorde.all('Task');
const pendingTasks = await concorde.where('Task', { completed: false });
const specificTask = await concorde.find('Task', taskId);

// Listen for changes
concorde.on('change', (event) => {
  console.log('Data changed:', event);
  refreshUI();
});

// Check sync status
console.log('Is online:', concorde.isOnline);
console.log('Is syncing:', concorde.isSyncing);
console.log('Pending changes:', concorde.pendingChanges);

Configuration

Configure Concorde in config/initializers/concorde.rb:

Concorde.configure do |config|
  # Models to enable for sync
  config.sync_models = [Task, Project, User]
  
  # Custom conflict resolution
  config.conflict_resolution_rules['Task'] = {
    'priority' => ->(old_val, new_val) { [old_val, new_val].max },
    'title' => ->(old_val, new_val) { new_val.length > old_val.length ? new_val : old_val }
  }
  
  # Authentication/Authorization
  config.authorization_adapter = ->(request) {
    request.env['warden'].authenticated?
  }
  
  # Callbacks
  config.on_conflict = ->(change_log) {
    Rails.logger.info "Conflict resolved: #{change_log.inspect}"
  }
  
  config.on_push_receive = ->(mutation, user) {
    # Validate or authorize mutations
    raise "Unauthorized" unless user.can_edit?(mutation['record_type'])
  }
end

How It Works

1. Local Changes: When offline, all changes are stored in IndexedDB and applied optimistically to the UI
2. Change Log: The server maintains an append-only log of all mutations
3. Sync Protocol:
   • Push: Client sends pending mutations to server
   • Pull: Client fetches changes since last sync
4. Conflict Resolution: When concurrent changes occur, they're resolved using CRDT rules
5. Real-Time: SolidCable broadcasts notifications when data changes

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Browser   │     │   Browser   │     │   Mobile    │
│  (Client 1) │     │  (Client 2) │     │   Client    │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                    │
       │     Push/Pull     │      WebSocket    │
       └───────────────────┴────────────────────┘
                           │
                    ┌──────┴──────┐
                    │   Rails 8   │
                    │   Server    │
                    │             │
                    │ ┌─────────┐ │
                    │ │Concorde │ │
                    │ │ Engine  │ │
                    │ └─────────┘ │
                    └──────┬──────┘
                           │
                    ┌──────┴──────┐
                    │  PostgreSQL │
                    │      or     │
                    │    MySQL    │
                    └─────────────┘

Advanced Features

Custom Conflict Resolution

# In your model
class Document < ApplicationRecord
  include Concorde::Syncable
  
  sync_conflict_strategy :custom_merge
end

# In config/initializers/concorde.rb
Concorde.configure do |config|
  config.conflict_resolution_rules['Document'] = {
    'content' => ->(old_content, new_content) {
      # Custom merge logic (e.g., three-way merge for text)
      merge_text(old_content, new_content)
    }
  }
end

Server-Side Changes

When making changes on the server that should sync to clients:

# Changes are automatically tracked
task = Task.create!(title: "Server-created task")

# For bulk operations, you can disable sync temporarily
Task.without_sync do
  Task.import(large_dataset)
end

Multi-User Sync

// Subscribe to specific data streams
const concorde = new Concorde({
  userId: currentUser.id,
  channels: ['personal', `team_${teamId}`]
});

Testing

Concorde includes test helpers for your application tests:

# In your test
class TaskSyncTest < ActionDispatch::IntegrationTest
  include Concorde::TestHelper
  
  test "syncs task between clients" do
    # Simulate offline client
    with_offline_client do |client|
      client.create('Task', title: 'Offline task')
    end
    
    # Sync and verify
    sync_all_clients
    assert_equal 1, Task.count
  end
end

Production Considerations

1. Database Indexes: Ensure proper indexes on high-traffic tables
2. Change Log Pruning: Set up a job to archive old change logs
3. Security: Implement proper authentication and authorization
4. Monitoring: Monitor sync performance and conflicts

Troubleshooting

Client not syncing

• Check browser console for errors
• Verify WebSocket connection is established
• Ensure client ID is being sent with requests

Conflicts occurring frequently

• Review your conflict resolution rules
• Consider using field-level timestamps
• Implement custom merge strategies for complex data

Performance issues

• Enable change log pruning
• Add database indexes for your sync queries
• Consider pagination for large datasets

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/felixclack/concorde.

License

The gem is available as open source under the terms of the MIT License.