Skip to content

feat: implement time-based offset seeking across broker and client APIs#29

Merged
samuel025 merged 1 commit into
mainfrom
time-seeking
Jul 14, 2026
Merged

feat: implement time-based offset seeking across broker and client APIs#29
samuel025 merged 1 commit into
mainfrom
time-seeking

Conversation

@samuel025

@samuel025 samuel025 commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Summary by CodeRabbit

  • New Features

    • Added time-based seeking for Java, Python, and TypeScript consumers.
    • Consumers can now start reading from the first message at or after a specified timestamp.
    • Added CLI support for seektime with Unix millisecond and ISO-8601 timestamps.
    • Added broker support for timestamp-based offset lookup, including retry and leader redirection handling.
  • Bug Fixes

    • Improved Python consumer error handling by reporting broker consume failures immediately.

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds timestamp-based offset lookup to the protocol and broker, exposes seekByTime/seek_by_time APIs across clients, adds CLI support, and verifies seeking behavior with a Raft integration test.

Changes

Timestamp-based offset seeking

Layer / File(s) Summary
Timestamp lookup protocol contracts
drmq-protocol/src/main/proto/messages.proto, drmq-ts-client/src/messages.ts, drmq-python-client/messages_pb2.py
Adds request/response message types and generated bindings for timestamp-to-offset lookup.
Broker timestamp lookup pipeline
drmq-broker/src/main/java/com/drmq/broker/persistence/*, drmq-broker/src/main/java/com/drmq/broker/MessageStore.java, drmq-broker/src/main/java/com/drmq/broker/ClientHandler.java
Scans log records for the first timestamp-matching offset and returns it through the new broker request handler.
Consumer seek APIs and retry handling
drmq-client/src/main/java/com/drmq/client/DRMQConsumer.java, drmq-python-client/drmq_client.py, drmq-ts-client/src/client.ts
Adds timestamp-seeking APIs that update local offsets, commit in group mode, redirect leaders, and retry connection failures. Python polling now raises connection errors for unsuccessful consume responses.
CLI command and integration validation
drmq-client/src/main/java/com/drmq/client/commandLineExample/ConsumerApp.java, drmq-integration-tests/src/test/java/com/drmq/integration/RaftIntegrationTest.java
Adds the seektime command with numeric and ISO-8601 timestamps and tests seeking from a follower node.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Consumer
  participant Broker
  participant LogStore
  participant ConsumerGroup
  Consumer->>Broker: Request offset for topic and timestamp
  Broker->>LogStore: Search records for timestamp
  LogStore-->>Broker: Return first matching offset
  Broker-->>Consumer: Return timestamp offset
  Consumer->>ConsumerGroup: Commit offset when group mode is enabled
  Consumer->>Consumer: Update local topic offset
Loading

Possibly related PRs

  • samuel025/DRMQ#19: Refactors log-segment structures used by the timestamp search implementation.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the core change: adding time-based offset seeking across broker and client APIs.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch time-seeking

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@samuel025

Copy link
Copy Markdown
Owner Author

@coderabbitai help

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

CodeRabbit commands

These commands are invoked using PR/Issue comments.

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai rate limit to show your current review rate limit status.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai evaluate custom pre-merge check --instructions <custom-checks-instructions> --name <custom-checks-title> [--mode <error|warning>] to validate the custom pre-merge checks instructions. Defaults to error when --mode is omitted.
  • @coderabbitai ignore pre-merge checks to override pre-merge checks and get an approval on PR.
  • @coderabbitai run pre-merge checks to run pre-merge checks on the pull request.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai emit path instructions to emit generated path-specific review instructions for this repository.
  • @coderabbitai resolve merge conflict to automatically resolve merge conflicts.
  • @coderabbitai autofix to automatically fix issues identified in unresolved review comments.
  • @coderabbitai resolve to resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai generate configuration to create a PR that adds the current resolved configuration as .coderabbit.yaml (or show it if already present).
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit configuration file (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, support, documentation and community

  • Visit our status page to check the current availability of CodeRabbit.
  • Create a ticket on our support page for assistance with any issues or questions.
  • Visit our documentation site for detailed information on how to use CodeRabbit.
  • Join our Discord community to connect with other users and get help from the community.
  • Follow us on X/Twitter for updates and announcements.

@samuel025

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
drmq-python-client/drmq_client.py (1)

384-386: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Stray print() alongside the raised exception.

The rest of the file logs via logger.warning/logger.error; this new branch uses a raw print(...) in addition to raising. Prefer logger.error(...) for consistency and to avoid unstructured output to stdout.

♻️ Proposed fix
                     else:
-                        print(f"POLL ERROR from broker: {resp.error_message}")
+                        logger.error(f"Poll failed: {resp.error_message}")
                         raise DRMQConnectionError(f"Poll failed: {resp.error_message}")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@drmq-python-client/drmq_client.py` around lines 384 - 386, Replace the raw
print in the poll error branch with logger.error, preserving the existing broker
error message and DRMQConnectionError raise in the surrounding polling logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@drmq-broker/src/main/java/com/drmq/broker/ClientHandler.java`:
- Around line 484-498: Update handleSearchOffsetByTimeRequest to match the
structural error handling and metrics telemetry used by handlers such as
handleFetchOffsetRequest: wrap payload parsing and offset lookup in the
established try-catch flow, record the corresponding request success/failure
metrics, and return the standard well-formed error response for malformed
payloads or runtime exceptions instead of propagating them to the channel.

In `@drmq-broker/src/main/java/com/drmq/broker/persistence/LogManager.java`:
- Around line 122-141: Update LogManager.findOffsetByTimestamp to avoid calling
LogSegment.findOffsetByTimestamp across every segment. Inspect each segment’s
first-message timestamp to identify the segment whose range bounds
targetTimestamp, then invoke findOffsetByTimestamp only on that segment;
preserve -1 for missing or out-of-range topics.

In `@drmq-broker/src/main/java/com/drmq/broker/persistence/LogSegment.java`:
- Around line 228-250: Remove synchronized from LogSegment.findOffsetByTimestamp
so the linear scan does not block append or appendBatch. Capture the volatile
currentSize once at method entry and use that local snapshot as the scan’s upper
bound, while preserving the existing read, corruption handling, and
offset-search behavior.

In `@drmq-client/src/main/java/com/drmq/client/DRMQConsumer.java`:
- Around line 318-333: The seekByTime method must register the topic even when
doSearchOffsetByTime returns no matching offset. Preserve a valid found offset,
but on failure add the topic using its prior offset when available or the
default offset 0, so doPoll continues to process it consistently with subscribe
and the other clients.

---

Nitpick comments:
In `@drmq-python-client/drmq_client.py`:
- Around line 384-386: Replace the raw print in the poll error branch with
logger.error, preserving the existing broker error message and
DRMQConnectionError raise in the surrounding polling logic.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e84fe451-b3e2-49d2-bb65-9b56fd579ebf

📥 Commits

Reviewing files that changed from the base of the PR and between 5d0a628 and 66718f6.

📒 Files selected for processing (12)
  • drmq-broker/src/main/java/com/drmq/broker/ClientHandler.java
  • drmq-broker/src/main/java/com/drmq/broker/MessageStore.java
  • drmq-broker/src/main/java/com/drmq/broker/persistence/LogManager.java
  • drmq-broker/src/main/java/com/drmq/broker/persistence/LogSegment.java
  • drmq-client/src/main/java/com/drmq/client/DRMQConsumer.java
  • drmq-client/src/main/java/com/drmq/client/commandLineExample/ConsumerApp.java
  • drmq-integration-tests/src/test/java/com/drmq/integration/RaftIntegrationTest.java
  • drmq-protocol/src/main/proto/messages.proto
  • drmq-python-client/drmq_client.py
  • drmq-python-client/messages_pb2.py
  • drmq-ts-client/src/client.ts
  • drmq-ts-client/src/messages.ts

Comment on lines +484 to +498
private MessageEnvelope handleSearchOffsetByTimeRequest(MessageEnvelope envelope) throws IOException {
SearchOffsetByTimeRequest request = SearchOffsetByTimeRequest.parseFrom(envelope.getPayload());

long offset = messageStore.findOffsetByTimestamp(request.getTopic(), request.getTimestamp());

SearchOffsetByTimeResponse.Builder response = SearchOffsetByTimeResponse.newBuilder()
.setSuccess(true)
.setOffset(offset);

return MessageEnvelope.newBuilder()
.setType(MessageType.SEARCH_OFFSET_BY_TIME_RESPONSE)
.setPayload(response.build().toByteString())
.build();
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add structural error handling and metrics telemetry.

Unlike other request handlers in this class (e.g., handleFetchOffsetRequest), this method lacks a try-catch block to handle malformed payloads or runtime exceptions and does not record metrics. If parseFrom fails, the exception will propagate up to the channel, disconnecting the client instead of returning a well-formed error response.

🛠️ Proposed fix to align with existing request handlers
-    private MessageEnvelope handleSearchOffsetByTimeRequest(MessageEnvelope envelope) throws IOException {
-        SearchOffsetByTimeRequest request = SearchOffsetByTimeRequest.parseFrom(envelope.getPayload());
-        
-        long offset = messageStore.findOffsetByTimestamp(request.getTopic(), request.getTimestamp());
-        
-        SearchOffsetByTimeResponse.Builder response = SearchOffsetByTimeResponse.newBuilder()
-                .setSuccess(true)
-                .setOffset(offset);
-                
-        return MessageEnvelope.newBuilder()
-                .setType(MessageType.SEARCH_OFFSET_BY_TIME_RESPONSE)
-                .setPayload(response.build().toByteString())
-                .build();
-    }
+    private MessageEnvelope handleSearchOffsetByTimeRequest(MessageEnvelope envelope) throws IOException {
+        long startNanos = System.nanoTime();
+        try {
+            SearchOffsetByTimeRequest request = SearchOffsetByTimeRequest.parseFrom(envelope.getPayload());
+            
+            long offset = messageStore.findOffsetByTimestamp(request.getTopic(), request.getTimestamp());
+            
+            SearchOffsetByTimeResponse.Builder response = SearchOffsetByTimeResponse.newBuilder()
+                    .setSuccess(true)
+                    .setOffset(offset);
+                    
+            BrokerMetrics.get().recordRequest("search_offset_by_time", true,
+                    System.nanoTime() - startNanos, 0, 0);
+
+            return MessageEnvelope.newBuilder()
+                    .setType(MessageType.SEARCH_OFFSET_BY_TIME_RESPONSE)
+                    .setPayload(response.build().toByteString())
+                    .build();
+        } catch (Exception e) {
+            logger.error("Error searching offset by time", e);
+            BrokerMetrics.get().recordRequest("search_offset_by_time", false,
+                    System.nanoTime() - startNanos, 0, 0);
+            SearchOffsetByTimeResponse response = SearchOffsetByTimeResponse.newBuilder()
+                    .setSuccess(false)
+                    .setOffset(-1)
+                    .setErrorMessage(e.getMessage() != null ? e.getMessage() : "Unknown error")
+                    .build();
+            return MessageEnvelope.newBuilder()
+                    .setType(MessageType.SEARCH_OFFSET_BY_TIME_RESPONSE)
+                    .setPayload(response.toByteString())
+                    .build();
+        }
+    }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
private MessageEnvelope handleSearchOffsetByTimeRequest(MessageEnvelope envelope) throws IOException {
SearchOffsetByTimeRequest request = SearchOffsetByTimeRequest.parseFrom(envelope.getPayload());
long offset = messageStore.findOffsetByTimestamp(request.getTopic(), request.getTimestamp());
SearchOffsetByTimeResponse.Builder response = SearchOffsetByTimeResponse.newBuilder()
.setSuccess(true)
.setOffset(offset);
return MessageEnvelope.newBuilder()
.setType(MessageType.SEARCH_OFFSET_BY_TIME_RESPONSE)
.setPayload(response.build().toByteString())
.build();
}
private MessageEnvelope handleSearchOffsetByTimeRequest(MessageEnvelope envelope) throws IOException {
long startNanos = System.nanoTime();
try {
SearchOffsetByTimeRequest request = SearchOffsetByTimeRequest.parseFrom(envelope.getPayload());
long offset = messageStore.findOffsetByTimestamp(request.getTopic(), request.getTimestamp());
SearchOffsetByTimeResponse.Builder response = SearchOffsetByTimeResponse.newBuilder()
.setSuccess(true)
.setOffset(offset);
BrokerMetrics.get().recordRequest("search_offset_by_time", true,
System.nanoTime() - startNanos, 0, 0);
return MessageEnvelope.newBuilder()
.setType(MessageType.SEARCH_OFFSET_BY_TIME_RESPONSE)
.setPayload(response.build().toByteString())
.build();
} catch (Exception e) {
logger.error("Error searching offset by time", e);
BrokerMetrics.get().recordRequest("search_offset_by_time", false,
System.nanoTime() - startNanos, 0, 0);
SearchOffsetByTimeResponse response = SearchOffsetByTimeResponse.newBuilder()
.setSuccess(false)
.setOffset(-1)
.setErrorMessage(e.getMessage() != null ? e.getMessage() : "Unknown error")
.build();
return MessageEnvelope.newBuilder()
.setType(MessageType.SEARCH_OFFSET_BY_TIME_RESPONSE)
.setPayload(response.toByteString())
.build();
}
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@drmq-broker/src/main/java/com/drmq/broker/ClientHandler.java` around lines
484 - 498, Update handleSearchOffsetByTimeRequest to match the structural error
handling and metrics telemetry used by handlers such as
handleFetchOffsetRequest: wrap payload parsing and offset lookup in the
established try-catch flow, record the corresponding request success/failure
metrics, and return the standard well-formed error response for malformed
payloads or runtime exceptions instead of propagating them to the channel.

Comment on lines +122 to +141
/**
* Finds the first offset across all segments for a topic with timestamp >= targetTimestamp.
*/
public long findOffsetByTimestamp(String topic, long targetTimestamp) throws IOException {
ConcurrentSkipListMap<Long, LogSegment> segments = topicSegments.get(topic);
if (segments == null || segments.isEmpty()) {
return -1;
}

// Iterate through segments to find the first one that has the timestamp.
for (LogSegment segment : segments.values()) {
long offset = segment.findOffsetByTimestamp(targetTimestamp);
if (offset != -1) {
return offset; // Found it in this segment!
}
}

return -1;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift

Optimize segment search to prevent O(N) disk scan.

The current implementation linearly scans all segments from oldest to newest by calling segment.findOffsetByTimestamp(targetTimestamp). If a topic has many segments and the target timestamp is recent, this will read and parse every historical message from disk, causing severe I/O amplification and potentially blocking the thread for seconds or minutes.

You can optimize this by checking only the first message of each segment to find the exact segment that bounds the target timestamp, reducing the scan to just one segment.

🚀 Proposed optimization
     public long findOffsetByTimestamp(String topic, long targetTimestamp) throws IOException {
         ConcurrentSkipListMap<Long, LogSegment> segments = topicSegments.get(topic);
         if (segments == null || segments.isEmpty()) {
             return -1;
         }
 
-        // Iterate through segments to find the first one that has the timestamp.
-        for (LogSegment segment : segments.values()) {
-            long offset = segment.findOffsetByTimestamp(targetTimestamp);
-            if (offset != -1) {
-                return offset; // Found it in this segment!
-            }
-        }
-        
-        return -1;
+        LogSegment candidate = null;
+        for (LogSegment segment : segments.values()) {
+            if (segment.getSize() == 0) continue;
+            
+            try {
+                StoredMessage firstMsg = segment.read(0);
+                if (firstMsg.getTimestamp() >= targetTimestamp) {
+                    if (candidate == null) {
+                        return firstMsg.getOffset();
+                    }
+                    long offset = candidate.findOffsetByTimestamp(targetTimestamp);
+                    return offset != -1 ? offset : firstMsg.getOffset();
+                }
+            } catch (CorruptRecordException e) {
+                // Ignore corruption at the start of a segment and fall back to tracking it
+                logger.warn("Corrupt first record in segment {}, skipping fast-path check", segment.getFilePath());
+            }
+            candidate = segment;
+        }
+
+        return candidate != null ? candidate.findOffsetByTimestamp(targetTimestamp) : -1;
     }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
/**
* Finds the first offset across all segments for a topic with timestamp >= targetTimestamp.
*/
public long findOffsetByTimestamp(String topic, long targetTimestamp) throws IOException {
ConcurrentSkipListMap<Long, LogSegment> segments = topicSegments.get(topic);
if (segments == null || segments.isEmpty()) {
return -1;
}
// Iterate through segments to find the first one that has the timestamp.
for (LogSegment segment : segments.values()) {
long offset = segment.findOffsetByTimestamp(targetTimestamp);
if (offset != -1) {
return offset; // Found it in this segment!
}
}
return -1;
}
/**
* Finds the first offset across all segments for a topic with timestamp >= targetTimestamp.
*/
public long findOffsetByTimestamp(String topic, long targetTimestamp) throws IOException {
ConcurrentSkipListMap<Long, LogSegment> segments = topicSegments.get(topic);
if (segments == null || segments.isEmpty()) {
return -1;
}
LogSegment candidate = null;
for (LogSegment segment : segments.values()) {
if (segment.getSize() == 0) continue;
try {
StoredMessage firstMsg = segment.read(0);
if (firstMsg.getTimestamp() >= targetTimestamp) {
if (candidate == null) {
return firstMsg.getOffset();
}
long offset = candidate.findOffsetByTimestamp(targetTimestamp);
return offset != -1 ? offset : firstMsg.getOffset();
}
} catch (CorruptRecordException e) {
// Ignore corruption at the start of a segment and fall back to tracking it
logger.warn("Corrupt first record in segment {}, skipping fast-path check", segment.getFilePath());
}
candidate = segment;
}
return candidate != null ? candidate.findOffsetByTimestamp(targetTimestamp) : -1;
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@drmq-broker/src/main/java/com/drmq/broker/persistence/LogManager.java` around
lines 122 - 141, Update LogManager.findOffsetByTimestamp to avoid calling
LogSegment.findOffsetByTimestamp across every segment. Inspect each segment’s
first-message timestamp to identify the segment whose range bounds
targetTimestamp, then invoke findOffsetByTimestamp only on that segment;
preserve -1 for missing or out-of-range topics.

Comment on lines +228 to +250
/**
* Linearly scans the segment to find the first offset with timestamp >= targetTimestamp.
* @param targetTimestamp the timestamp to search for.
* @return the offset, or -1 if no message in this segment matches.
*/
public synchronized long findOffsetByTimestamp(long targetTimestamp) throws IOException {
long position = 0;
long foundOffset = -1;
while (position < currentSize) {
try {
StoredMessage msg = read(position);
if (msg.getTimestamp() >= targetTimestamp) {
foundOffset = msg.getOffset();
break;
}
position += 4 + msg.getSerializedSize();
} catch (CorruptRecordException e) {
logger.warn("Corrupt record found while searching segment {} at pos {}", filePath, position);
break;
}
}
return foundOffset;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🔴 Critical | ⚡ Quick win

Remove synchronized to prevent blocking concurrent appends.

findOffsetByTimestamp linearly scans the segment, which can involve thousands of disk reads and Protobuf allocations. Holding the synchronized lock for the entire scan will block append and appendBatch operations on the active segment, causing severe latency spikes for producers.

Since read() uses position-based FileChannel.read() which is thread-safe, and currentSize is volatile, the scan can safely run concurrently without the lock. Capturing currentSize in a local variable at the start of the method also ensures a stable upper bound for the scan.

⚡ Proposed fix
-    public synchronized long findOffsetByTimestamp(long targetTimestamp) throws IOException {
+    public long findOffsetByTimestamp(long targetTimestamp) throws IOException {
         long position = 0;
         long foundOffset = -1;
-        while (position < currentSize) {
+        long sizeLimit = currentSize;
+        while (position < sizeLimit) {
             try {
                 StoredMessage msg = read(position);
                 if (msg.getTimestamp() >= targetTimestamp) {
                     foundOffset = msg.getOffset();
                     break;
                 }
                 position += 4 + msg.getSerializedSize();
             } catch (CorruptRecordException e) {
                 logger.warn("Corrupt record found while searching segment {} at pos {}", filePath, position);
                 break;
             }
         }
         return foundOffset;
     }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
/**
* Linearly scans the segment to find the first offset with timestamp >= targetTimestamp.
* @param targetTimestamp the timestamp to search for.
* @return the offset, or -1 if no message in this segment matches.
*/
public synchronized long findOffsetByTimestamp(long targetTimestamp) throws IOException {
long position = 0;
long foundOffset = -1;
while (position < currentSize) {
try {
StoredMessage msg = read(position);
if (msg.getTimestamp() >= targetTimestamp) {
foundOffset = msg.getOffset();
break;
}
position += 4 + msg.getSerializedSize();
} catch (CorruptRecordException e) {
logger.warn("Corrupt record found while searching segment {} at pos {}", filePath, position);
break;
}
}
return foundOffset;
}
/**
* Linearly scans the segment to find the first offset with timestamp >= targetTimestamp.
* `@param` targetTimestamp the timestamp to search for.
* `@return` the offset, or -1 if no message in this segment matches.
*/
public long findOffsetByTimestamp(long targetTimestamp) throws IOException {
long position = 0;
long foundOffset = -1;
long sizeLimit = currentSize;
while (position < sizeLimit) {
try {
StoredMessage msg = read(position);
if (msg.getTimestamp() >= targetTimestamp) {
foundOffset = msg.getOffset();
break;
}
position += 4 + msg.getSerializedSize();
} catch (CorruptRecordException e) {
logger.warn("Corrupt record found while searching segment {} at pos {}", filePath, position);
break;
}
}
return foundOffset;
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@drmq-broker/src/main/java/com/drmq/broker/persistence/LogSegment.java` around
lines 228 - 250, Remove synchronized from LogSegment.findOffsetByTimestamp so
the linear scan does not block append or appendBatch. Capture the volatile
currentSize once at method entry and use that local snapshot as the scan’s upper
bound, while preserving the existing read, corruption handling, and
offset-search behavior.

Comment on lines +318 to +333
/**
* Subscribe to a topic, starting from the first message at or after the given timestamp.
* Overrides the current offset for the topic.
*/
public void seekByTime(String topic, long timestamp) throws IOException {
long offset = doSearchOffsetByTime(topic, timestamp);
if (offset >= 0) {
topicOffsets.put(topic, offset);
if (groupMode) {
commit(topic, offset);
}
logger.info("Subscribed to topic '{}' from time-based offset {} (timestamp={})", topic, offset, timestamp);
} else {
logger.warn("Could not find any offset for topic '{}' at or after timestamp {}", topic, timestamp);
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Topic isn't registered on a failed seek, unlike subscribe() and the Python/TS equivalents.

topicOffsets.put(topic, offset) only runs when offset >= 0. If no matching offset is found, the topic is never added to topicOffsets, so doPoll() (which iterates topicOffsets.entrySet()) will silently skip it on subsequent poll() calls. subscribe() always registers the topic (even with a default), and the Python/TS seek_by_time/seekByTime unconditionally add the topic to subscriptions up front, falling back to offset 0 on the next poll if the seek didn't find a match. Consider registering the topic (e.g., at its prior offset or 0) even when no match is found, so behavior matches subscribe() and the other clients.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@drmq-client/src/main/java/com/drmq/client/DRMQConsumer.java` around lines 318
- 333, The seekByTime method must register the topic even when
doSearchOffsetByTime returns no matching offset. Preserve a valid found offset,
but on failure add the topic using its prior offset when available or the
default offset 0, so doPoll continues to process it consistently with subscribe
and the other clients.

@samuel025
samuel025 merged commit 561fa93 into main Jul 14, 2026
1 check passed
@samuel025
samuel025 deleted the time-seeking branch July 19, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant