Skip to content

Lettuce 7.x incompatibility with indy Implementation #19280

Description

@juszczakn

Describe the bug

When using the new default indy implementation, newer versions of Lettuce fail to startup. This appears to be due to the way the indy impl isolates the javaagent instrumentation to it's own classloader.

Steps to reproduce

Using JDK 25. I believe you need a running Redis instance to test this against; without one available, the test code stalls out on creating the client, prior to the instrumentation occurring.

Maven Dependencies:

        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-api</artifactId>
            <version>1.64.0</version>
        </dependency>
        <dependency>
            <groupId>io.lettuce</groupId>
            <artifactId>lettuce-core</artifactId>
            <version>7.6.0.RELEASE</version>
        </dependency>

Main.java must attempt to connect to a running redis server.

import io.lettuce.core.RedisClient;
import io.lettuce.core.api.StatefulRedisConnection;

public class Main {
    static void main() {
        RedisClient client = RedisClient.create("redis://localhost");
        StatefulRedisConnection<String, String> connection = client.connect();
    }
}

I'm running this with the latest Elastic distribution of OTel javaagent:

This issue does NOT occur when I use the latest 'standard' javaagent supplied by the main project:

Running:

JAVA_TOOL_OPTIONS="-javaagent:/path/to/elastic-otel-javaagent-1.11.0.jar" java -jar target/otel-redis-instr-error-1.0-SNAPSHOT-jar-with-dependencies.jar

Expected behavior

Starts up without error.

Actual behavior

Exception in thread "main" io.lettuce.core.RedisConnectionException: Unable to connect to localhost/<unresolved>:6379
        at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:63)
        at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:41)
        at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:315)
        at io.lettuce.core.RedisClient.connect(RedisClient.java:220)
        at io.lettuce.core.RedisClient.connect(RedisClient.java:205)
        at org.example.Main.main(Main.java:16)
Caused by: java.lang.IllegalAccessError: class io.lettuce.core.protocol.OtelCommandArgsUtil tried to access field io.lettuce.core.protocol.CommandArgs.singularArguments (io.lettuce.core.protocol.OtelCommandArgsUtil is in unnamed module of loader io.opentelemetry.javaagent.tooling.instrumentation.indy.InstrumentationModuleClassLoader @1efcba00; io.lettuce.core.protocol.CommandArgs is in unnamed module of loader 'app')
        at io.lettuce.core.protocol.OtelCommandArgsUtil.getCommandArgs(OtelCommandArgsUtil.java:28)
        at io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing$OpenTelemetrySpan.start(OpenTelemetryTracing.java:206)
        at io.lettuce.core.protocol.CommandHandler.attachTracing(CommandHandler.java:497)
        at io.lettuce.core.protocol.CommandHandler.writeSingleCommand(CommandHandler.java:433)
        at io.lettuce.core.protocol.CommandHandler.write(CommandHandler.java:401)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:798)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:859)
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1046)
        at io.netty.channel.Channel.writeAndFlush(Channel.java:257)
        at io.lettuce.core.RedisHandshake.dispatch(RedisHandshake.java:385)
        at io.lettuce.core.RedisHandshake.dispatchHello(RedisHandshake.java:260)
        at io.lettuce.core.RedisHandshake.initiateHandshakeResp3(RedisHandshake.java:242)
        at io.lettuce.core.RedisHandshake.tryHandshakeResp3(RedisHandshake.java:128)
        at io.lettuce.core.RedisHandshake.initialize(RedisHandshake.java:108)
        at io.lettuce.core.protocol.RedisHandshakeHandler.channelActive(RedisHandshakeHandler.java:85)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:221)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:69)
        at io.lettuce.core.ChannelGroupListener.channelActive(ChannelGroupListener.java:42)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:221)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1417)
        at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:862)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:355)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:385)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:432)
        at io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:388)
        at io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:596)
        at io.netty.channel.nio.NioIoHandler.processSelectedKeysPlain(NioIoHandler.java:541)
        at io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:514)
        at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:484)
        at io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
        at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1195)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1474)

Javaagent or library instrumentation version

elastic-otel-javaagent-1.11.0.jar

Environment

JDK: 25
OS: OpenSUSE, MacOS

Additional context

We appear to be able to work around this issue, for now, via otel.javaagent.experimental.indy=false. Presumably this workaround will go away at some point in the future.

Here's a working minimal example project: https://github.com/juszczakn/otel-redis-instr-error

Tip

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds triageNew issue that requires triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions