Two way TLS to schema registry doesn't work anymore with 3.31.0

[yvespp]

With version 3.31.0 we are no longer able to connect to the schema registry with two way TLS.
Test URL: https://my-host/topic/my-topic/messages?partition=0&offset=0&count=100&keyFormat=DEFAULT&format=AVRO&isAnyProto=false
Exception:

A 500 error has occurred: Request processing failed; nested exception is org.apache.kafka.common.errors.SerializationException: Error retrieving Avro value schema for id 3109
org.apache.kafka.common.errors.SerializationException: Error retrieving Avro value schema for id 3109 at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer$DeserializationContext.schemaFromRegistry(AbstractKafkaAvroDeserializer.java:345)
        at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer$DeserializationContext.schemaFromRegistry(AbstractKafkaAvroDeserializer.java:345)
        at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:152)
        at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:53)
...
Caused by: java.net.SocketException: Broken pipe (Write failed)
        at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
...
        at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer$DeserializationContext.schemaFromRegistry(AbstractKafkaAvroDeserializer.java:342)
        ... 87 more

With -Djavax.net.debug=all I can see, that no certificate was selected:

javax.net.ssl|DEBUG|33|XNIO-1 task-1|2023-04-04 08:53:02.440 CEST|ServerHelloDone.java:151|Consuming ServerHelloDone handshake message (
javax.net.ssl|DEBUG|33|XNIO-1 task-1|2023-04-04 08:53:02.440 CEST|CertificateMessage.java:299|No X.509 certificate for client authentication, use empty Certificate message instead

With 3.30.0 it was enough to configure this:

Properties in KAFKA_PROPERTIES_FILE:

security.protocol=SSL
ssl.keystore.password=...
ssl.keystore.location=...
ssl.truststore.password=...
ssl.truststore.location=...

JVM_OPTS:

-Djavax.net.ssl.keyStore=... -Djavax.net.ssl.keyStorePassword=...
-Djavax.net.ssl.trustStore=... -Djavax.net.ssl.trustStorePassword=...

I also tried setting schema.registry.ssl.keystore.* via properties and JVM_OPTS but it didn't make a difference.

I think the cause for this is the update of the Kafka libs from 7.2 to 7.3 because we had a similar problem with kafka-avro-console-consumer. In the newer version we had to set all the keystore/truststore settings via --property schema.registry.ssl.keystore.* to get it to work again.

Is there a way to set the KafkaAvroDeserializerConfig directly? Log:

2023-04-04 09:17:39.782  INFO 1 [  XNIO-1 task-2] o.a.k.c.c.AbstractConfig                 : KafkaAvroDeserializerConfig values:
...
        schema.registry.basic.auth.user.info = [hidden]
        schema.registry.ssl.cipher.suites = null
        schema.registry.ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        schema.registry.ssl.endpoint.identification.algorithm = https
        schema.registry.ssl.engine.factory.class = null
        schema.registry.ssl.key.password = null
        schema.registry.ssl.keymanager.algorithm = SunX509
        schema.registry.ssl.keystore.certificate.chain = null
        schema.registry.ssl.keystore.key = null
        schema.registry.ssl.keystore.location = null
        schema.registry.ssl.keystore.password = null
        schema.registry.ssl.keystore.type = JKS
        schema.registry.ssl.protocol = TLSv1.3
        schema.registry.ssl.provider = null
        schema.registry.ssl.secure.random.implementation = null
        schema.registry.ssl.trustmanager.algorithm = PKIX
        schema.registry.ssl.truststore.certificates = null
        schema.registry.ssl.truststore.location = null
        schema.registry.ssl.truststore.password = null
        schema.registry.ssl.truststore.type = JKS
        schema.registry.url = [https://my-schema-reg/]
...

[Bert-R]

@yvespp I'm new to this, so pardon my ignorance. Suppose you would be able to set the KafkaAvroDeserializerConfig directly, how would you configure it? The answer to that question might give a cue on the solution direction.

[yvespp]

@Bert-R I found an old pull request which adds configuration for the schema registry: #237
Here you can see how it's configured: https://github.com/obsidiandynamics/kafdrop/pull/237/files#diff-923baea4611f78228b876af3c478f4c527ce347860f7ea323a9b0143de1aa9e9

I think the scheme reg props could also be passed directly without creating a client via: kafkaAvroDeserializer.configure(config, false)

Why this stopped working is still a mystery to me, I looked at the schema registry code but couldn't find something suspicious. Code: https://github.com/confluentinc/schema-registry/blob/9f9a1e52b9f2c55a2b37162320d6ba1275bdf594/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroDeserializer.java#L56

[Bert-R]

@davideicardi We have two topics that relate:

• This issue about two-way TLS with the schema registry (Two way TLS to schema registry doesn't work anymore with 3.31.0 #498)
• The PR that makes the schema registry more configurable (Add SSL connection parameter for Schema registry #237)
• The PR that add support for the AWS Glue Schema registry (WIP - Do not Merge: Added support for AWS Glue Schema registry #488)

I propose we wait the refactoring that is coming with #488, then take up #237 and through that address #498.
Makes sense?

[davideicardi]

It seems a good idea for me 👍

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.