still in dev?

[r2evans]

Do you have the intention of continuing development on this package? It's not easily usable at the moment: devtools::install_github does not install it, there are no compile-notes or build instructions, and it is a bit behind librdkafka's version history. This is not a criticism, just an observation if somebody else can pick up the torch and run with it. None of the forks I found showed progress.

Thanks.

[pedrocava]

Not sure if it helps, @r2evans, but what I've done is wrap python's kafka client with reticulate and it works like a charm.

[r2evans]

@pedrocava, I had thought of that and have not put much effort into implementing it. Do you have a working gist or repo with working code? I'm sure I could work through it myself (most of the learning is in python's kafka client, with a little bit of learning reticulate), was hoping to capitalize on your "works like a charm".

(My challenge is that I need to deploy it in a k8s/swarm as a programmatic consumer/producer ... ugh.)

Thanks.

[pedrocava]

Your deployment requirements sound a lot tougher than mine (tested on an EC2 with cronjobs). Can't really share code I've used, but it would go like this:

library(reticulate)

producer <- reticulate::import("kafka")$KafkaProducer(bootstrap_servers = something, sasl_plain_username = something_else, ...)

prodeucer$send(topic, message)

[r2evans]

well that was simpler than I expected, thank you @pedrocava !

[pedrocava]

Or you can write the abstraction directly in python:

from kafka import KafkaProducer
from kafka.errors import KafkaError

producer = KafkaProducer(
    bootstrap_servers = os.environ["kafka-bootstrap-brokers"],
    sasl_plain_username = os.environ["kafka-username"],
    sasl_plain_password = os.environ["kafka-password"],
    security_protocol = "SASL_SSL",
    sasl_mechanism = "SCRAM-SHA-512",
    retries = 1)

def send_message(message : str, topic : str) -> None:

    try:
        
        producer.send(topic, message.encode('utf-8'))
    
    except Exception as e:

        print(e)

    finally:

        print("Message sent to {} topic".format(topic))

and import only higher levels functions in R, which would sure be nicer.