| layout | post |
|---|---|
| title | Produce and consume Kafka messages over HTTPS with OVH Queue DBaaS |
| categories | queue |
| author | sebgl |
| lang | en |
This tutorial assumes you already have an OVH account and have subscribed to the Queue DBaaS. If not, have a look at our Getting Started guide that will also introduce you to Kafka
Kafka protocol is built over TCP for multiple reasons, including performance and message ordering. However we realized that for some use cases, Kafka clients usage can be quite complex since many parameters have to be taken into consideration:
- Number of partitions and partition leaders
- Offset commit
- Consumer groups
According to your programming language of choice, Kafka clients can be more or less efficient and easy to use. Sometimes what you need is a simpler queue platform that you can use with any programming language. HTTP is the de-facto standard for that usage.
Queue DBaaS platform implements a simple HTTP layer that allows you to:
- Produce one or many messages to your topic with a POST request
- Consume one or many messages from your topic with a GET request
Moreover, it comes with HTTPS support, for free.
According to how you created your Queue DBaaS app, you chose one of the available region, which can be retrieved on your Sunrise Manager. Example:
sbg.queue.ovh.net:9092
This corresponds to the "normal" Kafka usage over TCP, required by most Kafka clients. If you want to use DBaaS Queue over HTTP, you have to use the associated HTTPS URL:
https://sbg.queue.ovh.net
As with traditionnal Queue DBaaS usage with Kafka Clients, you will have to use:
- your key (token)
- your topic prefix
See our Getting Started guide for more information.
User authentication is made with a custom HTTP header that you must pass in with your requests:
"X-Ovh-Queue-Token: <your-key>"
You can post one or multiple messages, using curl or HTTP clients in any language:
curl -H "X-Ovh-Queue-Token: <your-key>"
-XPOST https://sbg.queue.ovh.net/topic/<topic-prefix.topic-name>
-d '[{"Value": "first message"},
{"Value":"second message"}]'
You can retrieve one or multiple messages, using curl or HTTP clients in any language:
curl -H "X-Ovh-Queue-Token: <your-key>"
https://sbg.queue.ovh.net/topic/<topic-prefix.topic-name>[?limit=2]
The optional limit parameter allows you to specify how many messages you want to retrieve.
Offsets are managed automatically at each GET request. It means that each message will be read only once. We plan to support manual offset and consumer group specification later on.
As seen in the examples above, messages are sent through a JSON object, and must be strings. This does not prevent you from using other data types, by encoding your binary data as base64 for example.
Despite the ease-of-use, using HTTPS instead of TCP implies a performance hit. To put it simple: if you need a very high read/write throughput, use a normal TCP-based Kafka Client.
As an example, here are numbers on producing and consuming "hello world" messages:
- via a TCP Kafka Client: 200k+ messages per second
- via the HTTPS endpoint: 5-10k messages per second
Note that the number of messages you send or receive with a single HTTP request has an impact on performance: don't hesitate to batch messages together.
Please feel free to contact us for any question, feedback, or improvement that may come to your mind !