33,478 questions
0
votes
0
answers
19
views
Install php_rdkafka on windows php 8.0
I tried to install php_rdkafka and I followed the documentation here : https://arnaud-lb.github.io/php-rdkafka-doc/phpdoc/rdkafka.installation.windows.html
But I have this error
PHP Warning: PHP ...
- 5
0
votes
0
answers
15
views
How to perform CDC in Snowflake and push changes to Elasticsearch using Kafka and Python?
I'm working on a data pipeline where I need to implement Change Data Capture (CDC) in Snowflake and stream those changes to Elasticsearch. The pipeline should use Kafka for streaming and Python for ...
- 111
0
votes
0
answers
20
views
Kafka broker authentication failing after changing to SCRAM module
Currently I have a KRaft based kafka cluster, where both the inter-broker and controller communication uses the SASL_PLAIN auth protocol.
echo security.inter.broker.protocol=SASL_SSL >> $...
-1
votes
0
answers
37
views
How to handle both Kafka message headers and values in a single Spark UDF?
I'm working with a Spark Structured Streaming application that reads from Kafka and needs to process both message headers and values together. Currently, I'm using two separate UDFs, which feels ...
- 5,196
0
votes
1
answer
35
views
Kubernetes bitnami/kafka connecting failed
I tried to connect to my kafka server on my bearmetal kubernetes installation with default settings.
Install with helm
helm install kafka oci://registry-1.docker.io/bitnamicharts/kafka -n kafka
...
- 35
1
vote
1
answer
38
views
Who guarantees per-partition ordering in Kafka Producer with enable.idempotence=true? Producer or Broker?
I found out that in Kafka Producer, when enable.idempotence=true and max.in.flight.requests.per.connection <= 5, the order of messages is guaranteed within a single partition. However, I'm confused ...
- 105
1
vote
2
answers
44
views
Kafka stream in memory heap memory [closed]
I'm having a topic with a size of 3gb of raw data in 60 Million records. But when I consume those records and I allocate in memory (Not using RocksDB) the needed heap memory is around 10GB. I cannot ...
- 11
0
votes
0
answers
12
views
Kafka - how to know when an consumption is finished
I'm a client of a topic about a "Product has been updated".
This event contains all the fields of the product object.
When i consume a message i need to product another one, to ask the ...
- 591
0
votes
0
answers
24
views
MirrorMaker 2: topic gets created on target but records stay (or loop) on source
MirrorMaker 2: topic gets created on target but records stay (or loop) on source
Behaviour
Topology
Source cluster (Docker Compose, Confluent 7.9.0, PLAINTEXT) → firewall / DNAT → Target cluster (...
- 11
0
votes
0
answers
33
views
apicurio schema registry v2.6 - Java Kafka Producer not working
I've Apicurio Schema Registry Operator installed, also Schema Registry v2.6 installed. This is the default version of Schema Registry installed when using Apicurio Schema Registry Operator. Keycloak(...
- 1,063
0
votes
0
answers
29
views
How do I produce a message with kafka-python?
I try to use kafka-python to produce a message.
The error kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs. occurs.
I guess something is wrong with ...
- 119
0
votes
0
answers
20
views
Kafka Error: SerializationException: Error retrieving Avro value schema for id 2 - ONLY some records in Partition
I'm running a kafka consumer that experienced the issue below with a few records within a a Topic partition. The problem persisted until manually changed the offset to keep consuming from. The rest of ...
- 45
0
votes
0
answers
20
views
How to Pull Response from Consumer topic of Kafka Gateway
Struck with some issue on how we can pull response from Kafka Gateway based on some Parameters as we are getting bulk of response when we try to pull response.
ConsumerRecords<String, byte[]> ...
2
votes
0
answers
27
views
MongoDb Kafka Source Connector Pipeline is causing BsonInvalidOperationException
I have a collection which I need to covert to timeseries using mongodb kafka connector. The source collection have document of this shape
{
"_id": "6798680c885bb2a8fbe93485",
...
- 639
0
votes
0
answers
30
views
AWS MSK and Glue integration for processing batches of messages
I need to do a batch processing for the messages of a topic with a low throughput, so ideally, instead of a service running 24/7, the job is executed a few times a day. The topic is in an MSK cluster ...
