In this post, I will explain the Kafka Streams suppress concept. Even though, it seems quite easy to understand there are some inherent issues/things that one has to understand. This is the continuation from my previous blog post CDC for Analytics.
A Typical CDC architecture can be represented as:
In the above architecture,
Hi Abishek, Thank you so much for sharing the insights in to Kafka Streams which I was struggling to understand from the documentation.
You have summarized lot of things together.
Thanks a lot
Earlier we hire a documentation/Technical writer and the documentation that he generates was always kept in a separate and secluded place. Nowadays, tech writers have been asked to maintain their documentation as markdown/asciidocs. Then asciidoctor came into automatically generate HTML5/DocBook/PDF documentation on the fly from these static asciidocs.
Then, the project lead “Dan Allen” of Asciidoctor started an open-source project called Antora.
Antora is a multi-repository documentation site generator for tech writers who love writing in AsciiDoc.
A Typical Antora documentation layout can be represented as:
In one of my previous post you would have seen that the main components are Kafka in addition to the Kafka connectors. There are 2 different types of connectors (Source and Sink). Confluent have written quite nicely and provided to us interfaces through which we can configure them. The configuration files for the source and sink are JSON. There are some standard JSON elements dictated by the type of the Kafka connectors and other parts are defined by the specific source and sink connectors.
Take for example in our CDC architecture the source and sink connectors configuration.
First, we will…
For quite sometime (even now) many in the industry do not how to use Kafka in the right way. Is Kafka an “event-store” or an “event-streaming” platform? Address the following problems:
Gone are the days where in we wait for the ETL script to run in batch mode every night when there are:
From Jay Kreps (back in 2013)
Change Data Capture — There is a small industry around getting data out of databases, and this is the most log-friendly style of data extraction.
But, big data world and Kafka has evolved ever since. Now, CDC architecture has become a de-facto standard for extracting data from any RDBMS and transfer the same to…
What is Kafka Retention
Kafka retention provides the ability to control the size of the Topic logs and avoid outgrowing the existing disk size. The retention can be configured or controlled based on the size of the logs or based on the configured duration.
Also, the same retention can be set across all the Kafka topics or it can be configured per topic, depending on the nature of the topic we can set the retention accordingly.
For example, the topics which are used for logging purpose can be set with different retention compared to the topics which are used for…
Just another Tech savvy