In this post, I will explain the Kafka Streams suppress concept. Even though, it seems quite easy to understand there are some inherent issues/things that one has to understand. This is the continuation from my previous blog post CDC for Analytics.

Architecture

A Typical CDC architecture can be represented as:

CDC Architecture using Kafka & its components

In the above architecture,

  • Individual table transaction messages are stored in Kafka in separate topics. These can be transferred to the target destination using the Kafka sink connector.
  • To do aggregations like counting, statistics, joining with other streams (CRM or static content) we are using Kafka Streams. Some of the…


Hi Abishek, Thank you so much for sharing the insights in to Kafka Streams which I was struggling to understand from the documentation.

You have summarized lot of things together.

Thanks a lot


Earlier we hire a documentation/Technical writer and the documentation that he generates was always kept in a separate and secluded place. Nowadays, tech writers have been asked to maintain their documentation as markdown/asciidocs. Then asciidoctor came into automatically generate HTML5/DocBook/PDF documentation on the fly from these static asciidocs.

Then, the project lead “Dan Allen” of Asciidoctor started an open-source project called Antora.

Antora is a multi-repository documentation site generator for tech writers who love writing in AsciiDoc.

A Typical Antora documentation layout can be represented as:

  • Antora provides a nice way to split the content from the actual stitching parts.

In one of my previous post you would have seen that the main components are Kafka in addition to the Kafka connectors. There are 2 different types of connectors (Source and Sink). Confluent have written quite nicely and provided to us interfaces through which we can configure them. The configuration files for the source and sink are JSON. There are some standard JSON elements dictated by the type of the Kafka connectors and other parts are defined by the specific source and sink connectors.

Take for example in our CDC architecture the source and sink connectors configuration.

A Typical CDC Architecture

Creating the secrets

First, we will…


For quite sometime (even now) many in the industry do not how to use Kafka in the right way. Is Kafka an “event-store” or an “event-streaming” platform? Address the following problems:

  • What is your business use-case?
  • Do you even need even Kafka in the first place?
  • Is your data is really “big-data”? Sometimes people think that it is quite huge is not really huge according to others.

Setting up a Kafka Cluster and making it as reliable as possible itself will consume lot of your time. …


Gone are the days where in we wait for the ETL script to run in batch mode every night when there are:

  • Less frequent users
  • Slave node (Screaming out aloud)
  • Seeing thro’ the message in pagers/SMS alerts at midnights !!!

From Jay Kreps (back in 2013)

Change Data Capture — There is a small industry around getting data out of databases, and this is the most log-friendly style of data extraction.

But, big data world and Kafka has evolved ever since. Now, CDC architecture has become a de-facto standard for extracting data from any RDBMS and transfer the same to…


What is Kafka Retention

Kafka retention provides the ability to control the size of the Topic logs and avoid outgrowing the existing disk size. The retention can be configured or controlled based on the size of the logs or based on the configured duration.

Also, the same retention can be set across all the Kafka topics or it can be configured per topic, depending on the nature of the topic we can set the retention accordingly.

For example, the topics which are used for logging purpose can be set with different retention compared to the topics which are used for…

Ganesh Ramasubramanian

Just another Tech savvy

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store