Is Kafka an “event-store” or an “event-streaming” platform?
For quite sometime (even now) many in the industry do not how to use Kafka in the right way. Is Kafka an “event-store” or an “event-streaming” platform? Address the following problems:
- What is your business use-case?
- Do you even need even Kafka in the first place?
- Is your data is really “big-data”? Sometimes people think that it is quite huge is not really huge according to others.
Setting up a Kafka Cluster and making it as reliable as possible itself will consume lot of your time. Hence, confluent is giving you a platform to adopt and asks you to concentrate rather on the business use-case alone.
A Typical CDC Architecture using Kafka will look similar to this:
Then to setup this sort of architecture, you need
- Infrastructure (cloud/on-prem)
- Cluster Nodes (Capacity planning: CPU, Memory, Disks etc.,)
- Replication factor & Redundancy will determine the # of nodes and their location in Cloud.
- Setting up of the software components (Zookeeper, Kafka, Kafka connectors).
- Deploy all software components in a seamless manner in any cloud/on-prem infrastructure using the Orchestration tools like Ansible/puppet/chef from a CI/CD environment like Jenkins, Bamboo.
You might need additional sheets of paper to note down this software components (Mesh or Maze you can see it yourself)
In the Big Data world no one can be an expert of all these software components and their configuration. By the time one becomes an expert in them, there will be other software tools/technologies in place :). To put in perspective, the big data landscape in 2019 is:
Coming back to the problem, Now you can ask yourself the following questions:
- What is our business use-case?
- Do we even need even Kafka in the first place?
- Are we really talking “big-data”?
When you are convinced that you need Big Data environment with Kafka as your message carrier, you need to carefully look thro’ your business use-case.
- What are your events (web-clicks, call data records, sentiment analysis, IoT data, Router data, Network element data etc.,)
- Carefully take your time on the five V’s of BigData (Volume, velocity, variety, veracity and value)
- Do your math with Excel (Infrastructure, Nodes, Memory, CPU, Replication factor etc.,)
- How much level of Automation is needed?
- Do you need Real-time processing ? Or Batch processing is enough Or Both. (Lamda Architecture)
All of these Questions when not asked at the start of the project will make your life hell later. It is quite nice to be with latest/greatest version and on top of the technology stack… (It is good for resume!!!) but think about the “Business Value proposition” ?
Definitely,
One size doesn’t fit for all
Choosing the right software component in the BigData landspace is quite important and that determines your market value. This is also applicable for Kafka as your backbone message architecture.
Kafka when they started was meant completely for event streaming purposes only. Now, Confluent with the introduction of Kafka Connect, KSQL, KStreams and then providing hosted confluent platform are dwelling into the landscape of AWS/GCP/Azure platform.
To make use of the KSQL, KStreams effectively you might need to buffer/cache or store in Kafka itself the data for a longer period of time.(To join in real-time)
Kafka log disks needs to be carefully designed.
One can set the log.retention.ms to -1 and store the data forever in Kafka. What are implications of that? We can’t store the click-stream data forever whereas you can store let’s say IPAddress lookup (Maxmind data) forever… Carefully the Architects/Developers need to make a choice and then tune this Kafka broker log retention setting.
Please note that one can tune
log.retention.msat each kafka topic level
I think now it is time to wrap-up.
Take Aways:
- Take time to design.
- Try the PoC before deciding on the bigdata software component.
- Last but not the least automation of all these deployments. Doing all of these things manually is not feasible.
