Using Apache Kafka for Data Streaming | by Wei-Meng Lee | Mar, 2023

By Jessie Hobb On Mar 1, 2023

Learn how to install and use Kafka to send and receive messages

Apache Kafka is an open source application used for real-time streaming of big data. It is a publish-subscribe messaging system where you can use it to send messages between processes, applications, and servers. The following diagram shows a high-level architecture overview of Apache Kafka:

Unlike other messaging systems, Kafka also have additional features such as partitioning, replication, and has higher throughput and fault-tolerance than other messaging systems. All these features make Kafka ideal for high volume message processing.

I will talk more about clustering and partitioning and other features in future articles.

In this article, I will walk you through the process of installing Kafka on your system so that you have a better idea of how it works. Future articles will dive into specific use cases of Kafka.

Installing Kafka is pretty straight-forward. You just need to follow the 3 steps outlined below to install and get Kafka running:

Install Java
Download and install Kafka
Run ZooKeeper and Kafka

For this article, I will show how to install Kafka on macOS.

1. Install Java

Kafka is written in Scala and Java. So the first step you need to do is to install the latest JDK on your Mac. You can install Java by going to the Official Java Downloads page at: https://www.oracle.com/java/technologies/downloads/.

If you have an Intel-based Mac, download the x64 DMG Installer; otherwise, download the Arm 64 DMG Installer.

Alternatively if you have Homebrew (https://brew.sh/) installed on your Mac, you can use the following commands to install Java directly:

$ brew tap caskroom/cask
$ brew cask install java

2. Download Kafka

Go to the Official Apache Kafka Downloads page at https://kafka.apache.org/downloads and download the latest binary version of Kafka (kafka_2.13–3.4.0.tgz at the time of writing):

On the Mac, the downloaded binary will be saved in the default Downloads folder.

In Terminal, type the following commands to uncompress the Kafka installer:

$ cd ~/Downloads
$ tar -xzf kafka_2.13-3.4.0.tar

A new folder named kafka_2.13–3.4.0 would be created in the Downloads folder. Move the entire folder to your home directory (~).

Alternatively, you can use Homebrew to install Kafka using the following command:

$ brew install kafka

3. Run ZooKeeper & Kafka

To ensure that Kafka is running correctly, you need to run a tool known as ZooKeeper.

ZooKeeper is responsible for Kafka’s Cluster Management. It comes with the Kafka installation package.

To start ZooKeeper, type the following commands into Terminal:

$ cd ~/kafka_2.13-3.4.0
$ bin/zookeeper-server-start.sh config/zookeeper.properties

The kafka_2.13–3.4.0 folder is in your home directory.

Next, start the Kafka broker service. A Kafka broker is a mediator between the producer(s) and consumer(s):

In a new Terminal window, type the following commands to start the Kafka broker service:

$ cd ~/kafka_2.13-3.4.0
$ bin/kafka-server-start.sh config/server.properties

If there is no error, your Kafka installation is now ready to go!

With the Kafka broker up and running, you can now see Kafka in action! First, create a new topic.

Topics are categories used to manage and organzie messages. Producers send messages to specific topics and consumers subscribe to specific topics to receive messages sent to the individual topic.

Start a new Terminal window and type in the following commands:

$ cd ~/kafka_2.13-3.4.0
$ bin/kafka-topics.sh --create --topic chat --bootstrap-server localhost:9092

The above command creates the “chat” topic on the Kafka broker service listening at port 9092.

To view the topic that you have just created, use the following commands:

$ cd ~/kafka_2.13-3.4.0
$ bin/kafka-topics.sh --describe --topic chat --bootstrap-server localhost:9092

You should see the following output:

Topic: chat TopicId: TuTCBnOISzqhOpqh8FP2FA PartitionCount: 1 ReplicationFactor: 1 Configs: 
Topic: chat Partition: 0 Leader: 0 Replicas: 0 Isr: 0

To write an event (commonly known as a message) into the topic, first use the following command to start the Kafka producer console:

$ bin/kafka-console-producer.sh --topic chat --bootstrap-server localhost:9092

You should now see the > prompt:

Go ahead and type some messages:

The messages will be sent to the Kafka broker.

To read the events (messages) from the “chat” topic, launch a new Terminal window and type in the following commands to start the Kafka consumer console:

$ cd ~/kafka_2.13-3.4.0
$ bin/kafka-console-consumer.sh --topic chat --from-beginning --bootstrap-server localhost:9092

You should now see the messages that you have sent in earlier:

You can launch another new Terminal window and type in the same commands to launch another Kafka consumer console and you will also see the same messages.

Go to the Terminal window where you run the Kafka producer console and type in a new message. The two other Terminal windows running the Kafka consumer console will now receive the new message:

If you like reading my articles and that it helped your career/study, please consider signing up as a Medium member. It is $5 a month, and it gives you unlimited access to all the articles (including mine) on Medium. If you sign up using the following link, I will earn a small commission (at no additional cost to you). Your support means that I will be able to devote more time on writing articles like this.

In this article, I briefly talked about the use of Apache Kafka and some of the major components in Kafka. In order not to overwhelm you with all the details, I showed you how to get started with Kafka by installing it, and then starting the Kafka broker service. To get a feel of how Kafka works, you then created a topic where you used the Kafka producer and consumer to send and receive messages. Hopefully, this quick-start gives you a clearer picture of what Kafka can do and in future articles, I will dive into more specific use cases of Kafka!