Before installing and configuring Kafka, everyone should be aware of what is Kafka and why is it used. One might be wondering from where one can read and understand about the structure of Kafka. So, my previous post, What is Kafka and how it works?, will answer the above question. Before starting installation and configuration, I will recommend all to read my previous post.  

There are some pre-requisites for Kafka, that should be installed in your machine. I will go through every step and will also tell you what are the pre-requisites and how to install them on your machine.

1- Updating the Package’s list:

Run the following command to update the package’s list:

sudo apt-get update

2- Installing JAVA:

There are several methods of installing Java. You can install latest version of java by running the following command:

sudo apt-get install default-jre

After running above mentioned command, verify the installation of Java by running the command “java -version”. If this command gives error or doesn’t allow you to install Java, then you can install specific version of Java. For this, see my post “How to install Java in Linux”.

3- Installing ZooKeeper:

ZooKeeper is a centralised service for maintaining configuration information, naming, providing distributed synchronisation, and providing group services. ZooKeeper is also required for running Kafka on your machine. Run the following command to install ZooKeeper:

sudo apt-get install zookeeperd

Now, for verification, run the command

telnet localhost 2181

and then type

ruok

This means “Are You OK?” Asking the ZooKeeper port whether it is ok or not. ZooKeeper port will reply with “imok“. This means ZooKeeper port is replying “I am OK”.

4- Creating User for Kafka:

Create a user in order to minimize the risk. You can create user by running the following command:

sudo adduser –system –no-create-home –disabled-password –disabled-login kafka

“kafka”, in the above command, is the username. You can use any username you like.

5- Downloading Kafka:

Kafka .tar file is available for free. In order to install the Kafka, you are required to download that tar file first. Run the following commands in order to download the Kafka tar file.

~ cd

wget “http://www-eu.apache.org/dist/kafka/1.0.1/kafka_2.12-1.0.1.tgz” 

6- Installing Kafka:

For installing the Kafka, you have to extract the tar file, you downloaded in previous step. For this, run the following commands:

sudo mkdir /opt/kafka

sudo tar -xvzf kafka_2.12-1.0.1.tgz –directory /opt/kafka –strip-components 1

And now, Kafka is installed. But this won’t work fine unless it is configured properly. Let’s configure Kafka in  next step.

7- Configuring Kafka:

For configuring Kafka, you need to directories where configurations will be saved. Create the two directories by running the following commands:

sudo mkdir /var/lib/kafka

sudo mkdir /var/lib/kafka/data

Now, open server.properties file in order to set the configurations according to your requirement. Open the file by running this command:

sudo vim /opt/kafka/config/server.properties

Find the line “delete.topic.enable” and change it to “true” from “false“. Next find the line starting with “log.dirs”. Change this line to “log.dirs=/var/lib/kafka/data”. 

Find the line “listeners=PLAINTEXT://localhost:9092“. You need to edit it by removing localhost and writing the ip address of the server, if you want to access the Kafka remotely. It will look something like this: “listeners=PLAINTEXT://192.168.13.133:9092“.

Kafka deletes the logs automatically after specific time period. You can adjust the log deletion time by changing the line “log.retention.hours=168”.

8- Permitting Directories:

This step is important. In this step, you have to change the permissions of directories and give  all permissions to Kafka user, you created in fourth step. You can do this by following commands:

sudo chown -R kafka:nogroup /opt/kafka

sudo chown -R kafka:nogroup /var/lib/kafka

9- Creating ZooKeeper Service:

This step is optional but I will recommend to crezte a ZooKeeper Service and launch ZooKeeper as a service. For this, create a file:

sudo vim /etc/systemd/system/kafka-zookeeper.service

Copy Paste the following lines in the kafka-zookeeper.service file:

[Unit]
Description=Apache Zookeeper server (Kafka)
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=nano
Group=nano
Environment=JAVA_HOME=/usr/java/jdk1.8.0_102
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh
/opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh

[Install]
WantedBy=multi-user.target

10- Creating Kafka Service:

Just like the previous step, this step is also optional but I will recommend creating Kafka service and launching Kafka as a service. For this, create a file:

sudo vim /etc/systemd/system/kafka.service

Now Copy Paste the following lines in the kafka.service file:

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target kafka-zookeeper.service

[Service]
Type=simple
User=nano
Group=nano
Environment=JAVA_HOME=/usr/java/jdk1.8.0_102
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

11- Starting ZooKeepr and Kafka:

Up till step 10, you are done with installation of Kafka. Now, its time, you can start ZooKeeper and Kafka servers. You can do this by simply running the following commands:

systemctl deamon-reload

systemctl start kafka-zookeeper.service

systemctl start kafka.service

12- Testing:

Open another terminal and create a topic by running this command:

/opt/kafka/bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test

Verify the creation of topic by this command:

/opt/kafka/bin/kafka-topics.sh –list –zookeeper localhost:2181

This will show the list of all the topics. If “test” topic exists in the list, then this means Kafka is working fine and topic has been created.

Now, let’s start publishing messages on the topic test. For this, you need to start producer. Run the following command:

/opt/kafka/bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test

Open another terminal and start consumer by running the following command:

/opt/kafka/bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test 

Now enter some test on producer terminal. As soon as you will press enter, text message will be shown on consumer terminal.

Congratulations! Kafka has been installed and configured properly. Hope this guide has successfully helped you setup Kafka on Ubuntu. Please comment, if you know something else which I missed or stated incorrectly.

Thanks!