Setup and test Kafka

Scenario:

Setup Kafka on windows and test the messaging.

Solution:

Kafka is a distributed streaming platform which allows for Publishing and subscribing to streams of records,
store streams and lets consumer process streams of records.

Kafka runs as a cluster on servers. The records are stored as topics, which consists of key - value pair with  timestamp.

1. Install:
1. JRE - >  Latest version from 
1. https://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html
2. Control Panel -> System -> Advanced system settings -> Environment Variables - > System environment variables -> new variable -> "JAVA_HOME" as variable name and "C:\Program Files\Java\jre1.8.0_191\" as value. Edit path variable and add "%JAVA_HOME%\bin"
2. Zookeeper
1. http://zookeeper.apache.org/releases.html .. bin.tar file
2. Go to ../conf folder and change  “zoo_sample.cfg” to  “zoo.cfg”
3. Change dataDir to the folder you want like /data
3. Kafka
1. http://kafka.apache.org/downloads.html .. bin.tar file
2. Go to ../config/server.properties, change log.dir to the folder you want like /kafka_logs
3. To change the default port:9094 update the .\config\server.properties and add "port=9094"


2. Run below commands to start Zookeeper and Kafka. Open new Command prompt for each of the commands.



Zookeeper; ..\bin\ zkserver Kafka Broker Kafka root -> .\bin\windows\kafka-server-start.bat .\config\server.properties Create topic Kafka root -> .\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_1 Producer Kafka root -> .bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic test_1 Consumer Kafka root -> .\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test_1



3. Got to Producer and type "Hello world" and then the message would be rendered on Consumer.