Kafka colony record

hardware resource

1. Use three machines to arrange three nodes
2. Each machine /mnt/datas The directory mounts a special SSD disk to store data and logs

Add Users and User Groups

 sudo groupadd kafka sudo useradd -g kafka -M kafka

download kafka

 wget  http://mirror.bit.edu.cn/apache/kafka/1.1.0/kafka_2.11-1.1.0.tgz tar -xvzf kafka_2.11-1.1.0.tgz sudo mv kafka_2.11-1.1.0 /usr/local/

to configure zookeeper

edit /usr/local/kafka_2.11-1.1.0/config/zookeeper.properties , configure the following information

 initLimit=10 tickTime=2000 syncLimit=15 clientPort=2181 maxClientCnxns=0 dataDir=/mnt/datas/zookeeper_data/data dataLogDir=/mnt/datas/zookeeper_data/data_log server.1=kafka-zookeeper-1.prodenv.com:2888:3888 server.2=kafka-zookeeper-2.prodenv.com:2888:3888 server.3=kafka-zookeeper-3.prodenv.com:2888:3888

edit /etc/hosts File, add the following configuration

 192.168.32.59     kafka-zookeeper-1.prodenv.com 192.168.32.88     kafka-zookeeper-2.prodenv.com 192.168.32.97     kafka-zookeeper-3.prodenv.com

establish zookeeper Of dataDir

 sudo mkdir /mnt/datas/zookeeper_data sudo chown kafka:kafka -R /mnt/datas/zookeeper_data sudo -u kafka -H mkdir /mnt/datas/zookeeper_data/data sudo -u kafka -H mkdir /mnt/datas/zookeeper_data/data_log sudo -u kafka -H mkdir /mnt/datas/zookeeper_data/logs

establish myid file

 #Each machine needs to be configured with different IDs sudo -u kafka -H bash -c 'echo 1 > /mnt/datas/zookeeper_data/data/myid'

newly build zookeeper systemd service Startup Item /etc/systemd/system/kafka-zookeeper.service

 [Unit] Description=Apache Zookeeper server (Kafka) Documentation= http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target ConditionPathExists=/usr/local/kafka_2.11-1.1.0/config/zookeeper.properties [Service] Type=simple User=kafka Group=kafka Environment=JAVA_HOME=/usr/local/jdk1.8 Environment=LOG_DIR=/mnt/datas/zookeeper_data/logs ExecStart=/usr/local/kafka_2.11-1.1.0/bin/zookeeper-server-start.sh /usr/local/kafka_2.11-1.1.0/config/zookeeper.properties ExecStop=/usr/local/kafka_2.11-1.1.0/bin/zookeeper-server-stop.sh [Install] WantedBy=multi-user.target

to configure kafka

edit /usr/local/kafka_2.11-1.1.0/config/server.properties , configure the following information

 Broker. id=1 # Each node ID cannot be duplicate Listeners=PLAINTEXT://192.168.32.59: 9092 # Configure the corresponding IP address for each machine log.dirs=/mnt/datas/kafka zookeeper.connect=kafka-zookeeper-1.prodenv.com:2181,kafka-zookeeper-2.prodenv.com:2181,kafka-zookeeper-3.prodenv.com:2181

establish log.dirs

 sudo mkdir /mnt/datas/kafka sudo chown kafka:kafka -R /mnt/datas/kafka

newly build kafka systemd service Startup Item /etc/systemd/system/kafka.service

 [Unit] Description=Apache Kafka server (broker) Documentation= http://kafka.apache.org/documentation.html Requires=network.target remote-fs.target After=network.target remote-fs.target kafka-zookeeper.service ConditionPathExists=/usr/local/kafka_2.11-1.1.0/config/server.properties [Service] Type=simple User=kafka Group=kafka Environment=JAVA_HOME=/usr/local/jdk1.8 Environment=LOG_DIR=/mnt/datas/kafka Environment=JMX_PORT=9998 Environment="KAFKA_HEAP_OPTS=-Xmx5G -Xms5G" ExecStart=/usr/local/kafka_2.11-1.1.0/bin/kafka-server-start.sh /usr/local/kafka_2.11-1.1.0/config/server.properties ExecStop=/usr/local/kafka_2.11-1.1.0/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target

Start Service

 #Refresh systemd sudo systemctl daemon-reload #Configure power on auto start sudo systemctl enable kafka-zookeeper.service sudo systemctl enable kafka.service #Start zookeeper sudo systemctl start kafka-zookeeper.service #Start kafka sudo systemctl start kafka.service

After each node completes the above operations, you can use the following command to enter zookeeper shell :

 /usr/local/kafka_2.11-1.1.0/bin/zookeeper-shell.sh kafka-zookeeper-1.prodenv.com:2181, kafka-zookeeper-2.prodenv.com:2181,kafka-zookeeper-3.prodenv.com:2181

End~ 👊

This article is written by Chakhsu Lau Creation, adoption Knowledge Sharing Attribution 4.0 International License Agreement.
All articles on this website are original or translated by this website, except for the reprint/source. Please sign your name before reprinting.