Record the new Kafka cluster stress test process

Machine conditions

There are 5 machines in total, and the physical conditions of each machine are as follows:

CPU information is as follows: two

 Architecture:          x86_64 CPU op-mode(s):        32-bit, 64-bit Byte Order:            Little Endian CPU(s):                48 On-line CPU(s) list:   0-47 Thread(s) per core:    2 Core(s) per socket:    12 Seat: 2 NUMA node: 2 Manufacturer ID: GenuineIntel CPU series: 6 Model: 63 Model name: Intel (R) Xeon (R) CPU E5-2678 v3 @ 2.50GHz Step: 2 CPU MHz：             1244.628 BogoMIPS：            4993.38 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 30720K NUMA node 0 CPU: 0-11,24-35 NUMA node 1 CPU: 12-23, 36-47

The memory module information is as follows:

 Memory Device Array Handle: 0x0020 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: DIMM_A1 Bank Locator: NODE 1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MHz Manufacturer: Samsung Serial Number: 388A5EDE Asset Tag: Part Number: M393A4K40CB1-CRC Rank: 2 Configured Clock Speed: 2134 MHz

The disk information is as follows, consisting of two RAID 1

 === START OF INFORMATION SECTION === Model Family:     Samsung based SSDs Device Model:     Samsung SSD 850 EVO 1TB Serial Number:    S2RGNX0K100551F LU WWN Device Id: 5 002538 d4294cab3 Firmware Version: EMT02B6Q User Capacity:    1,000,204,886,016 bytes [1.00 TB] Sector Size:      512 bytes logical/physical Rotation Rate:    Solid State Device Form Factor:      2.5 inches Device is:        In smartctl database [for details use: -P show] ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is:    Thu May 24 18:03:34 2018 CST SMART support is: Available - device has SMART capability. SMART support is: Enabled

Kafka cluster

The kafka cluster is built with five machines with the same physical conditions.

Create api test topic, 3 replicas and 15 partitions, and press test this topic

Pressure test

10000 messages, 2kb per message

Producer pressure measurement

Execution: ./kafka-producer-perf-test.sh --topic api-test --num-records 10000 --record-size 2000 --throughput 10000 --producer-props bootstrap.servers=172.25.52.170:33092,172.25.52.171:33092,172.25.52.169:33092,172.25.52.168:33092,172.25.52.167:33092 batch.size=8196

 10000 records sent, 9680.542110 records/sec (18.46 MB/sec), 27.31 ms avg latency, 247.00 ms max latency, 2 ms 50th, 92 ms 95th, 108 ms 99th, 112 ms 99.9th.

Consumer pressure test

Execution: ./kafka-consumer-perf-test.sh --topic api-test --zookeeper 172.25.52.170:33181,172.25.52.171:33181,172.25.52.169:33181,172.25.52.168:33181,172.25.52.167:33181 --threads 1 --messages 10000

 start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 2018-05-24 18:39:30:930, 2018-05-24 18:39:31:061, 0.9537, 7.2800, 10000, 76335.8779

10 million messages 2 kb each

Producer pressure measurement

Execution: ./kafka-producer-perf-test.sh --topic api-test --num-records 1000000 --record-size 100 --throughput 1000000 --producer-props bootstrap.servers=172.25.52.170:33092,172.25.52.171:33092,172.25.52.169:33092,172.25.52.168:33092,172.25.52.167:33092 batch.size=8196

 561241 records sent, 112248.2 records/sec (214.10 MB/sec), 130.1 ms avg latency, 487.0 max latency. 556093 records sent, 111218.6 records/sec (212.13 MB/sec), 142.5 ms avg latency, 785.0 max latency. 567376 records sent, 113475.2 records/sec (216.44 MB/sec), 145.1 ms avg latency, 784.0 max latency. 571313 records sent, 114262.6 records/sec (217.94 MB/sec), 142.5 ms avg latency, 804.0 max latency. 570905 records sent, 114181.0 records/sec (217.78 MB/sec), 142.0 ms avg latency, 794.0 max latency. 556664 records sent, 111332.8 records/sec (212.35 MB/sec), 147.7 ms avg latency, 791.0 max latency. 568587 records sent, 113717.4 records/sec (216.90 MB/sec), 144.2 ms avg latency, 740.0 max latency. 591267 records sent, 118253.4 records/sec (225.55 MB/sec), 139.1 ms avg latency, 797.0 max latency. 592039 records sent, 118407.8 records/sec (225.84 MB/sec), 136.9 ms avg latency, 820.0 max latency. 615839 records sent, 123167.8 records/sec (234.92 MB/sec), 137.8 ms avg latency, 716.0 max latency. 649346 records sent, 129869.2 records/sec (247.71 MB/sec), 124.0 ms avg latency, 575.0 max latency. 638521 records sent, 127704.2 records/sec (243.58 MB/sec), 125.9 ms avg latency, 861.0 max latency. 599791 records sent, 119958.2 records/sec (228.80 MB/sec), 132.7 ms avg latency, 818.0 max latency. 405575 records sent, 75483.9 records/sec (143.97 MB/sec), 149.8 ms avg latency, 2766.0 max latency. 595724 records sent, 119144.8 records/sec (227.25 MB/sec), 189.6 ms avg latency, 2789.0 max latency. 751120 records sent, 150224.0 records/sec (286.53 MB/sec), 109.1 ms avg latency, 375.0 max latency. 10000000 records sent, 117752.343272 records/sec (224.59 MB/sec), 137.72 ms avg latency, 2789.00 ms max latency, 92 ms 50th, 743 ms 95th, 878 ms 99th, 2773 ms 99.9th.

Consumer pressure test

 start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 2018-05-24 17:40:51:586, 2018-05-24 17:41:27:446, 17261.5051, 481.3582, 10000000, 278862.2421

100 million messages, 2kb per message

Execution: ./kafka-producer-perf-test.sh --topic api-test --num-records 100000000 --record-size 2000 --throughput 100000000 --producer-props bootstrap.servers=172.25.52.170:33092,172.25.52.171:33092,172.25.52.169:33092,172.25.52.168:33092,172.25.52.167:33092 batch.size=8196

 642163 records sent, 128432.6 records/sec (244.97 MB/sec), 113.7 ms avg latency, 414.0 max latency. 729273 records sent, 145854.6 records/sec (278.20 MB/sec), 111.2 ms avg latency, 730.0 max latency. 717807 records sent, 143561.4 records/sec (273.82 MB/sec), 114.5 ms avg latency, 669.0 max latency. 667852 records sent, 133570.4 records/sec (254.77 MB/sec), 121.5 ms avg latency, 465.0 max latency. 561615 records sent, 112323.0 records/sec (214.24 MB/sec), 147.2 ms avg latency, 474.0 max latency. 783997 records sent, 156799.4 records/sec (299.07 MB/sec), 102.1 ms avg latency, 478.0 max latency. 730643 records sent, 146128.6 records/sec (278.72 MB/sec), 108.2 ms avg latency, 850.0 max latency. 624260 records sent, 124852.0 records/sec (238.14 MB/sec), 134.0 ms avg latency, 869.0 max latency. 518143 records sent, 103628.6 records/sec (197.66 MB/sec), 152.0 ms avg latency, 1248.0 max latency. 262529 records sent, 52505.8 records/sec (100.15 MB/sec), 326.7 ms avg latency, 3127.0 max latency. 715721 records sent, 143144.2 records/sec (273.03 MB/sec), 116.8 ms avg latency, 585.0 max latency. 709401 records sent, 141880.2 records/sec (270.62 MB/sec), 113.5 ms avg latency, 473.0 max latency. 677999 records sent, 135599.8 records/sec (258.64 MB/sec), 120.1 ms avg latency, 564.0 max latency. 737457 records sent, 147491.4 records/sec (281.32 MB/sec), 108.1 ms avg latency, 680.0 max latency. 704827 records sent, 140965.4 records/sec (268.87 MB/sec), 115.4 ms avg latency, 661.0 max latency. 676627 records sent, 135325.4 records/sec (258.11 MB/sec), 120.0 ms avg latency, 1023.0 max latency. 590074 records sent, 118014.8 records/sec (225.10 MB/sec), 137.6 ms avg latency, 1054.0 max latency. 659174 records sent, 131834.8 records/sec (251.45 MB/sec), 126.4 ms avg latency, 962.0 max latency. 523764 records sent, 104752.8 records/sec (199.80 MB/sec), 157.3 ms avg latency, 1769.0 max latency. 739164 records sent, 147832.8 records/sec (281.97 MB/sec), 112.6 ms avg latency, 612.0 max latency. 738231 records sent, 147646.2 records/sec (281.61 MB/sec), 107.2 ms avg latency, 721.0 max latency. 369107 records sent, 73821.4 records/sec (140.80 MB/sec), 222.5 ms avg latency, 2160.0 max latency. ··· ··· ··· 850695 records sent, 170139.0 records/sec (324.51 MB/sec), 94.8 ms avg latency, 497.0 max latency. 862939 records sent, 172587.8 records/sec (329.19 MB/sec), 94.8 ms avg latency, 664.0 max latency. 684993 records sent, 136998.6 records/sec (261.30 MB/sec), 118.5 ms avg latency, 846.0 max latency. 759886 records sent, 151977.2 records/sec (289.87 MB/sec), 108.9 ms avg latency, 507.0 max latency. 759156 records sent, 151831.2 records/sec (289.60 MB/sec), 108.2 ms avg latency, 528.0 max latency. 726247 records sent, 145249.4 records/sec (277.04 MB/sec), 112.8 ms avg latency, 509.0 max latency. 646276 records sent, 106488.1 records/sec (203.11 MB/sec), 113.3 ms avg latency, 1989.0 max latency. 555445 records sent, 111089.0 records/sec (211.89 MB/sec), 192.3 ms avg latency, 2665.0 max latency. 931803 records sent, 186360.6 records/sec (355.45 MB/sec), 85.8 ms avg latency, 497.0 max latency. 750725 records sent, 150145.0 records/sec (286.38 MB/sec), 107.9 ms avg latency, 821.0 max latency. 808483 records sent, 161696.6 records/sec (308.41 MB/sec), 104.4 ms avg latency, 646.0 max latency. 712690 records sent, 142538.0 records/sec (271.87 MB/sec), 109.7 ms avg latency, 746.0 max latency. 629610 records sent, 125922.0 records/sec (240.18 MB/sec), 131.2 ms avg latency, 784.0 max latency. 674141 records sent, 134828.2 records/sec (257.16 MB/sec), 124.9 ms avg latency, 708.0 max latency. 751786 records sent, 150357.2 records/sec (286.78 MB/sec), 109.1 ms avg latency, 429.0 max latency. 768948 records sent, 153789.6 records/sec (293.33 MB/sec), 106.1 ms avg latency, 363.0 max latency. 100000000 records sent, 140619.880557 records/sec (268.21 MB/sec), 116.07 ms avg latency, 3480.00 ms max latency, 2 ms 50th, 210 ms 95th, 451 ms 99th, 1661 ms 99.9th.

Consumer pressure test

 start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 2018-05-24 17:44:44:025, 2018-05-24 17:48:14:125, 75859.4952, 361.0638, 40722223, 193823.0509

Summary of pressure measurement

producer

Message volume	Production rate	average delay	Maximum Latency	50%	95%	99%	99.9%
ten thousand	18.46 MB/sec	27.31 ms	247.00 ms	2ms	92 ms	108 ms	112 ms
ten million	224.59 MB/sec	137.72 ms	2789.00 ms	92 ms	743 ms	878 ms	2773 ms
one hundred million	268.21 MB/sec	116.07 ms	3480.00 ms	2 ms	210 ms	451 ms	1661 ms

consumer

start time	End time	Total capacity (MB)	Capacity rate (MB/s)	Total Messages (nMsg)	Message rate (nMsg/s)
2018-05-24 18:39:30:930	2018-05-24 18:39:31:061	zero point nine five three seven	seven point two eight zero zero	ten thousand	seventy-six thousand three hundred and thirty-five point eight seven seven nine
2018-05-24 17:40:51:586	2018-05-24 17:41:27:446	seventeen thousand two hundred and sixty-one point five zero five one	four hundred and eighty-one point three five eight two	ten million	two hundred and seventy-eight thousand eight hundred and sixty-two point two four two one
2018-05-24 18:27:52:075	2018-05-24 18:35:19:864	one hundred and eighty-eight thousand nine hundred and twenty-two point eight eight two one	four hundred and twenty-one point nine zero one six	one hundred million	two hundred and twenty-three thousand three hundred and nineteen point four six five two

The following conclusions are drawn:

We have 3 replicas and 15 partitions: each partition The average ingestion capacity was 15 MB/s, and the average throughput was 30 MB/s;
The greater the production, the greater the delay. Another reason for the high delay is replica synchronization;
Consumption is not a bottleneck. There is a bottleneck in production. The production bottleneck comes from the disk write speed;

This test is not sufficient, but it is also sufficient. The purpose is to know roughly where the ceiling of the kafka cluster is, and know the endurance of the cluster, so as to facilitate future expansion.

This article is written by Chakhsu Lau Creation, adoption Knowledge Sharing Attribution 4.0 International License Agreement.
All articles on this website are original or translated by this website, except for the reprint/source. Please sign your name before reprinting.