kafka

• 启动3节点kafka集群
• kafka性能测试
  • kafka-producer-perf-test.sh脚本参数
  • kafka-consumer-perf-test.sh脚本参数
• 教程、资料
• kafkatool

启动3节点kafka集群

同一台机器可以copy三份配置文件server.properties，修改其中端口和路径即可。

• broker.id 每个kafka唯一
• listners=PLAINTEXT://192.168.3.29:9090 表示kakfa监听的地址和端口
• log.dirs=/xxx/kafka/logs 同一台机器的每个kafka节点不能相同
• zookeeper.connect=192.168.3.29:2181,192.168.3.29:2182,192.168.3.29:2182 zk集群节点

这样可以在同一台机器启动3个节点的kafka进行测试

jesse$ ./kafka-server-start.sh ../config/k1.properties

jesse$ ./kafka-server-start.sh ../config/k2.properties

jesse$ ./kafka-server-start.sh ../config/k3.properties

kafka性能测试

自带的性能测试工具：kafka-consumer-perf-test.sh、kafka-producer-perf-test.sh

测试producer性能：

~/kafka/bin$ ./kafka-producer-perf-test.sh --topic topic-t1 --num-records 1000000 --record-size 1000 --throughput 20000 --producer-props bootstrap.servers=192.168.3.29:9090,192.168.3.29:9091,192.168.3.29:9092

99982 records sent, 19992.4 records/sec (19.07 MB/sec), 92.4 ms avg latency, 483.0 ms max latency.
100071 records sent, 20014.2 records/sec (19.09 MB/sec), 0.7 ms avg latency, 10.0 ms max latency.
100029 records sent, 20005.8 records/sec (19.08 MB/sec), 0.8 ms avg latency, 11.0 ms max latency.
100020 records sent, 20000.0 records/sec (19.07 MB/sec), 0.8 ms avg latency, 12.0 ms max latency.
100040 records sent, 20004.0 records/sec (19.08 MB/sec), 0.8 ms avg latency, 11.0 ms max latency.
100020 records sent, 20004.0 records/sec (19.08 MB/sec), 0.8 ms avg latency, 11.0 ms max latency.
100020 records sent, 20000.0 records/sec (19.07 MB/sec), 0.8 ms avg latency, 17.0 ms max latency.
100032 records sent, 20006.4 records/sec (19.08 MB/sec), 0.8 ms avg latency, 16.0 ms max latency.
100008 records sent, 19997.6 records/sec (19.07 MB/sec), 0.8 ms avg latency, 10.0 ms max latency.
1000000 records sent, 19992.003199 records/sec (19.07 MB/sec), 9.95 ms avg latency, 483.00 ms max latency, 1 ms 50th, 2 ms 95th, 385 ms 99th, 473 ms 99.9th.

结果：发送100w记录，每秒发送2w，每秒平均向kafka写入19MB数据，平均时延9.95ms，最大时延483ms，50%的消息处理耗时1ms内，95%的2ms内，99%的385ms内。

测试consumer性能：

~/kafka/bin$ ./kafka-consumer-perf-test.sh --broker-list 192.168.3.29:9090,192.168.3.29:9091,192.168.3.29:9092 --topic topic-t1 --fetch-size 1000000 --messages 1000000 --threads 1

WARNING: option [threads] and [num-fetch-threads] have been deprecated and will be ignored by the test
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

2021-07-29 01:14:10:209, 2021-07-29 01:14:15:785, 953.6744, 171.0320, 1000007, 179341.2841, 370, 5206, 183.1875, 192087.3992

结果：5秒左右，消费数据953MB，每秒171MB，总消息数100w，每秒17w，rebalance时间370ms。

kafka-producer-perf-test.sh脚本参数

• --topic topic名称
• --num-records 总共需要发送的消息数
• --record-size 每个记录的字节数
• --throughput 每秒钟发送的记录数（限流控制，小于0表示不控制）
• --print-metrics 测试完成之后会打印很多指标信息
• --producer-props 生产者配置
  • bootstrap.servers=ip1:port1,ip2:port2,ip3:port3 kafka的服务端ip端口

kafka-consumer-perf-test.sh脚本参数

• --topic
• --fetch-size 每次fetch数据的大小
• --messages 总消费的消息数
• --threads 线程数

教程、资料

易百教程 (opens new window)

kafkatool

下载地址 (opens new window)

填写zk地址即可连接kafka集群