Kafka吞吐量测试案例
Kafka吞吐量测试案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
领英公司参考连接：https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600?qid=84026ff8-243f-49a7-a4d0-69976cf317b7&v=&b=&from_search=9(温馨提示，需要VPN哟～)。
一.kafka-producer-perf-test.sh 使用案例
1>.测试kafka写入的吞吐量
我用的kafka版本是0.10.2.1，领英用的貌似是0.10.0，我使用我的版本按照它的方式敲击，发现不太好事，这就有点小尴尬了！于是它表红线的参数我就没有使用。
[root@node105 ~]# nohup kafka-producer-perf-test.sh --num-records --record-size --topic yinzhengjie-kafka --throughput --producer-props bootstrap.servers=
0.1.2.101:,10.1.2.102:,10.1.2.103:,10.1.2.104:,10.1.2.105: > yinzhengjie-kafka.log &
[root@node105 ~]# 相关参数说明：
—topic
指定Kafka集群的topic名称，本例为yinzhengjie-kafka
--num-records
总共需要发送的消息数，本例为100000000 --record-size
每个记录的字节数，本例为1000 --throughput
每秒钟发送的记录数，本例为20000 --producer-props bootstrap.servers=10.1.2.101:,10.1.2.102:,10.1.2.103:,10.1.2.104:,10.1.2.105:
很显然是kafka集群的broker地址
2>.计算生产者的吞吐量
[root@node105 ~]# cat yinzhengjie-kafka.log
records sent, 96673.4 records/sec (92.19 MB/sec), 284.4 ms avg latency, 648.0 max latency.
records sent, 118340.6 records/sec (112.86 MB/sec), 260.7 ms avg latency, 795.0 max latency.
records sent, 113612.2 records/sec (108.35 MB/sec), 268.2 ms avg latency, 870.0 max latency.
records sent, 115034.1 records/sec (109.71 MB/sec), 268.7 ms avg latency, 732.0 max latency.
records sent, 106912.4 records/sec (101.96 MB/sec), 257.9 ms avg latency, 913.0 max latency.
records sent, 112575.7 records/sec (107.36 MB/sec), 294.4 ms avg latency, 1283.0 max latency.
records sent, 111598.0 records/sec (106.43 MB/sec), 279.4 ms avg latency, 828.0 max latency.
records sent, 92898.6 records/sec (88.60 MB/sec), 329.3 ms avg latency, 1213.0 max latency.
records sent, 97656.2 records/sec (93.13 MB/sec), 312.9 ms avg latency, 1020.0 max latency.
records sent, 103512.0 records/sec (98.72 MB/sec), 294.8 ms avg latency, 1062.0 max latency.
records sent, 41949.0 records/sec (40.01 MB/sec), 740.2 ms avg latency, 3722.0 max latency.
records sent, 78980.2 records/sec (75.32 MB/sec), 388.7 ms avg latency, 2079.0 max latency.
records sent, 89035.6 records/sec (84.91 MB/sec), 342.4 ms avg latency, 1274.0 max latency.
records sent, 138789.2 records/sec (132.36 MB/sec), 223.2 ms avg latency, 469.0 max latency.
records sent, 62616.3 records/sec (59.72 MB/sec), 247.3 ms avg latency, 3303.0 max latency.
records sent, 126282.4 records/sec (120.43 MB/sec), 381.9 ms avg latency, 3304.0 max latency.
records sent, 78668.6 records/sec (75.02 MB/sec), 386.0 ms avg latency, 1857.0 max latency.
records sent, 88207.0 records/sec (84.12 MB/sec), 347.2 ms avg latency, 1018.0 max latency.
records sent, 104223.6 records/sec (99.40 MB/sec), 296.4 ms avg latency, 698.0 max latency.
records sent, 127779.4 records/sec (121.86 MB/sec), 238.2 ms avg latency, 560.0 max latency.
records sent, 101747.5 records/sec (97.03 MB/sec), 296.9 ms avg latency, 855.0 max latency.
records sent, 116543.6 records/sec (111.14 MB/sec), 269.8 ms avg latency, 633.0 max latency.
records sent, 2589.8 records/sec (2.47 MB/sec), 262.5 ms avg latency, 6291.0 max latency.
records sent, 34393.3 records/sec (32.80 MB/sec), 954.0 ms avg latency, 6731.0 max latency.
records sent, 105786.4 records/sec (100.89 MB/sec), 595.4 ms avg latency, 5857.0 max latency.
records sent, 91770.3 records/sec (87.52 MB/sec), 335.8 ms avg latency, 1688.0 max latency.
records sent, 119815.1 records/sec (114.26 MB/sec), 245.9 ms avg latency, 931.0 max latency.
records sent, 63321.6 records/sec (60.39 MB/sec), 502.0 ms avg latency, 2900.0 max latency.
records sent, 116084.8 records/sec (110.71 MB/sec), 262.5 ms avg latency, 540.0 max latency.
records sent, 125801.0 records/sec (119.97 MB/sec), 244.7 ms avg latency, 517.0 max latency.
records sent, 97554.6 records/sec (93.04 MB/sec), 298.4 ms avg latency, 1026.0 max latency.
records sent, 18197.9 records/sec (17.35 MB/sec), 1709.4 ms avg latency, 4843.0 max latency.
records sent, 86022.6 records/sec (82.04 MB/sec), 370.9 ms avg latency, 1119.0 max latency.
records sent, 63720.6 records/sec (60.77 MB/sec), 478.7 ms avg latency, 2001.0 max latency.
records sent, 76732.5 records/sec (73.18 MB/sec), 379.7 ms avg latency, 1345.0 max latency.
records sent, 82659.4 records/sec (78.83 MB/sec), 391.5 ms avg latency, 1016.0 max latency.
records sent, 51729.2 records/sec (49.33 MB/sec), 528.6 ms avg latency, 2228.0 max latency.
records sent, 87490.0 records/sec (83.44 MB/sec), 377.5 ms avg latency, 1178.0 max latency.
records sent, 70227.4 records/sec (66.97 MB/sec), 424.4 ms avg latency, 1456.0 max latency.
records sent, 80437.4 records/sec (76.71 MB/sec), 394.4 ms avg latency, 1452.0 max latency.
records sent, 81178.1 records/sec (77.42 MB/sec), 383.9 ms avg latency, 1344.0 max latency.
records sent, 66965.4 records/sec (63.86 MB/sec), 426.7 ms avg latency, 2243.0 max latency.
records sent, 81169.8 records/sec (77.41 MB/sec), 406.3 ms avg latency, 1204.0 max latency.
records sent, 78674.2 records/sec (75.03 MB/sec), 369.5 ms avg latency, 980.0 max latency.
records sent, 74235.5 records/sec (70.80 MB/sec), 376.5 ms avg latency, 1539.0 max latency.
records sent, 44141.1 records/sec (42.10 MB/sec), 730.9 ms avg latency, 2348.0 max latency.
records sent, 56142.7 records/sec (53.54 MB/sec), 537.1 ms avg latency, 1441.0 max latency.
records sent, 69909.2 records/sec (66.67 MB/sec), 480.5 ms avg latency, 1660.0 max latency.
records sent, 66674.6 records/sec (63.59 MB/sec), 437.3 ms avg latency, 1390.0 max latency.
records sent, 72936.6 records/sec (69.56 MB/sec), 440.6 ms avg latency, 1651.0 max latency.
records sent, 76708.8 records/sec (73.16 MB/sec), 326.2 ms avg latency, 1466.0 max latency.
records sent, 76875.4 records/sec (73.31 MB/sec), 468.7 ms avg latency, 1773.0 max latency.
records sent, 62170.6 records/sec (59.29 MB/sec), 492.0 ms avg latency, 1382.0 max latency.
records sent, 65983.6 records/sec (62.93 MB/sec), 432.4 ms avg latency, 1238.0 max latency.
records sent, 79912.1 records/sec (76.21 MB/sec), 420.5 ms avg latency, 1244.0 max latency.
records sent, 66623.8 records/sec (63.54 MB/sec), 445.0 ms avg latency, 1054.0 max latency.
records sent, 70929.0 records/sec (67.64 MB/sec), 413.4 ms avg latency, 1165.0 max latency.
records sent, 61176.8 records/sec (58.34 MB/sec), 536.8 ms avg latency, 1517.0 max latency.
records sent, 85953.5 records/sec (81.97 MB/sec), 354.6 ms avg latency, 1919.0 max latency.
records sent, 80229.4 records/sec (76.51 MB/sec), 363.1 ms avg latency, 1194.0 max latency.
records sent, 60894.0 records/sec (58.07 MB/sec), 509.9 ms avg latency, 1365.0 max latency.
records sent, 77083.6 records/sec (73.51 MB/sec), 417.5 ms avg latency, 1468.0 max latency.
records sent, 80059.2 records/sec (76.35 MB/sec), 343.1 ms avg latency, 1147.0 max latency.
records sent, 83940.4 records/sec (80.05 MB/sec), 402.7 ms avg latency, 1136.0 max latency.
records sent, 54539.6 records/sec (52.01 MB/sec), 554.7 ms avg latency, 2612.0 max latency.
records sent, 65514.8 records/sec (62.48 MB/sec), 452.0 ms avg latency, 1112.0 max latency.
records sent, 88871.4 records/sec (84.75 MB/sec), 304.7 ms avg latency, 1523.0 max latency.
records sent, 65193.8 records/sec (62.17 MB/sec), 546.3 ms avg latency, 2538.0 max latency.
records sent, 78229.4 records/sec (74.61 MB/sec), 380.9 ms avg latency, 1704.0 max latency.
records sent, 83671.0 records/sec (79.79 MB/sec), 380.5 ms avg latency, 1402.0 max latency.
records sent, 63656.4 records/sec (60.71 MB/sec), 465.3 ms avg latency, 2996.0 max latency.
records sent, 85916.7 records/sec (81.94 MB/sec), 366.3 ms avg latency, 1606.0 max latency.
records sent, 79776.6 records/sec (76.08 MB/sec), 380.9 ms avg latency, 1387.0 max latency.
records sent, 109791.4 records/sec (104.71 MB/sec), 270.6 ms avg latency, 1512.0 max latency.
records sent, 96645.0 records/sec (92.17 MB/sec), 317.9 ms avg latency, 988.0 max latency.
records sent, 58519.4 records/sec (55.81 MB/sec), 473.6 ms avg latency, 2136.0 max latency.
records sent, 67665.8 records/sec (64.53 MB/sec), 491.5 ms avg latency, 2083.0 max latency.
records sent, 84384.6 records/sec (80.48 MB/sec), 384.9 ms avg latency, 1260.0 max latency.
records sent, 73026.0 records/sec (69.64 MB/sec), 410.0 ms avg latency, 1084.0 max latency.
records sent, 69260.8 records/sec (66.05 MB/sec), 456.2 ms avg latency, 1526.0 max latency.
records sent, 69544.0 records/sec (66.32 MB/sec), 420.3 ms avg latency, 1192.0 max latency.
records sent, 74969.2 records/sec (71.50 MB/sec), 421.8 ms avg latency, 1478.0 max latency.
records sent, 66404.8 records/sec (63.33 MB/sec), 451.3 ms avg latency, 1840.0 max latency.
records sent, 82079.6 records/sec (78.28 MB/sec), 355.0 ms avg latency, 1328.0 max latency.
records sent, 91202.6 records/sec (86.98 MB/sec), 362.7 ms avg latency, 1106.0 max latency.
records sent, 85744.9 records/sec (81.77 MB/sec), 302.0 ms avg latency, 1187.0 max latency.
records sent, 78694.1 records/sec (75.05 MB/sec), 431.8 ms avg latency, 1545.0 max latency.
records sent, 67820.7 records/sec (64.68 MB/sec), 443.9 ms avg latency, 1530.0 max latency.
records sent, 84655.6 records/sec (80.73 MB/sec), 304.3 ms avg latency, 1762.0 max latency.
records sent, 66952.7 records/sec (63.85 MB/sec), 546.6 ms avg latency, 2097.0 max latency.
records sent, 78982.8 records/sec (75.32 MB/sec), 402.1 ms avg latency, 1448.0 max latency.
records sent, 57679.3 records/sec (55.01 MB/sec), 517.6 ms avg latency, 2158.0 max latency.
records sent, 74469.4 records/sec (71.02 MB/sec), 397.6 ms avg latency, 1819.0 max latency.
records sent, 81696.0 records/sec (77.91 MB/sec), 399.9 ms avg latency, 1226.0 max latency.
records sent, 73579.8 records/sec (70.17 MB/sec), 423.3 ms avg latency, 1575.0 max latency.
records sent, 54090.0 records/sec (51.58 MB/sec), 513.7 ms avg latency, 1724.0 max latency.
records sent, 78276.1 records/sec (74.65 MB/sec), 387.1 ms avg latency, 1762.0 max latency.
records sent, 87411.2 records/sec (83.36 MB/sec), 380.6 ms avg latency, 1763.0 max latency.
records sent, 69182.0 records/sec (65.98 MB/sec), 421.1 ms avg latency, 1361.0 max latency.
records sent, 87202.2 records/sec (83.16 MB/sec), 359.5 ms avg latency, 1487.0 max latency.
records sent, 75623.6 records/sec (72.12 MB/sec), 265.6 ms avg latency, 2355.0 max latency.
records sent, 67005.2 records/sec (63.90 MB/sec), 625.9 ms avg latency, 2568.0 max latency.
records sent, 75301.2 records/sec (71.81 MB/sec), 389.1 ms avg latency, 1071.0 max latency.
records sent, 56347.6 records/sec (53.74 MB/sec), 541.7 ms avg latency, 2047.0 max latency.
records sent, 66255.5 records/sec (63.19 MB/sec), 461.9 ms avg latency, 1747.0 max latency.
records sent, 69244.4 records/sec (66.04 MB/sec), 408.8 ms avg latency, 1318.0 max latency.
records sent, 53079.4 records/sec (50.62 MB/sec), 630.5 ms avg latency, 1979.0 max latency.
records sent, 84722.4 records/sec (80.80 MB/sec), 387.0 ms avg latency, 1130.0 max latency.
records sent, 60500.6 records/sec (57.70 MB/sec), 504.8 ms avg latency, 1415.0 max latency.
records sent, 71213.9 records/sec (67.91 MB/sec), 397.4 ms avg latency, 1366.0 max latency.
records sent, 76887.6 records/sec (73.33 MB/sec), 405.1 ms avg latency, 1269.0 max latency.
records sent, 87168.4 records/sec (83.13 MB/sec), 378.4 ms avg latency, 1388.0 max latency.
records sent, 70801.7 records/sec (67.52 MB/sec), 411.4 ms avg latency, 1323.0 max latency.
records sent, 89877.4 records/sec (85.71 MB/sec), 355.8 ms avg latency, 1039.0 max latency.
records sent, 77376.8 records/sec (73.79 MB/sec), 340.2 ms avg latency, 1702.0 max latency.
records sent, 80242.9 records/sec (76.53 MB/sec), 421.3 ms avg latency, 1226.0 max latency.
records sent, 72686.2 records/sec (69.32 MB/sec), 417.1 ms avg latency, 1447.0 max latency.
records sent, 87568.4 records/sec (83.51 MB/sec), 359.2 ms avg latency, 2027.0 max latency.
records sent, 89986.6 records/sec (85.82 MB/sec), 343.0 ms avg latency, 1238.0 max latency.
records sent, 93639.0 records/sec (89.30 MB/sec), 318.6 ms avg latency, 887.0 max latency.
records sent, 76260.0 records/sec (72.73 MB/sec), 416.3 ms avg latency, 1537.0 max latency.
records sent, 77205.0 records/sec (73.63 MB/sec), 398.5 ms avg latency, 1199.0 max latency.
records sent, 71027.8 records/sec (67.74 MB/sec), 431.3 ms avg latency, 1298.0 max latency.
records sent, 74770.4 records/sec (71.31 MB/sec), 353.8 ms avg latency, 1290.0 max latency.
records sent, 79481.4 records/sec (75.80 MB/sec), 438.1 ms avg latency, 1648.0 max latency.
records sent, 72144.0 records/sec (68.80 MB/sec), 410.4 ms avg latency, 1585.0 max latency.
records sent, 72849.8 records/sec (69.47 MB/sec), 413.8 ms avg latency, 1505.0 max latency.
records sent, 80272.6 records/sec (76.55 MB/sec), 388.3 ms avg latency, 1054.0 max latency.
records sent, 59220.0 records/sec (56.48 MB/sec), 495.4 ms avg latency, 1338.0 max latency.
records sent, 59626.8 records/sec (56.86 MB/sec), 551.7 ms avg latency, 1610.0 max latency.
records sent, 59134.6 records/sec (56.40 MB/sec), 522.3 ms avg latency, 1689.0 max latency.
records sent, 77297.6 records/sec (73.72 MB/sec), 400.0 ms avg latency, 1311.0 max latency.
records sent, 84495.4 records/sec (80.58 MB/sec), 359.9 ms avg latency, 1357.0 max latency.
records sent, 72980.6 records/sec (69.60 MB/sec), 405.4 ms avg latency, 1113.0 max latency.
records sent, 63415.4 records/sec (60.48 MB/sec), 451.5 ms avg latency, 1377.0 max latency.
records sent, 81400.6 records/sec (77.63 MB/sec), 390.1 ms avg latency, 1321.0 max latency.
records sent, 77076.2 records/sec (73.51 MB/sec), 408.1 ms avg latency, 1285.0 max latency.
records sent, 77760.2 records/sec (74.16 MB/sec), 396.4 ms avg latency, 1162.0 max latency.
records sent, 76613.8 records/sec (73.06 MB/sec), 417.4 ms avg latency, 1343.0 max latency.
records sent, 73735.4 records/sec (70.32 MB/sec), 391.5 ms avg latency, 2222.0 max latency.
records sent, 77178.0 records/sec (73.60 MB/sec), 413.3 ms avg latency, 1166.0 max latency.
records sent, 88910.2 records/sec (84.79 MB/sec), 353.1 ms avg latency, 1147.0 max latency.
records sent, 69296.5 records/sec (66.09 MB/sec), 438.3 ms avg latency, 1684.0 max latency.
records sent, 62508.9 records/sec (59.61 MB/sec), 446.3 ms avg latency, 1352.0 max latency.
records sent, 86085.8 records/sec (82.10 MB/sec), 387.1 ms avg latency, 1173.0 max latency.
records sent, 74890.4 records/sec (71.42 MB/sec), 385.6 ms avg latency, 1550.0 max latency.
records sent, 67133.2 records/sec (64.02 MB/sec), 453.3 ms avg latency, 1589.0 max latency.
records sent, 73993.1 records/sec (70.57 MB/sec), 412.4 ms avg latency, 1358.0 max latency.
records sent, 70586.5 records/sec (67.32 MB/sec), 456.5 ms avg latency, 1546.0 max latency.
records sent, 55046.4 records/sec (52.50 MB/sec), 512.0 ms avg latency, 1688.0 max latency.
records sent, 88498.0 records/sec (84.40 MB/sec), 370.3 ms avg latency, 1130.0 max latency.
records sent, 62622.0 records/sec (59.72 MB/sec), 498.3 ms avg latency, 1636.0 max latency.
records sent, 80061.6 records/sec (76.35 MB/sec), 395.6 ms avg latency, 1373.0 max latency.
records sent, 66512.8 records/sec (63.43 MB/sec), 395.0 ms avg latency, 1690.0 max latency.
records sent, 89393.4 records/sec (85.25 MB/sec), 386.0 ms avg latency, 1827.0 max latency.
records sent, 87988.0 records/sec (83.91 MB/sec), 355.3 ms avg latency, 1251.0 max latency.
records sent, 52464.4 records/sec (50.03 MB/sec), 516.0 ms avg latency, 1680.0 max latency.
records sent, 62918.6 records/sec (60.00 MB/sec), 544.6 ms avg latency, 1431.0 max latency.
records sent, 61985.3 records/sec (59.11 MB/sec), 481.0 ms avg latency, 1430.0 max latency.
records sent, 79951.4 records/sec (76.25 MB/sec), 386.0 ms avg latency, 1160.0 max latency.
records sent, 80193.0 records/sec (76.48 MB/sec), 381.1 ms avg latency, 1262.0 max latency.
records sent, 69998.5 records/sec (66.76 MB/sec), 398.8 ms avg latency, 1203.0 max latency.
records sent, 107223.2 records/sec (102.26 MB/sec), 317.9 ms avg latency, 1063.0 max latency.
records sent, 60962.7 records/sec (58.14 MB/sec), 425.6 ms avg latency, 2061.0 max latency.
records sent, 48151.1 records/sec (45.92 MB/sec), 674.7 ms avg latency, 1933.0 max latency.
records sent, 40023.9 records/sec (38.17 MB/sec), 752.9 ms avg latency, 1802.0 max latency.
records sent, 51344.2 records/sec (48.97 MB/sec), 622.6 ms avg latency, 1677.0 max latency.
records sent, 37986.2 records/sec (36.23 MB/sec), 762.4 ms avg latency, 1840.0 max latency.
records sent, 96311.1 records/sec (91.85 MB/sec), 308.9 ms avg latency, 1841.0 max latency.
records sent, 76401.7 records/sec (72.86 MB/sec), 443.2 ms avg latency, 1543.0 max latency.
records sent, 80602.9 records/sec (76.87 MB/sec), 356.3 ms avg latency, 1044.0 max latency.
records sent, 74727.6 records/sec (71.27 MB/sec), 430.1 ms avg latency, 1650.0 max latency.
records sent, 67295.4 records/sec (64.18 MB/sec), 438.0 ms avg latency, 1648.0 max latency.
records sent, 70173.4 records/sec (66.92 MB/sec), 458.6 ms avg latency, 1661.0 max latency.
records sent, 99843.6 records/sec (95.22 MB/sec), 312.3 ms avg latency, 1054.0 max latency.
records sent, 72428.6 records/sec (69.07 MB/sec), 433.5 ms avg latency, 1483.0 max latency.
records sent, 60269.1 records/sec (57.48 MB/sec), 491.1 ms avg latency, 1619.0 max latency.
records sent, 75138.4 records/sec (71.66 MB/sec), 422.2 ms avg latency, 1286.0 max latency.
records sent, 76908.7 records/sec (73.35 MB/sec), 374.8 ms avg latency, 1611.0 max latency.
records sent, 62428.9 records/sec (59.54 MB/sec), 464.3 ms avg latency, 1787.0 max latency.
records sent, 70842.0 records/sec (67.56 MB/sec), 484.1 ms avg latency, 2107.0 max latency.
records sent, 87147.0 records/sec (83.11 MB/sec), 306.1 ms avg latency, 1225.0 max latency.
records sent, 75395.6 records/sec (71.90 MB/sec), 433.9 ms avg latency, 1668.0 max latency.
records sent, 72348.6 records/sec (69.00 MB/sec), 453.9 ms avg latency, 1441.0 max latency.
records sent, 68590.2 records/sec (65.41 MB/sec), 397.4 ms avg latency, 1394.0 max latency.
records sent, 56012.5 records/sec (53.42 MB/sec), 593.2 ms avg latency, 1667.0 max latency.
records sent, 68109.8 records/sec (64.95 MB/sec), 480.5 ms avg latency, 1633.0 max latency.
records sent, 60750.4 records/sec (57.94 MB/sec), 451.4 ms avg latency, 1519.0 max latency.
records sent, 75778.0 records/sec (72.27 MB/sec), 423.8 ms avg latency, 1378.0 max latency.
records sent, 75078.8 records/sec (71.60 MB/sec), 403.5 ms avg latency, 1199.0 max latency.
records sent, 88334.3 records/sec (84.24 MB/sec), 367.1 ms avg latency, 935.0 max latency.
records sent, 87000.4 records/sec (82.97 MB/sec), 360.3 ms avg latency, 1660.0 max latency.
records sent, 69621.0 records/sec (66.40 MB/sec), 429.2 ms avg latency, 1201.0 max latency.
records sent, 75780.2 records/sec (72.27 MB/sec), 403.9 ms avg latency, 1311.0 max latency.
records sent, 48419.8 records/sec (46.18 MB/sec), 587.5 ms avg latency, 2074.0 max latency.
records sent, 75475.5 records/sec (71.98 MB/sec), 441.9 ms avg latency, 1329.0 max latency.
records sent, 59701.4 records/sec (56.94 MB/sec), 494.5 ms avg latency, 1573.0 max latency.
records sent, 74834.2 records/sec (71.37 MB/sec), 375.8 ms avg latency, 1409.0 max latency.
records sent, 84547.7 records/sec (80.63 MB/sec), 371.5 ms avg latency, 1183.0 max latency.
records sent, 72840.4 records/sec (69.47 MB/sec), 456.6 ms avg latency, 1347.0 max latency.
records sent, 81730.0 records/sec (77.94 MB/sec), 348.7 ms avg latency, 1212.0 max latency.
records sent, 77244.0 records/sec (73.67 MB/sec), 430.5 ms avg latency, 1499.0 max latency.
records sent, 118130.0 records/sec (112.66 MB/sec), 257.1 ms avg latency, 1053.0 max latency.
records sent, 63208.8 records/sec (60.28 MB/sec), 349.1 ms avg latency, 1879.0 max latency.
records sent, 81708.4 records/sec (77.92 MB/sec), 433.5 ms avg latency, 2531.0 max latency.
records sent, 67405.8 records/sec (64.28 MB/sec), 505.4 ms avg latency, 1556.0 max latency.
records sent, 73265.7 records/sec (69.87 MB/sec), 378.6 ms avg latency, 1670.0 max latency.
records sent, 72019.4 records/sec (68.68 MB/sec), 428.8 ms avg latency, 1715.0 max latency.
records sent, 80748.8 records/sec (77.01 MB/sec), 388.2 ms avg latency, 1085.0 max latency.
records sent, 76266.7 records/sec (72.73 MB/sec), 387.5 ms avg latency, 1689.0 max latency.
records sent, 78154.9 records/sec (74.53 MB/sec), 448.3 ms avg latency, 1543.0 max latency.
records sent, 77319.0 records/sec (73.74 MB/sec), 400.5 ms avg latency, 1478.0 max latency.
records sent, 84893.6 records/sec (80.96 MB/sec), 360.7 ms avg latency, 1372.0 max latency.
records sent, 63604.0 records/sec (60.66 MB/sec), 431.9 ms avg latency, 1677.0 max latency.
records sent, 66125.6 records/sec (63.06 MB/sec), 467.7 ms avg latency, 1670.0 max latency.
records sent, 41101.9 records/sec (39.20 MB/sec), 710.9 ms avg latency, 2273.0 max latency.
records sent, 48237.6 records/sec (46.00 MB/sec), 688.9 ms avg latency, 1916.0 max latency.
records sent, 88764.0 records/sec (84.65 MB/sec), 364.9 ms avg latency, 1812.0 max latency.
records sent, 142269.4 records/sec (135.68 MB/sec), 216.4 ms avg latency, 453.0 max latency.
records sent, 115460.6 records/sec (110.11 MB/sec), 263.4 ms avg latency, 1072.0 max latency.
records sent, 55510.7 records/sec (52.94 MB/sec), 212.9 ms avg latency, 3721.0 max latency.
records sent, 60705.7 records/sec (57.89 MB/sec), 761.2 ms avg latency, 3900.0 max latency.
records sent, 57390.0 records/sec (54.73 MB/sec), 592.4 ms avg latency, 2092.0 max latency.
records sent, 100423.1 records/sec (95.77 MB/sec), 305.8 ms avg latency, 1828.0 max latency.
records sent, 90549.6 records/sec (86.35 MB/sec), 340.5 ms avg latency, 1327.0 max latency.
records sent, 57789.0 records/sec (55.11 MB/sec), 258.8 ms avg latency, 3117.0 max latency.
records sent, 84780.6 records/sec (80.85 MB/sec), 556.3 ms avg latency, 3368.0 max latency.
records sent, 84988.1 records/sec (81.05 MB/sec), 315.4 ms avg latency, 1309.0 max latency.
records sent, 95053.0 records/sec (90.65 MB/sec), 363.2 ms avg latency, 1313.0 max latency.
records sent, 114770.9 records/sec (109.45 MB/sec), 264.1 ms avg latency, 861.0 max latency.
records sent, 112661.4 records/sec (107.44 MB/sec), 230.4 ms avg latency, 1321.0 max latency.
records sent, 137957.4 records/sec (131.57 MB/sec), 265.3 ms avg latency, 1400.0 max latency.
records sent, 79928.1 records/sec (76.23 MB/sec), 384.9 ms avg latency, 2438.0 max latency.
records sent, 73913.2 records/sec (70.49 MB/sec), 410.9 ms avg latency, 2295.0 max latency.
records sent, 86908.0 records/sec (82.88 MB/sec), 318.5 ms avg latency, 1511.0 max latency.
records sent, 117196.9 records/sec (111.77 MB/sec), 253.9 ms avg latency, 1139.0 max latency.
records sent, 73062.4 records/sec (69.68 MB/sec), 438.2 ms avg latency, 1618.0 max latency.
records sent, 108792.2 records/sec (103.75 MB/sec), 309.4 ms avg latency, 975.0 max latency.
records sent, 89811.4 records/sec (85.65 MB/sec), 338.8 ms avg latency, 1284.0 max latency.
records sent, 92292.2 records/sec (88.02 MB/sec), 336.2 ms avg latency, 1115.0 max latency.
records sent, 84438.9 records/sec (80.53 MB/sec), 360.2 ms avg latency, 1078.0 max latency.
records sent, 105462.8 records/sec (100.58 MB/sec), 288.8 ms avg latency, 1230.0 max latency.
records sent, 103215.0 records/sec (98.43 MB/sec), 300.2 ms avg latency, 1021.0 max latency.
records sent, 102607.3 records/sec (97.85 MB/sec), 299.7 ms avg latency, 1108.0 max latency.
records sent, 78436.8 records/sec (74.80 MB/sec), 377.8 ms avg latency, 1352.0 max latency.
records sent, 97715.3 records/sec (93.19 MB/sec), 261.6 ms avg latency, 1416.0 max latency.
records sent, 102411.8 records/sec (97.67 MB/sec), 325.1 ms avg latency, 1509.0 max latency.
records sent, 73980.4 records/sec (70.55 MB/sec), 422.2 ms avg latency, 1893.0 max latency.
records sent, 100791.8 records/sec (96.12 MB/sec), 332.4 ms avg latency, 1506.0 max latency.
records sent, 94407.8 records/sec (90.03 MB/sec), 321.2 ms avg latency, 1079.0 max latency.
records sent, 78434.325371 records/sec (74.80 MB/sec), 390.67 ms avg latency, 6731.00 ms max latency, ms 50th, ms 95th, ms 99th, ms .9th.
[root@node105 ~]#
[root@node105 ~]# cat yinzhengjie-kafka.log #查看Kafka写入日志
records sent, 78434.325371 records/sec (74.80 MB/sec), 390.67 ms avg latency, 6731.00 ms max latency, ms 50th, ms 95th, ms 99th, ms .9th. 当你执行完kafka-producer-perf-test.sh 脚本时，会输出类似上面的一行，从上面一行我们可以得出一些消息
>.每秒平均向kafka写入了74. MB的数据(大概是：78433条消息，计算方式：74.8**/);
>.每次写入的平均延迟为390.67毫秒；
>.最大延迟为6731.00毫秒；
>.87毫秒内占95%；
>.158毫秒内占99%；
>.275毫秒内占99.%；
二.kafka-consumer-perf-test.sh 使用案例
1>. 测试Kafka消费的吞吐量
[root@node105 ~]# nohup kafka-consumer-perf-test.sh --broker-list 10.1.2.102: --topic yinzhengjie-kafka --messages --fetch-size --threads > yinzhengjie-kafka-consumer.log &
[root@node105 ~]# 相关参数说明：
--topic
指定topic的名称，本例为 yinzhengjie-kafka --fetch-size
指定每次fetch的数据的大小，本例为1048576，也就是1M --messages
总共要消费的消息个数，本例为1000000，100w --topic
指定消费的topic --threads
指定消费的线程数为10
2>.计算消费者的吞吐量
[root@node105 ~]# cat yinzhengjie-kafka-consumer.log
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
-- :::, -- :::, 953.9165, 358.3458, , 375811.7956
[root@node105 ~]# 当你执行完 kafka-consumer-perf-test.sh 脚本时，会输出类似于上面的一行，从上面一行我们可以得出一些消息
>.共消费了953.9165M的数据；
>.消费速度为358.3458M/s；
>.总共消费了1000411 条消息；
>.每秒消费375811.7956条消息；
三.编写脚本案例
1>.执行脚本
[root@node105 ~]# cat yinzhengjie-kafka.sh
nohup kafka-consumer-perf-test.sh --broker-list 10.1.3.116:,10.1.3.117:,10.1.3.118:,10.1.3.119:,10.1.3.120: --topic yinzhengjie-kafka --messages --fetch-size --threads > /root/yinzhengjie-kafka-consumer.log & nohup kafka-producer-perf-test.sh --num-records --record-size --topic yinzhengjie-kafka --throughput --producer-props bootstrap.servers=10.1.3.116:,10.1.3.117:,10.1.3.118:,10.1.3.119:,10.1.3.120: > /root/yinzhengjie-kafka-producer.log & [root@node105 ~]#
2>.运行脚本时查看Kafka manager的监控
3>.运行结束后查看Kafka manager的监控
4>.查看我的Kafka性能测试结果
生产端：
总共需要发送的消息数：1亿条，每个记录的字节数是：1000字节，每秒钟发送的记录数是： 消费端：
模拟了10个消费者：进行消费 指定每次fetch的数据的大小：1M
总共要消费的消息个数：1亿条 生产端结论：
>.每秒平均向kafka写入了225. MB的数据；
>.每次写入的平均延迟为 .81毫秒；
>.最大延迟为2422.00毫秒； 消费端结论：
>. 共消费了95367.4316M的数据；
>.消费速度为225.2047M/s；
>.总共消费了1亿条消息；
>.每秒消费236144.2369条消息；
Kafka吞吐量测试案例的更多相关文章
- 提高 Kafka 吞吐量
提高 Kafka 吞吐量 1.了解分区的数据速率,以确保提供合适的数据保存空间 2.除非您有其他架构上的需要,否则在写 Topic 时请使用随机分区 3.如果 Consumers 运行的是比 Kafk ...
- Storm自带测试案例的运行
之前Storm安装之后,也知道了Storm的一些相关概念,那么怎么样才可以运行一个例子对Storm流式计算有一个感性的认识呢,那么下面来运行一个Storm安装目录自带的测试案例,我们的Storm安装在 ...
- Azure磁盘的吞吐量测试
Azure的高级存储具有吞吐量大,延迟低的特点,非常适合时间关键型的应用程序(如SQL Server, Oracle, Redis等). 但高级存储同时具有价格高的特点,用户往往对其实际的性能数据较为 ...
- 编写优美的GTest测试案例
http://www.cnblogs.com/coderzh/archive/2010/01/09/beautiful-testcase.html 使用gtest也有很长一段时间了,这期间也积累了一些 ...
- nginx整合tomcat集群并做session共享----测试案例
最近出于好奇心,研究了一下tomcat集群配置,并整合nginx,实现负载均衡,session共享,写篇记录,防止遗忘.---------菜鸡的自我修炼. 说明:博主采用一个web项目同时部署到两台t ...
- robotframework+selenium搭配chrome浏览器，web测试案例（搭建篇）
这两天发布版本 做的事情有点多,都没有时间努力学习了,先给自己个差评,今天折腾了一天, 把robotframework 和 selenium 还有appnium 都研究了一下 ,大概有个谱,先说说we ...
- 运行RF测试案例，显示unable to open socket to "localhost:56505" error: [Errno 10061] 错误，且关闭RF卡死的解决办法
问题描述: 执行WEB ui测试案例后,执行请他的测试案例显示unable to open socket to "localhost:56505" error: [Errno 10 ...
- 全网最详细Apache Kylin1.5安装(单节点)和测试案例
转:http://blog.itpub.net/30089851/viewspace-2121221/ 微视频链接: Apache Kylin初识 1.版本(当前实验版本组合,版本一定要兼容 ...
- gtest命令行测试案例
使用gtest编写的测试案例通常本身就是一个可执行文件,因此运行起来非常方便.同时,gtest也为我们提供了一系列的运行参数(环境变量.命令行参数或代码里指定),使得我们可以对案例的执行进行一些有效的 ...
随机推荐
- table-layui
本文章为原创文章,转载请注明出处 html <div class="layui-btn-group tableBtn"> <button class=" ...
- python爬虫requests模块
requests库的七个主要方法 1. requests.requests(method, url, **kwargs) 构造一个请求,支撑以下各方法的基础方法 method:请求方式,对应get/p ...
- Codeforces 719A 月亮
参考自:https://www.cnblogs.com/ECJTUACM-873284962/p/6395221.html A. Vitya in the Countryside time limit ...
- 图灵机器人API接口
调用图灵API接口实现人机交互 流程一: 注册 图灵机器人官网: http://www.tuling123.com/ 第一步: 先注册, 然后创建机器人, 拿到一个32位的key 编码方式 UTF-8 ...
- Tmutarakan Exams URAL - 1091（莫比乌斯函数 || 容斥）
题意: 求1 - s 中 找出k个数 使它们的gcd > 1 求这样的k个数的对数 解析: 从每个素数的倍数中取k个数 求方案数 然后素数组合,容斥一下重的 奇加偶减 莫比乌斯函数的直接套模 ...
- 「POJ-3608」Bridge Across Islands （旋转卡壳--求两凸包距离）
题目链接 POJ-3608 Bridge Across Islands 题意 依次按逆时针方向给出凸包,在两个凸包小岛之间造桥,求最小距离. 题解 旋转卡壳的应用之一:求两凸包的最近距离. 找到凸包 ...
- LOJ #2048. 「HNOI2016」最小公倍数
题意 有 \(n\) 个点,\(m\) 条边,每条边连接 \(u \Leftrightarrow v\) 且权值为 \((a, b)\) . 共有 \(q\) 次询问,每次询问给出 \(u, v, q ...
- iptable四表五链
链(内置): PREROUTING:对数据包作路由选择前应用此链中的规则: INPUT:进来的数据包应用此规则链中的策略: FORWARD:转发数据包时应用此规则链中的策略: OUTPUT:外出的数据 ...
- 【Luogu4707】重返现世（min-max容斥）
[Luogu4707]重返现世(min-max容斥) 题面 洛谷 求全集的\(k-max\)的期望 题解 \(min-max\)容斥的证明不难,只需要把所有元素排序之后考虑组合数的贡献,容斥系数先设出 ...
- 使用unittest单元测试框架对加法做单元测试
import unittest from parameterized import parameterized def cacl(a, b): return a+b class MyCacl(unit ...