原文链接: https://nsq.io/overview/performance.html

Distributed Performance 分布式性能

The main repo contains a script (bench/bench.py) that automates a distributed benchmark on EC2.

​ 主代码仓库中包含一个脚本(bench/ benchmark .py ),它在亚马逊 EC2上自动执行分布式基准测试。

It bootstraps N nodes, some running nsqd and some running load-generating utilities (PUB and SUB), and then parses their output to provide an aggregate total.

它引导 N 节点,一些运行 nsqd 和一些运行负载生成实用程序(生产者 消费者 ),然后解析它们的输出以提供总输出。

Setup 设置

The following runs reflect the default parameters of 6 c3.2xlarge, the cheapest instance type that supports 1gbit links. 3 nodes run an nsqd instance and the rest run instances of bench_reader (SUB) and bench_writer (PUB), to generate load depending on the benchmark mode.

下面在6台 亚马逊 计算型 c3.2xlarge实例类型运行默认参数,支持1gbit链接的最便宜的实例类型。

3个节点运行一个“nsqd”实例,其余运行读取(消费者模型)和写入(消费者模型)实例,根据基准模式生成负载。

$ ./bench/bench.py --access-key=... --secret-key=... --ssh-key-name=...
[I 140917 10:58:10 bench:102] launching 6 instances
[I 140917 10:58:12 bench:111] waiting for instances to launch...
...
[I 140917 10:58:37 bench:130] (1) bootstrapping ec2-54-160-145-64.compute-1.amazonaws.com (i-0a018ce1)
[I 140917 10:59:37 bench:130] (2) bootstrapping ec2-54-90-195-149.compute-1.amazonaws.com (i-0f018ce4)
[I 140917 11:00:00 bench:130] (3) bootstrapping ec2-23-22-236-55.compute-1.amazonaws.com (i-0e018ce5)
[I 140917 11:00:41 bench:130] (4) bootstrapping ec2-23-23-40-113.compute-1.amazonaws.com (i-0d018ce6)
[I 140917 11:01:10 bench:130] (5) bootstrapping ec2-54-226-180-44.compute-1.amazonaws.com (i-0c018ce7)
[I 140917 11:01:43 bench:130] (6) bootstrapping ec2-54-90-83-223.compute-1.amazonaws.com (i-10018cfb)

Producer Throughput 生产者吞吐量

This benchmark measures just producer throughput, with no additional load. The message size is 100 bytes and messages are distributed over 3 topics.

这个基准测试只测量生产者吞吐量,没有额外的负载。消息大小为100字节,消息分布在3个主题上。

$ ./bench/bench.py --access-key=... --secret-key=... --ssh-key-name=... --mode=pub --msg-size=100 run
[I 140917 12:39:37 bench:140] launching nsqd on 3 host(s)
[I 140917 12:39:41 bench:163] launching 9 producer(s) on 3 host(s)
...
[I 140917 12:40:20 bench:248] [bench_writer] 10.002s - 197.463mb/s - 2070549.631ops/s - 4.830us/op

An ingress of ~2.07mm msgs/sec, consuming an aggregate 197mb/s of bandwidth.

一个~2.07mm 消息/秒的入口,消耗总带宽197mb/s

Producer and Consumer Throughput 生产者和消费者吞吐量

This benchmark more accurately reflects real-world conditions by servicing both producers and consumers. Again, the message size is 100 bytes and messages are distributed over 3 topics, each with a single channel (24 clients per channel).

通过同时为生产者和消费者提供服务,该基准更准确地反映了实际情况。同样,消息大小为100字节,消息分布在3个主题上,每个主题都有一个通道(每个通道有24个客户机)。

$ ./bench/bench.py --access-key=... --secret-key=... --ssh-key-name=... --msg-size=100 run
[I 140917 12:41:11 bench:140] launching nsqd on 3 host(s)
[I 140917 12:41:15 bench:163] launching 9 producer(s) on 3 host(s)
[I 140917 12:41:22 bench:186] launching 9 consumer(s) on 3 host(s)
...
[I 140917 12:41:55 bench:248] [bench_reader] 10.252s - 76.946mb/s - 806838.610ops/s - 12.706us/op
[I 140917 12:41:55 bench:248] [bench_writer] 10.030s - 80.315mb/s - 842149.615ops/s - 11.910us/op

At an ingress of ~842k and egress of ~806k msgs/s, consuming an aggregate 156mb/s of bandwidth, we’re now maxing out the CPU capacity on the nsqd nodes. By introducing consumers, nsqd needs to maintain per-channel in-flight accounting so the load is naturally higher.

~842k的入口和~806k msgs/s的出口,消耗了总156mb/s的带宽,我们现在正在最大限度地使用nsqd节点上的CPU容量。通过引入消费者,nsqd需要维护每个频道的动态计算,因此负载自然会更高。

The consumer numbers are slightly lower than producers because consumers send twice the number of commands as producers (a FIN command must be sent for each message), impacting throughput.

消费者数量略低于生产者,因为消费者发送的命令数量是生产者的两倍(必须为每个消息发送一个“FIN”命令),这会影响吞吐量。

Adding another 2 nodes (one nsqd and one load-generating) attains over 1mm msgs/s:

添加另外两个节点(一个“nsqd”和一个负载生成节点)可以获得超过1mm msgs/s:

$ ./bench/bench.py --access-key=... --secret-key=... --ssh-key-name=... --msg-size=100 run
[I 140917 13:38:28 bench:140] launching nsqd on 4 host(s)
[I 140917 13:38:32 bench:163] launching 16 producer(s) on 4 host(s)
[I 140917 13:38:43 bench:186] launching 16 consumer(s) on 4 host(s)
...
[I 140917 13:39:12 bench:248] [bench_reader] 10.561s - 100.956mb/s - 1058624.012ops/s - 9.976us/op
[I 140917 13:39:12 bench:248] [bench_writer] 10.023s - 105.898mb/s - 1110408.953ops/s - 9.026us/op

Single Node Performance 单节点性能

DISCLAIMER: Please keep in mind that NSQ is designed to be used in a distributed fashion.

​ 免责声明:请记住NSQ是设计用于分布式方式的。

Single node performance is important, but not the end-all-be-all of what we’re looking to achieve.

​ 单节点性能很重要,但不是我们希望实现的最终目标。

Also, benchmarks are stupid, but here’s a few anyway to ignite the flame:

​ 另外,基准测试是愚蠢的,但无论如何都要有人来做

  • 2012 MacBook Air i7 2ghz

计算机型号

  • go1.2

go语言版本

  • NSQ v0.2.24

nsq 版本

  • 200 byte messages

消息量

GOMAXPROCS=1 (single publisher, single consumer)

并发 Go语言默认CPU 硬件线程数量(单 生产者 ,单 消费者 )

$ ./bench.sh 
results...
PUB: 2014/01/12 22:09:08 duration: 2.311925588s - 82.500mb/s - 432539.873ops/s - 2.312us/op
SUB: 2014/01/12 22:09:19 duration: 6.009749983s - 31.738mb/s - 166396.273ops/s - 6.010us/op

GOMAXPROCS=4 (4 publishers, 4 consumers)

并发 Go语言默认CPU 硬件线程数量(4 生产者 ,4 消费者 )

$ ./bench.sh 
results...
PUB: 2014/01/13 16:58:05 duration: 1.411492441s - 135.130mb/s - 708469.965ops/s - 1.411us/op
SUB: 2014/01/13 16:58:16 duration: 5.251380583s - 36.321mb/s - 190426.114ops/s - 5.251us/op
最后修改:2021 年 02 月 19 日 10 : 23 PM
如果觉得我的文章对你有用,请随意赞赏