MR 746 is a relatively simple code change, but it has important implications for BCH's UX and scalability.
tl;dr: BCHN will now by default relay transactions 10x faster than before. This will make BCHN's transaction relay much faster than ABC, though not as fast as BU. On the flip side, BCHN will require more bandwidth than ABC at low transaction throughput rates, but less than BU. BCHN has also added the -txbroadcastinterval command-line option to allow node operators to configure this behavior more precisely.
Background on inv message batching
The inv messages for transactions are responsible for the vast majority of a node's total network traffic, since inv messages are sent per peer regardless of whether that peer needs the transaction or not. If a node is sending one inv per tx per peer, and the node has 50 peers, that results in about 120 bytes/tx/peer * 50 peers = 6000 bytes of network traffic per transaction, or roughly 6x as much traffic as is required for receiving and sending the transaction itself (assuming a typical 500 byte transaction). In this scenario, about 2/3 of the traffic is actually protocol overhead -- mostly TCP/IP overhead, but also bitcoin p2p protocol overhead. This overhead can be mitigated with batching. Using IPv4, inv messages take up about 80 + 40n bytes per message, where n is the number of transaction hashes being sent. This means that batching behavior can reduce the average traffic per peer per tx from around 120 bytes (for a batch size of 1) to around 44 bytes (for a batch size of 10). The way to achieve large batch sizes is to have a wait time before sending each tx inv to allow extra txs to be ready in each transmission. A batch size of 10 can be achieved with 20 tx/sec and an average wait time of 2 sec, for example.
If the wait time is mismatched with the transaction throughput rate, it can result in an increase in transaction propagation time without any substantial improvement in bandwidth efficiency. For example, the transaction throughput rate is currently about 0.25 tx/sec. If the wait time is 1 sec, then the typicalinv message will contain only 1 tx hash. If the wait time is increased to 2 sec, then the typical inv message will still contain only 1 tx hash. That extra second of delay would make a difference in batch size if the transaction throughput rate were 1 tx/sec, but it makes almost no difference at 0.25 tx/sec.
On the other hand, delays like this will significantly slow down transaction propagation, and will adversely affect UX. A delay of 2 seconds per connection with 20 connections means that a node will broadcast an inv to one peer every 100 ms on average. This delay, coupled with the natural network ping times of ~100 ms, and the 1.5 round-trip time (RTT) communication needed to send a tx, means that the number of nodes who have a transaction will double roughly every (100 ms + 150 ms) = 250 ms if all network nodes used a 2 second delay with 20 peers. If the network is comprised of 8192 nodes, it would take around 3.25 seconds for a transaction to propagate through the full netowrk. This worsens BCH's UX by making transactions feel slower in 0-conf mode, and gives a much larger time window for double-spends than is desirable. Shorter delays before broadcasting inv messages mean a better experience for BCH users, and slightly better 0-conf security.
Changes in !746
With !746, BCHN makes two major changes:
The default behavior for BCHN will be to wait an average of 500 ms between inv broadcasts, rather than 5000 ms. Furthermore, this value can be configured at the command-line with the -txbroadcastinterval=<ms> command-line option. Lower values (shorter delays) can accelerate transaction propagation; higher values (longer delays) can reduce upstream traffic for your node and downstream traffic for your node's peers. As before, this value halved for outgoing connections: with the default value of -txbroadcastinterval=500, BCHN will make on average one batched broadcast every 500 ms on incoming connections, and one every 250 ms on outgoing connections.
BCHN also adds a new option to configure the maximum throughput for inv announcement messages: -txbroadcastrate=<tx/sec/MB>. The default value for this option is the same as before: for each 1 MB of the block size limit, BCHN will allow 7 tx/sec on incoming connections (14 tx/sec/MB on outgoing connections). This allows a maximum-sized block (e.g. 32 MB) to be filled with 238-byte average sized transactions in 600 seconds via an incoming connection, or 300 seconds via an outgoing connection. Larger transaction sizes will result in less time being needed to reliably fill blocks. Most users will not need to touch this setting, as the default value is reasonable for most use cases. This limit provides protection against spam attacks while still allowing normal network operation to reliably fill blocks. However, modifying this setting can be useful in some circumstances, such as during stress tests, or for severely bandwidth-constrained applications. Users who are doing stress testing may wish to set this value very high in order to delimit tx broadcasting (e.g. -txbroadcastrate=9999999). Users with non-Starlink-based satellite internet (e.g. slow upstream, fast downstream) may wish to experiment with setting this to zero (i.e. -txbroadcastrate=0) to avoid relaying transactions, as an alternative to -blocksonly mode.
Comparison to other nodes
Bitcoin ABC and Bitcoin Core use an average delay of 5 seconds between broadcasts, which makes batching start at around 0.2 tx/sec. Bitcoin Unlimited uses a delay of about 10 ms, so BU's batching begins at around 100 tx/sec. BCHN will now default to using a delay of about 500 ms, or batching starting at about 2 tx/sec.
In practice, the Bitcoin Unlimited nodes on the BCH network dominate transaction propagation. While !746 makes BCHN about 10x faster at propagating transactions, it will likely only make BCH about 10% faster overall because the majority of transaction propagation is done by Bitcoin Unlimited. However, since BU nodes comprise a little less than half of all BCH nodes, there is still a significant chance for BU or non-BU nodes to form network islands, and for some nodes to have no contiguous BU-to-BU node paths to the original network sender. !746 makes BCH less reliant on BU nodes for fast transaction propagation, and makes BCH more resilient and reliably fast.
Acknowledgements
Thanks to mtrycz for writing the functional tests and to mtrycz, freetrader and Calin Culianu for code review.
No comments:
Post a Comment