Grokking Bitcoin

Suppose you want to add a new “feature” to Bitcoin Core’s networking code. You want to add a new network message type called kill that one Bitcoin node can send to another Bitcoin node. This message’s recipient node will immediately shut itself down. Only New nodes will know how to deal with an incoming kill message. Old nodes will ignore the—for them—unknown message (Figure 2).

Most people consider your change a huge security risk. They don’t want their nodes shut down by a random stranger on the internet. You’ll have a hard time convincing them to use Bitcoin New. You can’t force this software on anyone; people will have to actively want it and install it for Bitcoin New to get network-wide adoption.

Stupid changes like the kill message won’t make it in the world of open source.

As described in Section 11.1, a hard fork is a software change that relaxes the consensus rules. New blocks, created by New nodes, might be rejected by Old nodes. In the example with the vegetarian restaurant, a hard fork would be when the vegetarian restaurant starts to serve meat.

Suppose you create a fork that changes the maximum allowed block weight—discussed in [increasing-the-block-size-limit]—from 4,000,000 WU to 8,000,000 WU. This would allow for more transactions to be stuffed into each block. On the other hand, a higher limit could negatively affect some nodes in the Bitcoin network, as we talked about in [ch10].

Anyhow, you make this change and start using it in the Bitcoin network. When your node receives a block from a Bitcoin Old node, you’ll accept it because the block is definitely ≤ 8,000,000 WU; the Old node won’t create or relay blocks larger than 4,000,000 WU.

Suppose you’re a miner running Bitcoin New. You’re lucky enough to find a valid proof of work, and you publish your block. This block will definitely be ≤ 8,000,000 WU, but it might or might not be ≤ 4,000,000 WU. If it is ≤ 4,000,000 WU, it will be accepted by Old nodes. But if not, Old nodes will reject your block. Your blockchain will diverge from the Bitcoin Old blockchain. You’ve caused a blockchain split (Figure 5).

When your New node mines a new block, it might get rejected by the Old nodes, depending on whether it’s ≤ 4,000,000 WU. For the blocks that are rejected, you’ll have wasted a lot of electricity and time mining blocks that don’t make it into the main chain.

But suppose a majority of the hashrate likes your Bitcoin New program and starts using it instead of Bitcoin Old. What happens then? Let’s see how it plays out (Figure 6).

When a New node mines a big block, all New nodes will try to extend that block, but all Old nodes will keep on trying to extend the latest—valid, according to Old rules—block.

New nodes win more blocks over time than Old nodes because they collectively have more hashrate than Old nodes. It seems like the New nodes’ branch will stay intact because it gets a reassuring lead in accumulated proof of work.

New nodes have apparently created a lasting chain split. But if some miners decide to go back to running Bitcoin Old, or if additional miners enter the race using Old nodes so that Old gets a majority of the hashrate again, the New chain might face problems, as Figure 7 shows.

When Old nodes have a hashrate majority, they will outperform the New nodes and eventually catch up with the New nodes and surpass them. New nodes acknowledge this fact by switching back to mining on the Old chain. We say that the branch created by the New nodes was wiped out by a chain reorganization, commonly known as a reorg.

We’ve discussed soft forks several times throughout this book. A soft fork is a change in the consensus rules in which New blocks are accepted by Old nodes. The consensus rules are tightened. In the case with the vegetarian restaurant, a soft fork would be when the restaurant changes its food to vegan.

Segwit is an example of a soft fork. The change was carefully designed so that Old nodes won’t fail in verifying blocks that contain segwit transactions. All Old nodes will accept any valid New blocks and incorporate them into the blockchain.

On the other hand, an Old node could create a block that isn’t valid according to Bitcoin New. For example, a non-segwit miner could include in its block a transaction that spends a segwit output as if it were an anyone-can-spend output (Figure 9).

Suppose there is only a single miner with a small hashrate running Bitcoin New. Also assume that the Old miners produce a block that’s invalid according to New nodes, as in the earlier example with the non-segwit transaction. The result would be that the Old nodes build a block that’s not accepted by the New miner. The New miner would reject the invalid Old block. This is the point where the blockchain splits in two (Figure 10).

In this situation, the Old chain is at risk of being wiped out by a reorg. Suppose more miners decide to upgrade to Bitcoin New, causing a hashrate majority to support the New blockchain. After a while, we’ll probably see a reorg (Figure 11).

The Bitcoin New branch will become the stronger branch, so the remaining Old miners will abandon their branch and start working on the same branch as the New nodes. But as soon as an Old node creates a block that’s invalid on New nodes, it will lose out on the block reward because it won’t be accepted on the New branch.

Let’s look again at what differentiates soft forks from hard forks, as a general rule:

This is a simple, yet true, distinction. We can summarize the effects of a chain split caused by a hard fork versus a soft fork as follows:

To protect users against replay during a chain split due to a hard fork, the transaction format on the New chain can be changed in such a way that the transaction is valid on at most one branch.

When Bitcoin Cash did its chain split, it made sure Old transactions weren’t valid on New nodes and New transactions weren’t valid on Old nodes (Figure 15).

To achieve this, a transaction on the New branch must use a new SIGHASH type, FORKID, in transaction signatures. This type doesn’t do anything, but using it makes the transaction invalid on the Old chain and valid on the New chain. If a transaction doesn’t use FORKID, the transaction is valid on the Old chain and invalid on the New chain.

Using a new SIGHASH type for signatures is, of course, not the only way to achieve replay protection. Any change that makes transactions valid on at most one chain will do. You can, for example, require that New transactions subtract 1 from the input txid. Suppose the UTXO you want to spend has this txid:

If you want to spend the UTXO on the Old chain, you use this hash in the input of your transaction. If you want to spend the UTXO on the New chain, you use this instead:

Note that this is just a silly example, not a full-fledged proposal.

When p2sh was introduced in 2012, the Bitcoin community had no experience in upgrading. It had to come up with a way to avoid a blockchain split. The community implemented soft-fork signaling using the coinbase. New miners signaled support for p2sh by putting the string /P2SH/ into the coinbase of the blocks they produced (Figure 16).

On a specific day, the Bitcoin developers checked if at least 550 of the last 1,000 blocks contained /P2SH/. They did, so the developers made a new software release that would start enforcing the p2sh rules on 1 April 2012, a flag day.

This worked out well; miners quickly adopted the soft fork, and the entire network upgraded within a reasonable time. No split occurred because at least 50% of the hashrate had upgraded prior to the flag day.

I haven’t talked about it much before, but the block header comes with a version (Figure 17). This version is encoded in the first 4 bytes before the previous block hash.

The version is the only thing missing from our previous block headers. This is the actual 80-byte Bitcoin block header:

The block version can be used to signal support for certain new features.

The first soft fork deployment using block-version signaling was done in 2013. This soft fork added a rule that all new blocks must contain the block’s height in their coinbase transaction (Figure 18).

The activation of the soft fork was performed in steps using block-version signaling to avoid a blockchain split:

During step 1, nothing has changed. Only Bitcoin Old rules are in effect. But when 750 of the last 1,000 blocks have version 2, we enter the next step. Here, nodes running the soft fork start ensuring that every new block of version 2 has the height in the coinbase. If not, the block is dropped. One reason is that nodes might be deliberately or accidentally using block version 2 for other purposes than this soft fork. The 75% rule removes false positives before evaluating the 95% rule.

From this point, some Old miner could cause a chain split by creating a block of version 2 that violates the “height in coinbase” rule (Figure 20).

The Old miners would build on top of that block, whereas the New miners would build on top of the previous block. But the New miners probably—depending on the amount of “false” version 2 signaling—have more hashrate and will outperform the Old miners and wipe out the Bitcoin Old branch.

When a greater portion of the blocks—95% of the last 1,000—signals support with version 2 blocks, we enter the last step, step 5. From this point forward, all blocks with versions < 2 will be dropped.

Why did we go through these stages? It isn’t entirely clear why the 75% rule was used, but it does remove false positives, as described. The deployment might have worked fine with the 95% rule only. We won’t explore the rationale behind the 75% rule—just accept that it was used for this deployment and a few others. Table 2 lists soft forks that were introduced using this mechanism.

The upgrade mechanism just described is called a miner-activated soft fork. The miners start enforcing the new rules, and all or most full nodes will follow because the New blocks are accepted by both Old and New full nodes.

Bitcoin’s developers collected a lot of experience from previous soft forks. A few problems needed to be addressed:

The most annoying problem is that you can’t roll out multiple soft forks at once. This is because previous deployment mechanisms, such as the one used for BIP34, checked whether a block version was greater than or equal to a certain number, for example, 2.

Suppose you wanted to deploy both BIP34 and BIP66 simultaneously. BIP34 would use block version 2, and BIP66 would use block version 3. This would mean you couldn’t selectively signal support for only BIP66; you’d also have to signal support for BIP34 because your block’s version 3 is greater than or equal to 2.

The developers came up with a bitcoin improvement proposal, BIP9, that describes a process for how to deploy several soft forks simultaneously.

This process also uses the block version, but in a different way. The developers decided to change the way block version bytes are interpreted. Block versions that have the top 3 bits set to exactly 001 are treated differently.

First, all such block versions are greater than 4 because the smallest such block version is 20000000, which is a lot bigger than 00000004. So, blocks using BIP9 will always support the already-deployed BIP34, 66, and 65. Good.

Next, the 29 bits to the right of the leftmost 001 bits can be used to signal support for at most 29 simultaneous soft forks. Each of the 29 rightmost version bits can be used to independently deploy a single feature or group of features (Figure 21). If a bit is set to 1, then the miner that produced the block supports the feature represented by that bit number.

Several parameters need to be defined for each deployable feature:

The deployment goes through a number of states (see Figure 22). The state is updated after each retarget period.

When the deployment is ACTIVE or FAILED, the bit used to signal support should be reset to 0 so that it can then be reused for other deployments.

Let’s look at an example of how a deployment using version bits can play out. We’ll look at how relative lock time was deployed. The developers of this new feature defined the following BIP9 parameters:

The timeout was one year after the start time, which gave the miners about one year to upgrade to the soft fork implementing this feature.

Figure 23 shows the state transitions that occurred.

This deployment went quickly and smoothly. It took only three retarget periods for 95% of the miners to upgrade to the new software.

Unfortunately, all deployments aren’t as smooth.

Segwit, described in [ch10], also used BIP9 for its deployment, but things didn’t work out as anticipated. It started out the same way csv deployment did. The parameters selected for this deployment were as follows:

A new version of Bitcoin Core was released with these segwit deployment parameters. Users adopted this new version pretty quickly, but for some reason, miners seemed hesitant. The signaling plateaued at around 30%, and the deployment process got stuck in the STARTED state, as Figure 24 shows.

The segwit deployment was at risk of failing—entering the FAILED state after timeout. If this happens, a whole new deployment cycle must be put in place and executed, which could take another year.

To underscore the importance of the economic majority (you, me, and everyone else using Bitcoin), and to avoid having miners vetoing proposals that the economic majority wants, people started thinking more about user-activated soft forks.

Let’s look at a fictitious example of a user-activated soft fork.

Suppose 99% of Bitcoin users (end users, exchanges, merchants, and so on) want a rule change—for example, smaller blocks—that would be a soft fork. Also suppose no miner wants smaller blocks, so they all refuse to comply. Assume also that 99% of the nonmining full nodes change their software to reject all big blocks after a certain block height.

What will happen when that block height passes? Miners that produce big blocks will build a blockchain that users will deem invalid (Figure 27).

The value of the block rewards in the “miner” chain will be unknown because the exchanges don’t deal with the miner chain. Miners won’t be able to exchange their block rewards to pay their electricity bills. Even if the electricity provider takes Bitcoin, the miners won’t be able to pay with their block rewards because the electricity provider won’t accept the miner’s blocks as valid. The electricity provider is also a Bitcoin user, remember?

But if a single miner decides to comply with users’ demands, the blocks that miner produces will be the only blocks users actually accept (Figure 28).

This single miner will be rewarded for the block it created because the economic majority accepts the block. The blocks on the miner (big-block) chain are still pretty worthless because no users accept them. On top of this, the single small-block miner will be able to charge more fees than before because the total amount of block space is smaller—both because the maximum block weight is smaller and because the total number of blocks is smaller.

Some more big-block miners will probably realize they’re quickly running out of money and decide to switch to the user-accepted branch (Figure 29).

When more miners move over to the users’ branch, that branch will eventually grow stronger than the big-blocks branch. When this happens, the big-blocks branch will get wiped out (Figure 30), and the remaining miners will automatically switch to the small-blocks branch because the change is a soft fork.

Users win.

One of the first soft forks in Bitcoin, the deployment of BIP16 (p2sh), was a user-activated soft fork. The deployment was manual in the sense that developers, on a specific day, manually counted the number of blocks that signaled support and then decided on a flag day that they put in the next release of the Bitcoin software. After this date, all blocks that didn’t comply with the new rules were rejected by nodes running this software.

To use the insights from the recent segwit deployment, a new deployment mechanism is in the making as of this writing. It’s generally called a user-activated soft fork. The idea is to start with a BIP9-like deployment, but with the exception that if the deployment doesn’t get LOCKED_IN well before the timeout, blocks that don’t signal for the fork will be dropped. This will effectively cause 100% support because noncompliant blocks won’t count anymore, and the deployment will soon get LOCKED_IN.