Is Poland a Real Country?

Poland has the things you’d expect of a real country: a flag, a currency, and its own phone number. But it lacks others, like borders that stay in the same place. The national anthem, entitled Poland Is Not Yet Lost, suggests something’s not quite right.

The national anthem was written in 1797, when Poland was no longer a real country, having just been divvied up by Catherine the Great of Russia and Frederick William II of Prussia, an arrangement that would be echoed in 1939 with the Nazi-Soviet Pact.

Continue reading “Is Poland a Real Country?”

Nobbling the Nobility

Democracy is not a natural state of affairs. Ever since the first Mesopotamian states formed, ten thousand years ago, kings and warlords have wielded power over the rest of the population. These rulers have maintained their positions with violent enthusiasm. Any who didn’t were quickly replaced.

EmperorAugustus.png

Democracies have been rare and fleeting, popping up briefly in Ancient Greece and Rome and more recently in Europe and America.

There’s a sense today that democracy is failing. Tweets, news reports, and books point to well-funded lobbyists, foreign meddling, the corrosive effects of social media, and the inability of legislatures to decide on anything. There’s palpable worry about what’s being lost.

From our vantage point, it’s easy to think that democracy is the natural way of things. Lapses into dictatorship, as happened in Spain, Italy, and Germany in the 1930s, are the aberration. The truth is that our viewpoint, from a place of prosperity and safety, is unusual.

Instead of asking how democracy might be failing, let’s first ask: how did it ever succeed?

Continue reading “Nobbling the Nobility”

BTC#: Verifying Transactions

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: Spending with Combined Scripts

There are several things that a node does to verify a transaction. Programming Bitcoin covers two of them in detail: making sure that coins aren’t being created out of nothing and checking that signature scripts correctly unlock the value they’re spending.

TransactionValidation.png

Fee Calculation

To ensure that coins aren’t being created out of nothing we check the value of the inputs compared to the value of the outputs on a transaction. The value of the inputs must be equal to or greater than the value of the outputs. Where they’re not equal, the difference is a fee collected by the miners as an incentive for their work.

Continue reading “BTC#: Verifying Transactions”

BTC#: Spending with Combined Scripts

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: Stack Machine Operations

Next: Verifying Transactions »

The simplest Bitcoin transactions have outputs that pay to a public key. That is, they have a locking script (or “pubkey script”) on the output that consists of a public key supplied by the recipient of the funds and the OP_CHECKSIG command.

PayToPubKeyPurchase.png

Let’s say that Alice pays funds to Bob’s public key. To spend the resulting output, or UTXO, Bob needs to provide the magic number that will unlock his funds.

OpenSesamePrivateKey.png

For a payment to a public key, that magic number is his digital signature, derived from the same private key that he used to generate the public key he gave to Alice.

The OP_CHECKSIG command pops two elements off the stack. It assumes the first to be a public key and it assumes the next is the signature.

public static bool OP_CHECKSIG(Stack<byte[]> stack, BigInteger z) {
    var pkBytes = stack.Pop();
    var pk = PublicKey.Parse(pkBytes);
    var sigBytes = stack.Pop();
    var sig = Signature.Parse(sigBytes);
    stack.Push(EncodeNumber(pk.Verify(sig, z) ? 1 : 0));
    return true;
}

This operation also takes an argument z for the document hash, which needs to match the document hash that was used to generate the signature.

To validate a transaction, the signature script on the input is concatenated with the pubkey script on the output and then the combined script is executed. The unlocking script on the input contains the signature that the locking script from the previous output needs to validate.

P2pkUnlockScript.png

The two scripts are combined by adding an addition operator to the script class. This takes the two sets of commands and concatenates them into a new script.

public static Script operator +(Script a, Script b)
{
    return new Script(a.Commands.Concat(b.Commands).ToList());
}

You may have noticed the EncodeNumber() call  in the OP_CHECKSIG method. This deals with some of details of how numbers are serialised onto the stack. The details are in the book and the code is in the repo.

To execute scripts rather than individual commands we need to add an Execute() method to the StackMachine and supply it with the script.

The StackMachine has a dictionary of functions keyed by op-code, so we can pull an op-code off the stack, pull its function reference from the dictionary, and execute it.

private static readonly Dictionary<byte, Func<Stack<byte[]>, bool>> Operations = new Dictionary<byte, Func<Stack<byte[]>, bool>>
{
    { 0, OP_0 },
    { 118, OP_DUP },
    { 169, OP_HASH160 },
    { 172, OP_CHECKSIG }
};

The Execute method takes each command of the stack in turn, determines whether it’s an element or an operation and then either pushes the element to the stack or executes the operation. (This is a simplified version to illustrate the basics. It doesn’t yet handle complexities like conditionals.)

public bool Execute(Script script, BigInteger docHash)
{
    var stack = new Stack<byte[]>();

    foreach (var command in script.Commands)
    {
        if (command.Length > 1 || Operations[command[0]] == null)
        {
            stack.Push(command);
            continue;
        }
        var operation = Operations[command[0]];
        if (operation == OP_IF || operation == OP_NOTIF)
        {
            // handle command list for conditionals
        }
        else if (operation == OP_TOALTSTACK || operation == OP_FROMALTSTACK)
        {
            // handle alternate stack
        }
        else if (operation == OP_CHECKSIG || operation == OP_CHECKSIGVERIFY ||
            operation == OP_CHECKMULTISIG || operation == OP_CHECKMULTISIGVERIFY)
        {
            // handle document hash
        }
        else
        {
            var result = operation(stack);
        }
    }
    return (stack.Count > 0 && DecodeNumber(stack.Peek()) > 0);
}

The result of the execution is a Boolean value indicating whether the script was successful. In the case of OP_CHECKSIG, if the signature was valid the method returns true and transaction validation is successful. Bob can spend his money.

Programming Bitcoin goes on to describe locking and unlocking Pay to Pubkey Hash (p2pkh) scripts, which are more secure than Pay to Public Key scripts and have a shorter locking script. Their workings are similar.

P2pkhUnlockScript.png

The book also touches on Pay to Script Hash scripts variations introduced with Segwit, which we’ll cover later.

« Previous: Stack Machine Operations

Next: Verifying Transactions »

BTC#: Stack Machine Operations

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: The Stack

Next: Stack Machine Operations »

The Bitcoin stack machine operates on a stack of commands. Those commands can take two forms – elements and operations. Elements are numbers, or you can think of them as byte arrays. They could be numbers like zero and one or things like signatures and public keys. Operations are the instructions to the stack machine to do something. Typical operations remove (“pop”) an item from the stack, do something with it, and push a transformed value or a result back onto the stack.

BitcoinMechanics

I’ve created a StackMachine class that will have an Execute method, that we’ll come to later, and a collection of other static methods that perform the various stack machine operations.

Continue reading “BTC#: Stack Machine Operations”

BTC#: The Stack

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: Script Parsing

Next: Stack Machine Operations »

Before getting into the content of the chapter on Bitcoin script, let’s have a quick refresher on the stack.

ComputingBasics

Bitcoin Script is a stack-oriented language. So just remind me again exactly what a stack is and how it works…

A stack is a last-in-first-out (LIFO) filing system. It’s like a stack of papers and books on a desk that have been piled up in whatever order they’ve been used. The oldest thing is at the bottom and the most recent thing is at the top.

Continue reading “BTC#: The Stack”

BTC#: Script Parsing

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: Transaction Objects

Next: BTC#: The Stack »

Last time, in the Script.Parse method I just stored a raw byte array of the right length. Now I want to get into what the script parsing should look like.

ScriptPubKeyParsing.png

Test First

First, here’s a test method to show what we’re aiming at. I start with the script serialised as hex. I want to parse it into a Script object and be able to read what the script actually says. This is the expected text in the code below: four op-codes and a public key hash element.

[TestMethod]
public void Parse_outputCommandText()
{
    var scriptHex = "1976a914ab0c0b2e98b1ab6dbf67d4750b0a56244948a87988ac";
    var reader = new BinaryReader(new MemoryStream(scriptHex.GetBytesFromHex()));

    var script = Script.Parse(reader);
    var expectedText = "OP_DUP OP_HASH160 ab0c0b2e98b1ab6dbf67d4750b0a56244948a879 OP_EQUALVERIFY OP_CHECKSIG";
    var actualText = script.ToString();

    Assert.AreEqual(expectedText, actualText);
}

The Parse Loop

The first byte is the length of the script. In this example it’s 0x19, or 25 in decimal. The Parse loop then runs until the correct number of bytes have been consumed from the stream.

public static Script Parse(BinaryReader reader)
{
    var scriptLength = (int)reader.ReadVarInt();
    var initialPosition = reader.BaseStream.Position;
    if (initialPosition + scriptLength > reader.BaseStream.Length)
    {
        throw new ParsingException($"Script parsing failed. Script length ({scriptLength} bytes) too long.");
    }

    var currentPosition = initialPosition;
    var commands = new List<byte[]>();
    while (currentPosition < initialPosition + scriptLength)
    {
        var currentValue = reader.ReadByte();
        if (currentValue >= 1 && currentValue <= 75)
        {
            commands.Add(reader.ReadBytes(currentValue));
        }
        else if (currentValue == (byte)OpCode.OP_PUSHDATA1)
        {
            var dataLength = reader.ReadByte();
            commands.Add(reader.ReadBytes(dataLength));
        }
        else if (currentValue == (byte)OpCode.OP_PUSHDATA2)
        {
            var dataLength = reader.ReadInt16();
            commands.Add(reader.ReadBytes(dataLength));
        }
        else
        {
            commands.Add(new byte[] { currentValue });
        }

        currentPosition = reader.BaseStream.Position;
    }
    var bytesConsumed = currentPosition - initialPosition;
    if (bytesConsumed != scriptLength)
    {
        throw new ParsingException($"Script parsing failed. {bytesConsumed} bytes consumed. Script length was {scriptLength}.");
    }

    return new Script(commands);
}

The Parse loop reads a byte, decides whether it’s dealing with an op-code or single-byte element, or a multi-byte element. If it’s a multi-byte element it reads the correct number of bytes. Then the op-code or element is added to the command list.

Making It Readable

To turn this into something human-readable, I’ve overridden the ToString() method. To render the op-codes I’ve created an OpCode enum that matches names to numbers. We can use the Enum.GetName method to get the op-code name when we render the script to text.

public enum OpCode
{
    OP_PUSHDATA1 = 76,
    OP_PUSHDATA2 = 77,
    OP_DUP = 118,
    OP_EQUALVERIFY = 136,
    OP_HASH160 = 169,
    OP_CHECKSIG = 172
}

There are many more values than this, which appear in the full code.

The ToString method works its way through the command list and renders op-code names if they appear in the enum or byte arrays encoded as hexadecimal for the other elements.

public override string ToString()
{
    var scriptBuilder = new StringBuilder();
    foreach (var command in Commands)
    {
        string commandText;
        if (command.Length == 1)
        {
            var opCodeName = Enum.GetName(typeof(OpCode), command[0]);
            commandText = string.IsNullOrEmpty(opCodeName) ? $"{command[0]:x2}" : opCodeName;
        }
        else
        {
            commandText = command.EncodeAsHex();
        }
        scriptBuilder.Append($"{commandText} ");
    }

    return scriptBuilder.ToString().TrimEnd();
}

Serialisation

The final piece of the puzzle is the Serialise method, which writes the entire script back out into a binary stream.

public void Serialise(BinaryWriter writer)
{
    var rawBytes = new List<byte>();
    foreach (var command in Commands)
    {
        if (command.Length > 1)
        {
            if (command.Length < 75)
            {
                rawBytes.Add((byte)command.Length);
            }
            else if (command.Length < 256)
            {
                rawBytes.Add((byte)OpCode.OP_PUSHDATA1);
                rawBytes.Add((byte)command.Length);
            }
            else if (command.Length <= 520)
            {
                rawBytes.Add((byte)OpCode.OP_PUSHDATA2);
                rawBytes.AddRange(BitConverter.GetBytes((short)command.Length));
            }
        }
        rawBytes.AddRange(command);
    }

    writer.WriteVarInt((ulong)rawBytes.Count);
    writer.Write(rawBytes.ToArray());
}

For multi-byte elements there’s some conditional code based on the length of the element, making sure that the PUSHDATA op-codes are put in correctly. Otherwise, the single-byte command is added directly to the buffer.

Once the script buffer is complete, its length is written to the stream as a variable-width integer, followed by the buffered byte array.

« Previous: Transaction Objects

Next: The Stack »