BTC#: The Stack

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: Script Parsing

Before getting into the content of the chapter on Bitcoin script, let’s have a quick refresher on the stack.

ComputingBasics

Bitcoin Script is a stack-oriented language. So just remind me again exactly what a stack is and how it works…

A stack is a last-in-first-out (LIFO) filing system. It’s like a stack of papers and books on a desk that have been piled up in whatever order they’ve been used. The oldest thing is at the bottom and the most recent thing is at the top.

The reason it’s referred to as a last-in-first-out system is that the most accessible item is the last one to be added. It has to be removed first to get to those underneath.

AlchemistStack.png

In computing, a stack is an area of memory where numbers can be temporarily stored while something else happens and then recovered when that task is complete.

For example, a program may have a set of numbers in the CPU’s registers and need to call a subroutine. Before calling the subroutine it pushes the current state of the registers onto the stack and then, once the subroutine completes, it pops the values back off into the original registers.

In a stack-oriented language like PostScript or Bitcoin script the operands are pushed onto the stack and the operator pops them off to perform its operation. An addition might be expressed something like

2 3 ADD

Putting the operator at the end, after all the operands, is called postfix notation.

Program execution proceeds from left to right. The first two things encountered are numbers. They’re not operators so they’re pushed onto the stack. When the ADD instruction is encountered that instruction is executed. The ADD operator takes two arguments, which it pops from the stack. It adds them together and then pushes the result to the stack so that some other calculation can use it later.

AddPostfix.png

Subsequent calculations can carry on using whatever’s been put on the stack by previous operations.

ComplexPostfix.png

In Bitcoin, transactions are validated by combining scripts from transaction inputs with scripts from the transaction outputs (UTXOs) they’re spending from. The two script parts are concatenated and run together. Typically the input script will put values on to the stack that satisfy the spending conditions in the output script.

« Previous: Script Parsing

BTC#: Script Parsing

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: Transaction Objects

Next: BTC#: The Stack »

Last time, in the Script.Parse method I just stored a raw byte array of the right length. Now I want to get into what the script parsing should look like.

ScriptPubKeyParsing.png

Test First

First, here’s a test method to show what we’re aiming at. I start with the script serialised as hex. I want to parse it into a Script object and be able to read what the script actually says. This is the expected text in the code below: four op-codes and a public key hash element.

[TestMethod]
public void Parse_outputCommandText()
{
    var scriptHex = "1976a914ab0c0b2e98b1ab6dbf67d4750b0a56244948a87988ac";
    var reader = new BinaryReader(new MemoryStream(scriptHex.GetBytesFromHex()));

    var script = Script.Parse(reader);
    var expectedText = "OP_DUP OP_HASH160 ab0c0b2e98b1ab6dbf67d4750b0a56244948a879 OP_EQUALVERIFY OP_CHECKSIG";
    var actualText = script.ToString();

    Assert.AreEqual(expectedText, actualText);
}

The Parse Loop

The first byte is the length of the script. In this example it’s 0x19, or 25 in decimal. The Parse loop then runs until the correct number of bytes have been consumed from the stream.

public static Script Parse(BinaryReader reader)
{
    var scriptLength = (int)reader.ReadVarInt();
    var initialPosition = reader.BaseStream.Position;
    if (initialPosition + scriptLength > reader.BaseStream.Length)
    {
        throw new ParsingException($"Script parsing failed. Script length ({scriptLength} bytes) too long.");
    }

    var currentPosition = initialPosition;
    var commands = new List<byte[]>();
    while (currentPosition < initialPosition + scriptLength)
    {
        var currentValue = reader.ReadByte();
        if (currentValue >= 1 && currentValue <= 75)
        {
            commands.Add(reader.ReadBytes(currentValue));
        }
        else if (currentValue == (byte)OpCode.OP_PUSHDATA1)
        {
            var dataLength = reader.ReadByte();
            commands.Add(reader.ReadBytes(dataLength));
        }
        else if (currentValue == (byte)OpCode.OP_PUSHDATA2)
        {
            var dataLength = reader.ReadInt16();
            commands.Add(reader.ReadBytes(dataLength));
        }
        else
        {
            commands.Add(new byte[] { currentValue });
        }

        currentPosition = reader.BaseStream.Position;
    }
    var bytesConsumed = currentPosition - initialPosition;
    if (bytesConsumed != scriptLength)
    {
        throw new ParsingException($"Script parsing failed. {bytesConsumed} bytes consumed. Script length was {scriptLength}.");
    }

    return new Script(commands);
}

The Parse loop reads a byte, decides whether it’s dealing with an op-code or single-byte element, or a multi-byte element. If it’s a multi-byte element it reads the correct number of bytes. Then the op-code or element is added to the command list.

Making It Readable

To turn this into something human-readable, I’ve overridden the ToString() method. To render the op-codes I’ve created an OpCode enum that matches names to numbers. We can use the Enum.GetName method to get the op-code name when we render the script to text.

public enum OpCode
{
    OP_PUSHDATA1 = 76,
    OP_PUSHDATA2 = 77,
    OP_DUP = 118,
    OP_EQUALVERIFY = 136,
    OP_HASH160 = 169,
    OP_CHECKSIG = 172
}

There are many more values than this, which appear in the full code.

The ToString method works its way through the command list and renders op-code names if they appear in the enum or byte arrays encoded as hexadecimal for the other elements.

public override string ToString()
{
    var scriptBuilder = new StringBuilder();
    foreach (var command in Commands)
    {
        string commandText;
        if (command.Length == 1)
        {
            var opCodeName = Enum.GetName(typeof(OpCode), command[0]);
            commandText = string.IsNullOrEmpty(opCodeName) ? $"{command[0]:x2}" : opCodeName;
        }
        else
        {
            commandText = command.EncodeAsHex();
        }
        scriptBuilder.Append($"{commandText} ");
    }

    return scriptBuilder.ToString().TrimEnd();
}

Serialisation

The final piece of the puzzle is the Serialise method, which writes the entire script back out into a binary stream.

public void Serialise(BinaryWriter writer)
{
    var rawBytes = new List<byte>();
    foreach (var command in Commands)
    {
        if (command.Length > 1)
        {
            if (command.Length < 75)
            {
                rawBytes.Add((byte)command.Length);
            }
            else if (command.Length < 256)
            {
                rawBytes.Add((byte)OpCode.OP_PUSHDATA1);
                rawBytes.Add((byte)command.Length);
            }
            else if (command.Length <= 520)
            {
                rawBytes.Add((byte)OpCode.OP_PUSHDATA2);
                rawBytes.AddRange(BitConverter.GetBytes((short)command.Length));
            }
        }
        rawBytes.AddRange(command);
    }

    writer.WriteVarInt((ulong)rawBytes.Count);
    writer.Write(rawBytes.ToArray());
}

For multi-byte elements there’s some conditional code based on the length of the element, making sure that the PUSHDATA op-codes are put in correctly. Otherwise, the single-byte command is added directly to the buffer.

Once the script buffer is complete, its length is written to the stream as a variable-width integer, followed by the buffered byte array.

« Previous: Transaction Objects

Next: The Stack »

BTC#: Transaction Objects

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: Base 58 Encoded Addresses

Next: Script Parsing »

Now for the real meat. The work we’ve done so far, the maths and the serialisation, has provided the scaffolding for what we really want to do, which is transfer value.

BankerUtxos.png

We talked earlier about how there are no “coins” in Bitcoin. What we’re spending are Unspent Transaction Outputs (UTXOs). A transaction is a collection of inputs, which point back to previous UTXOs and a collection of outputs, which become the new UTXOs.

Continue reading “BTC#: Transaction Objects”

BTC#: Base 58 Encoded Addresses

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: DER Serialisation

Next: Transaction Objects »

BitcoinMechanics

Base 58 Encoding

Base 58 encoding is a compromise between hexadecimal, i.e. base 16 encoding, which can store 4 bits per character, and base 64 encoding, which can store 6 bits per character but can be confusing for humans to read. Base 58 tries to remove the confusion by eliminating characters that get mixed up, like O and 0 and 1 and l.

Base 58 seems slightly unnatural because we’ve worked in base 10 since we were toddlers and in powers to two – base 2, base 16, and base 64 – since we learned about computers. Despite feeling odd, the principle is the same. We cycle through a 58-character alphabet and when that loops over we move to the next column. Instead of ones, tens, and hundreds, we have ones, 58s, and 58-squareds, etc.

The Base 58 alphabet is held as a constant on the Serialisation class.

public const string BASE58_ALPHABET = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";

That class also contains the method to encode.

public static string EncodeAsBase58(byte[] buffer)
{
    var base58 = new StringBuilder();
    var leadingZeroesCount = buffer.TakeWhile(b => b == 0).Count();
    base58.Append('1', leadingZeroesCount);

    var number = buffer.ToBigInteger(ByteArrayFormat.BigEndianUnsigned);
    var base58CharBuffer = new StringBuilder(); //Least significant character first
    while (number > 0)
    {
        base58CharBuffer.Append(BASE58_ALPHABET[(int)number.Mod(58)]);
        number = number / 58;
    }

    base58.Append(base58CharBuffer.ToString().Reverse().ToArray());
    return base58.ToString();
}

The algorithm involves converting the byte array to a BigInteger and then repeatedly dividing by 58 and using the remainders to look up the character to go into each column. In contrast to the code in the book, the base 58 string here is built up in the opposite order, so needs to be reversed once we have all the characters.

Base 58 with Checksum

Part of the motivation for base 58 encoding was to allow addresses to be copied by hand, from printed text, or read out loud. To help prevent mistakes, the Base58Check format is used. This hashes the data prior to encoding and adds the first four bytes of the hash at the end. If there’s an error in copying, the hash won’t match and an address with a type in it won’t be accepted as valid.

The method for Base58Check is also on the serialisation class. The Base58CheckType enum indicates which leading bytes should be added based on what’s being encoded.

public static string EncodeAsBase58Check(byte[] addressBytes, Base58CheckType type)
{
    var prefix = GetBase58CheckPrefix(type);
    var prefixedLength = prefix.Length + addressBytes.Length;
    var addressBuffer = new byte[prefixedLength + 4];

    prefix.CopyTo(addressBuffer, 0);
    addressBytes.CopyTo(addressBuffer, prefix.Length);

    var checkBytes = Hash.DoubleSha256(addressBuffer.Segment(0, prefixedLength)).Segment(0, 4);
    checkBytes.CopyTo(addressBuffer, prefixedLength);

    return EncodeAsBase58(addressBuffer);
}

The method generates the checksum hash, concatenates all the pieces and then base-58 encodes the result.

Addresses and Hash 160

Bitcoin payments could be made to public keys but generally are not. The more common payment destination is an address derived from the public key.

Addresses are shorter than SEC format public keys because they take the SEC formatted public key and hash it using SHA-256 followed by another hash algorithm called RIPEMD-160, which results in 20 bytes rather than 33. This combination is known as Hash160.

I’ve created a Hash160 implementation on the static Hash class using C#’s implementation of RIPEMD-160.

public static byte[] Hash160(byte[] buffer)
{
    var sha256 = SHA256.Create();
    var ripemd160 = RIPEMD160.Create();
    return ripemd160.ComputeHash(sha256.ComputeHash(buffer));
}

The code to create a Bitcoin address is on the PublicKey class. It’s very short as it simply composes pieces we’ve already looked at.

public string ToBase58Address(SerialisationFormat format)
{
    var type = (format & SerialisationFormat.TestNet) > 0 ?
        Base58CheckType.TESTNET_ADDRESS : Base58CheckType.MAINNET_ADDRESS;
    return Serialisation.EncodeAsBase58Check(Hash.Hash160(ToSecFormat(format)), type);
}

There’s a header byte to indicate which network the address belongs to and then a Base58Check encoded hash of the SEC formatted public key.

Wallet Import Formatted Private Keys

Wallet Import Format (WIF) is used to serialise private keys. Generally you don’t want to do this because the keys should remain secret and they shouldn’t be left lying around. There are times, though, when you may need to transfer them from one place to another.

WIF is very similar to address format we saw above with the header bytes and the Base58Check encoding.

Prior to now, the private key was stored as a raw number inside a KeyPair object but, since were adding behaviour, I’ve broken it out into its own class.

public PrivateKey(BigInteger secret)
{
    Secret = secret;
}

public string ToWifFormat(SerialisationFormat format)
{
    var type = (format & SerialisationFormat.TestNet) > 0 ?
        Base58CheckType.PRIVATE_KEY_WIF_TESTNET : Base58CheckType.PRIVATE_KEY_WIF;
    var wifBuffer = Secret.ToByteArray(ByteArrayFormat.BigEndianUnsigned, 32);
    if ((format & SerialisationFormat.Compressed) > 0)
    {
        wifBuffer = wifBuffer.Concat(new byte[] { 1 }).ToArray();
    }
    return Serialisation.EncodeAsBase58Check(wifBuffer, type);
}

That’s the end of the scaffolding work. We have all the maths and the serialisation code in place. The next chapters get into Bitcoin transactions and scripting.

« Previous: DER Serialisation

Next: Transaction Objects »

BTC#: DER Serialisation

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: SEC Serialisation

Next: Base 58 Encoded Addresses »

DER Format for Signatures

DER format does a similar job to SEC format, encoding a pair of 256-bit numbers. There’s no compressed format because there’s no way to derive a signature’s s-component from the r-value.

DER format is a little more complicated than SEC because it allows for the encoding of negative numbers, with the sign of the number encoded by the most significant bit. This means that, to ensure all numbers are positive, a leading zero is added to all numbers that have that bit set.

The optional leading zero makes the size of the r- and s- fields vary, which means that the format needs to include length information for each component and for the total length of the serialised data.

DerSerialisation.png

Prefixing and postfixing zeroes to byte array was something I’ve found myself doing a few times. In the spirit of keeping the code DRY (“Don’t Repeat Yourself”), I’ve added byte array extension method to do this.

public static byte[] PrefixZeroes(this byte[] buffer, int count)
{
    if (count == 0) return buffer;

    var result = new byte[buffer.Length + count];
    buffer.CopyTo(result, count);
    return result;
}

The method to serialise signatures to DER format is on the Signature class.

public byte[] ToDerFormat()
{
    var rBytes = R.ToByteArray(ByteArrayFormat.BigEndianUnsigned);
    if (rBytes[0] > 127)
    {
        rBytes = rBytes.PrefixZeroes(1);
    }
    var sBytes = S.ToByteArray(ByteArrayFormat.BigEndianUnsigned);
    if (sBytes[0] > 127)
    {
        sBytes = sBytes.PrefixZeroes(1);
    }

    var totalLength = rBytes.Length + sBytes.Length + 6;
    var derBuffer = new byte[totalLength];
    derBuffer[0] = Serialisation.DER_START_MARKER;
    derBuffer[1] = (byte)(totalLength - 2);
    derBuffer[2] = Serialisation.DER_INTEGER_MARKER;
    derBuffer[3] = (byte)rBytes.Length;
    rBytes.CopyTo(derBuffer, 4);
    derBuffer[rBytes.Length + 4] = Serialisation.DER_INTEGER_MARKER;
    derBuffer[rBytes.Length + 5] = (byte)sBytes.Length;
    sBytes.CopyTo(derBuffer, rBytes.Length + 6);

    return derBuffer;
}

The code is a little more involved than the Python code in the book but it’s not complex. In C# we need to explicitly cast integers to bytes and I’ve used a buffer of the right size to copy the values into rather than concatenating as we go.

« Previous: SEC Serialisation

Next: Base 58 Encoded Addresses »

 

BTC#: SEC Serialisation

Series: BTC# – Learning to Program Bitcoin in C#

« Previous: Endianness

Next: DER Serialisation »

So far, we’ve just created objects like public keys and signatures and manipulated them in memory. Soon, we’ll want to serialise them to the network or to disk as we construct scripts and transactions.

BitcoinMechanics.png

SEC for Public Keys

The first type of serialisation Programming Bitcoin introduces is the SEC format used for public keys. The uncompressed format is very straightforward: a marker byte to indicate that it’s uncompressed followed by the x- and y-values of the public key in big-endian hexadecimal.

SecSerialisation.png

The compressed format takes advantage of the fact that the elliptic curve is symmetrical. If you know the x-value, you can narrow the y-value down to two possibilities, one even and one odd. The compressed serialisation consists of a marker byte indicating the parity (the even- or odd-ness) of the y-value followed by the x-value serialised as above.

I’ve added a SerialisationFormat enum to hold the different serialisation rules we’ll come across and a static Serialisation class to hold constants for things like the values of the marker bytes.

public enum SerialisationFormat
{
    Uncompressed = 0,
    Compressed = 1,
}
public static class Serialisation
{
    public const byte SEC_MARKER_COMPRESSED_EVEN = 2;
    public const byte SEC_MARKER_COMPRESSED_ODD = 3;
    public const byte SEC_MARKER_UNCOMPRESSED = 4;
}

The serialisation code is on the PublicKey class and makes use of the Endianness helpers from the previous post.

public byte[] ToSecFormat(SerialisationFormat format)
{
    return (format & SerialisationFormat.Compressed) == SerialisationFormat.Compressed ?
        ToCompressedSecFormat() :
        ToUncompressedSecFormat();
}

private byte[] ToUncompressedSecFormat()
{
    return (new byte[] { Serialisation.SEC_MARKER_UNCOMPRESSED })
        .Concat(Point.X.Value.ToByteArray(ByteArrayFormat.BigEndianUnsigned, 32))
        .Concat(Point.Y.Value.ToByteArray(ByteArrayFormat.BigEndianUnsigned, 32))
        .ToArray();
}

private byte[] ToCompressedSecFormat()
{
    var markerByte = Point.Y.Value % 2 == 0 ?
        Serialisation.SEC_MARKER_COMPRESSED_EVEN :
        Serialisation.SEC_MARKER_COMPRESSED_ODD;

    return (new byte[] { markerByte })
        .Concat(Point.X.Value.ToByteArray(ByteArrayFormat.BigEndianUnsigned, 32))
        .ToArray();
}

SEC serialisation is very simple. Deserialising the uncompressed form is also very simple because all the necessary data is already present. Deserialising the compressed form is a little bit harder because the y-value needs to be derived. One necessary extra is the ability to find the square root of a finite field element. The book shows the derivation but the resulting code is quite simple.

public FiniteFieldElement Sqrt()
{
    return this.Pow((Order + 1) / 4);
}

The Parse method is a static method that returns a PublicKey.

public static PublicKey Parse(byte[] secBuffer)
{
    var marker = secBuffer[0];

    if(marker < Serialisation.SEC_MARKER_MIN || marker > Serialisation.SEC_MARKER_MAX)
    {
        throw new ArgumentException($"Invalid marker byte. {marker:x2} supplied.");
    }

    var curve = Secp256k1.Curve;
    FiniteFieldElement x, y;

    if (marker == Serialisation.SEC_MARKER_UNCOMPRESSED)
    {
        if (secBuffer.Length != Serialisation.SEC_LENGTH_UNCOMPRESSED)
        {
            throw new ArgumentException($"Uncompressed SEC buffer (prefix {marker:x2}) must be {Serialisation.SEC_LENGTH_UNCOMPRESSED} bytes long. {secBuffer.Length} bytes supplied.");
        }
        x = new FiniteFieldElement(
            secBuffer.Segment(1, 32).ToBigInteger(ByteArrayFormat.BigEndianUnsigned),
            Secp256k1.P
        );
        y = new FiniteFieldElement(
            secBuffer.Segment(33, 32).ToBigInteger(ByteArrayFormat.BigEndianUnsigned),
            Secp256k1.P
        );
        return new PublicKey(new EllipticCurveFiniteFieldPoint(x, y, curve));
    }

    if (secBuffer.Length != Serialisation.SEC_LENGTH_COMPRESSED)
    {
        throw new ArgumentException($"Compressed SEC buffer (prefix {marker:x2}) must be {Serialisation.SEC_LENGTH_COMPRESSED} bytes long. {secBuffer.Length} bytes supplied.");
    }
    x = new FiniteFieldElement(
        secBuffer.Segment(1, 32).ToBigInteger(ByteArrayFormat.BigEndianUnsigned),
        Secp256k1.P
    );
    var alpha = x.Pow(3) + curve.B;
    var beta = alpha.Sqrt();
    y = (marker == 2 ^ beta.Value % 2 == 0) ?
        new FiniteFieldElement(Secp256k1.P - beta.Value, Secp256k1.P) :
        beta;
    return new PublicKey(new EllipticCurveFiniteFieldPoint(x, y, curve));
}

The code checks the marker byte and the length of the buffer supplied – 65 bytes when uncompressed, 33 when compressed. If that’s OK, it segments the byte array into the appropriate pieces and deserialises them into BigInteger objects. For compressed data it then works out what the y-value is and returns the new public key.

« Previous: Endianness

Next: DER Serialisation »

 

Methode Glaswegienne

“Scottish cuisine” is not a phrase that fills you with hope. It doesn’t suggest the sophistication of French, the urgent, exotic freshness of Thai, or the “what the hell did that used to be?” of Chinese. Well, maybe the last. No, “Scottish cuisine” makes you think of mashed up sheep’s organs stuffed into a different, unmashed-up sheep’s organ.

Haggis

Nonetheless, Scottish nationalists have reacted with outrage and denial at the discovery that haggis may have originated in the south of England rather than in Caledonia. Food historian Catherine Brown has made news with her claim that a haggis recipe published in 1615 in The English Hus-Wife predates any Scottish mention by a hundred years.

The claims have been rebutted by a representative of the Scottish Institute for Arts and Sciences who said, “If yer repeat that again I’ll fuckin’ nut yer, yer little gobshite.”

However, the claim rings true. English cuisine is shaped by England’s climate. That is, it’s crap. Traditional English dishes are, by-and-large, horrible – jellied eels, damp chips with mushy peas, and vegetables boiled until they’re grey. Things have changed a bit recently with the now-widespread addition of Jamie Oliver’s frothing spittle.

So haggis will fit right in in England. With its loss, the only item remaining on the traditional Scottish menu is the deep-fried Mars bar. While this sounds disgusting, and is enough to give everyone at the Heart Foundation a stroke, it is in fact a work of genius. But you will only ever appreciate this if you consume one when you’re pisseder than a tankful of ill-disciplined newts. I discovered this while living in the Edinburgh of the South.

The unlikely saviour of Scotland’s culinary tradition could be chicken tikka masala. Ali Ahmed Aslam, founder of the Shish Mahal restaurant in Glasgow, lays claim to inventing the dish. With the help of his local MP, he has applied to the European Union for “Protected Designation of Origin” status.

Protected Designation of Origin status is what’s responsible for rules like the one saying that fizzy wine that doesn’t come from the Champagne region of France has to go by the clumsy appellation of “Methode Champenoise.” Likewise Parma ham that’s not from Parma, Newcastle Brown Ale that’s not from Newcastle, and Stilton cheese that’s not from some rigorously defined bit of the English Midlands. (It’s illegal to make Stilton cheese in Stilton, which is near Cambridge, but you don’t need all those acres of bureaucrats to come up with rules that are simple.)

Unlike these products though, chicken tikka masala doesn’t have the word “Glasgow” in its name so I’m not sure what they’re trying to protect. My Hindi’s not that great (although it’s better than my Glaswegian) but I think “chicken tikka masala” means something like “chicken lump mixture.” Presumably, under the proposed rules, restaurants outside Glasgow’s West End would have to refer to the dish as “Glaswegian-style chicken lump mixture” – an advertiser’s dream.

ChickenTikkaMasala.png

The EU’s meddling would at least clear up any confusion that the dish might be of Indian origin. A tin of Campbell’s condensed tomato soup is not a traditional ingredient in the Punjab. England, however, looks likely to get stuck with the haggis unless they can pass the blame on to the Vikings.


This article first appeared on Not PC many years ago. It is almost as true and relevant as the day it was written. Make of that what you will.