Polkadart Logo
SCALE Codec

Compact Encoding

Understanding SCALE's space-efficient compact integer encoding

What is Compact Encoding?

Compact encoding is a variable-length integer encoding scheme in SCALE that optimizes storage by using fewer bytes for smaller numbers. This is crucial for blockchain applications where storage space is at a premium.

Instead of always using 4 or 8 bytes for every integer, compact encoding uses:

  • 1 byte for values 0-63
  • 2 bytes for values 64-16,383
  • 4 bytes for values 16,384-1,073,741,823
  • 5+ bytes for larger values

Why Use Compact Encoding?

In blockchain applications, many values tend to be small:

  • Transaction counts (nonces)
  • Collection lengths
  • Small balances
  • Indices and counters

Compact encoding can save significant space:

  • Regular u32: 42 → 4 bytes [42, 0, 0, 0]
  • Compact: 42 → 1 byte [168] (42 << 2)

Over thousands of transactions, this adds up to substantial savings.

Compact Integer Codec

The CompactCodec encodes standard Dart int values:

Basic Usage

import 'package:polkadart_scale_codec/polkadart_scale_codec.dart';

final codec = CompactCodec.codec;

// Small value (0-63): 1 byte
final output1 = HexOutput();
codec.encodeTo(42, output1);
print(output1.toString()); // 0xa8 (42 << 2 = 168 = 0xa8)

// Medium value (64-16,383): 2 bytes
final output2 = HexOutput();
codec.encodeTo(1000, output2);
print(output2.toString()); // 0xa10f

// Decode
final input = Input.fromHex('0xa8');
final value = codec.decode(input);
print(value); // 42

Encoding Rules

Compact integers use the last 2 bits as a mode indicator:

Mode 00 (Single-byte): Values 0-63

codec.encodeTo(0, output);   // 0x00
codec.encodeTo(42, output);  // 0xa8 (42 << 2 | 0b00)
codec.encodeTo(63, output);  // 0xfc (63 << 2 | 0b00)

Mode 01 (Two-byte): Values 64-16,383

codec.encodeTo(64, output);    // 0x0101
codec.encodeTo(1000, output);  // 0xa10f
codec.encodeTo(16383, output); // 0xfdff

Mode 10 (Four-byte): Values 16,384-1,073,741,823

codec.encodeTo(16384, output);      // 0x02000100
codec.encodeTo(1000000, output);    // 0x02093d00

Mode 11 (Big integer): Values ≥ 1,073,741,824

// First byte: ((bytesNeeded - 4) << 2) | 0b11
// Followed by the value in little-endian
codec.encodeTo(100000000000, output);

Size Hints

The codec can predict encoded size:

final codec = CompactCodec.codec;

print(codec.sizeHint(42));          // 1
print(codec.sizeHint(1000));        // 2
print(codec.sizeHint(100000));      // 4
print(codec.sizeHint(100000000000)); // Variable

Compact BigInt Codec

For values larger than Dart's int range, use CompactBigIntCodec:

Basic Usage

final codec = CompactBigIntCodec.codec;

// Large value
final value = BigInt.from(1234567890123456789);
final output = HexOutput();
codec.encodeTo(value, output);

// Decode
final decoded = codec.decode(Input.fromHex(output.toString()));
print(decoded); // 1234567890123456789

Working with Token Amounts

Blockchain tokens often have large values when represented in smallest unit:

// 1 DOT = 10^10 Plancks
final oneDot = BigInt.from(10000000000);

final codec = CompactBigIntCodec.codec;
final encoded = codec.encode(oneDot);

print('Encoding 1 DOT:');
print('  Regular U128: 16 bytes');
print('  Compact: ${encoded.length} bytes');

Maximum Values by Size

// 1 byte: 0-63
CompactBigIntCodec.codec.encodeTo(BigInt.from(63), output);

// 2 bytes: 64-16,383
CompactBigIntCodec.codec.encodeTo(BigInt.from(16383), output);

// 4 bytes: 16,384-1,073,741,823
CompactBigIntCodec.codec.encodeTo(BigInt.from(1073741823), output);

// 5+ bytes: 1,073,741,824+
CompactBigIntCodec.codec.encodeTo(
  BigInt.parse('18446744073709551615'), // u64::MAX
  output,
);

When to Use Compact Encoding

Use Compact When:

Collection Lengths

Vector and sequence lengths are always compact-encoded

final codec = SequenceCodec(U32Codec.codec);
// Length is automatically compact-encoded

Small Integers

Values that are usually small (< 16,384)

// Transaction nonce
final nonce = 5;
CompactCodec.codec.encodeTo(nonce, output);

Token Amounts

Large token amounts in smallest unit

final balance = BigInt.from(1000000000000);
CompactBigIntCodec.codec.encodeTo(balance, output);

Indices

Array indices, reference counts, etc.

final index = 42;
CompactCodec.codec.encodeTo(index, output);

Don't Use Compact When:

Fixed-size encoding is required (e.g., in cryptographic operations or when exact byte size matters)
Values are consistently large and compact encoding would not save space
Compatibility with external systems that don't support compact encoding

Decoding Examples

Decoding Collection Lengths

// Vec<u32> encoding includes compact length
final input = Input.fromHex('0x0c64000000c8000000e8030000');
// 0x0c = compact(3) - three elements
// Followed by three u32 values

// Decode the length
final length = CompactCodec.codec.decode(input);
print('Length: $length'); // 3

// Decode the elements
final elements = List.generate(
  length,
  (_) => U32Codec.codec.decode(input),
);
print(elements); // [100, 200, 1000]

Decoding with Unknown Size

void decodeCompact(Input input) {
  // Read first byte to determine mode
  final firstByte = input.read();
  input.resetOffset(); // Go back

  final mode = firstByte & 0b11;

  switch (mode) {
    case 0b00:
      print('Single-byte mode');
      break;
    case 0b01:
      print('Two-byte mode');
      break;
    case 0b10:
      print('Four-byte mode');
      break;
    case 0b11:
      final bytes = (firstByte >> 2) + 4;
      print('Big integer mode: $bytes bytes');
      break;
  }

  // Now decode normally
  final value = CompactCodec.codec.decode(input);
  print('Value: $value');
}

Complete Example: Transaction Data

Real-world example encoding transaction information:

import 'package:polkadart_scale_codec/polkadart_scale_codec.dart';

class Transaction {
  final int nonce;
  final BigInt tip;
  final List<int> callData;

  Transaction(this.nonce, this.tip, this.callData);

  Uint8List encode() {
    final output = ByteOutput();

    // Nonce: usually small, use compact
    CompactCodec.codec.encodeTo(nonce, output);

    // Tip: can be large, use compact BigInt
    CompactBigIntCodec.codec.encodeTo(tip, output);

    // Call data: length-prefixed bytes
    CompactCodec.codec.encodeTo(callData.length, output);
    output.write(callData);

    return output.toBytes();
  }

  static Transaction decode(Input input) {
    final nonce = CompactCodec.codec.decode(input);
    final tip = CompactBigIntCodec.codec.decode(input);

    final callDataLength = CompactCodec.codec.decode(input);
    final callData = input.readBytes(callDataLength);

    return Transaction(nonce, tip, callData);
  }
}

void main() {
  // Create transaction
  final tx = Transaction(
    5,                          // nonce
    BigInt.from(1000000000),   // tip (0.1 DOT)
    [1, 2, 3, 4],              // call data
  );

  // Encode
  final encoded = tx.encode();
  print('Encoded size: ${encoded.length} bytes');

  // Decode
  final decoded = Transaction.decode(Input.fromBytes(encoded));
  print('Nonce: ${decoded.nonce}');
  print('Tip: ${decoded.tip}');
  print('Call data: ${decoded.callData}');
}

Space Savings Analysis

Comparison Table

ValueRegular u32CompactSavings
04 bytes1 byte75%
424 bytes1 byte75%
634 bytes1 byte75%
644 bytes2 bytes50%
1,0004 bytes2 bytes50%
16,3834 bytes2 bytes50%
16,3844 bytes4 bytes0%
1,000,0004 bytes4 bytes0%

Real-World Impact

In a typical block:

  • 1000 transactions with nonces 0-999
  • Regular encoding: 4,000 bytes
  • Compact encoding: ~2,000 bytes
  • Savings: ~50%

Advanced Topics

Custom Compact Types

You can wrap compact encoding for domain-specific types:

class CompactBalance {
  final BigInt value;
  const CompactBalance(this.value);
}

class CompactBalanceCodec with Codec<CompactBalance> {
  const CompactBalanceCodec();

  @override
  void encodeTo(CompactBalance value, Output output) {
    CompactBigIntCodec.codec.encodeTo(value.value, output);
  }

  @override
  CompactBalance decode(Input input) {
    return CompactBalance(CompactBigIntCodec.codec.decode(input));
  }

  @override
  bool isSizeZero() => false;
}

Compact in Composite Types

Compact encoding is commonly used within composite structures:

final accountCodec = CompositeCodec({
  'nonce': CompactCodec.codec,  // Use compact for nonce
  'balance': CompactBigIntCodec.codec,  // Compact for balance
  'data': U8SequenceCodec.codec,  // Sequence has compact length
});

Common Pitfalls

Don't Mix Compact with Fixed-Size

Never mix compact encoding with contexts expecting fixed sizes:
// Wrong: Array of compact values
ArrayCodec(CompactCodec.codec, 10); // Don't do this!

// Right: Array of regular values
ArrayCodec(U32Codec.codec, 10);

Watch for Overflow

CompactCodec uses int, which has limits:
// For very large values, use CompactBigIntCodec
final large = BigInt.parse('9999999999999999999');
CompactBigIntCodec.codec.encodeTo(large, output);

Use Size Hints

When pre-allocating buffers, use sizeHint:
final value = 1000000;
final expectedSize = CompactCodec.codec.sizeHint(value);
// Pre-allocate buffer of expectedSize

Next Steps

Primitive Types

Understanding and using SCALE primitive type codecs

Collections

Working with SCALE collection types - sequences, arrays, maps, and sets