Advanced

Custom Serialization Patterns

Advanced patterns for building custom serializers, handling complex data structures, versioning, and performance optimization.

Overview

While Sia's built-in add* and read* methods cover all primitive types, real-world applications require serializing complex structures: nested objects, optional fields, discriminated unions, and evolving schemas. This guide covers patterns for building maintainable serialization logic.

Embedding Sia Instances

The embedSia method lets you compose serialized data from multiple Sia instances. This is useful for building packets with separately constructed headers and bodies:

import { Sia } from "@timeleap/sia";

// Build the header
const header = new Sia();
header.addUInt8(1).addUInt16(42).addUInt64(Date.now());

// Build the body
const body = new Sia();
body.addString16("Hello from the body");

// Compose into a single buffer
const packet = new Sia();
packet.embedSia(header).embedSia(body);

const bytes = packet.toUint8Array();

You can also embed raw bytes with embedBytes:

const checksum = computeChecksum(payload);
packet.embedBytes(payload).addByteArray8(checksum);

embedSia copies the written portion of the source Sia (from offset 0 to the current write position). The source instance is not modified.

Creating Custom Serializers

Pattern 1: Type-Safe Functions

The simplest approach: define a pair of arrow functions per type:

interface Sensor {
  id: number;
  name: string;
  value: number;
  timestamp: number;
}

const serializeSensor = (sia: Sia, sensor: Sensor): void => {
  sia
    .addUInt16(sensor.id)
    .addAscii8(sensor.name)
    .addInt32(sensor.value)
    .addUInt64(sensor.timestamp);
};

const deserializeSensor = (sia: Sia): Sensor => ({
  id: sia.readUInt16(),
  name: sia.readAscii8(),
  value: sia.readInt32(),
  timestamp: sia.readUInt64(),
});

// Use directly
const sia = new Sia();
serializeSensor(sia, {
  id: 1,
  name: "temp",
  value: 2350,
  timestamp: Date.now(),
});

sia.seek(0);
const sensor = deserializeSensor(sia);

// Use with arrays
const sia2 = new Sia();
sia2.addArray16(sensors, serializeSensor);
sia2.seek(0);
const decoded = sia2.readArray16(deserializeSensor);

Pattern 2: Class-Based Serializer

Encapsulate buffer management and provide a clean public API:

class SensorSerializer {
  private writer = new Sia();

  serialize(sensor: Sensor): Uint8Array {
    this.writer.seek(0);
    serializeSensor(this.writer, sensor);
    return this.writer.toUint8Array();
  }

  deserialize(data: Uint8Array): Sensor {
    const reader = new Sia(data);
    return deserializeSensor(reader);
  }

  serializeBatch(sensors: Sensor[]): Uint8Array {
    this.writer.seek(0);
    this.writer.addArray32(sensors, serializeSensor);
    return this.writer.toUint8Array();
  }

  deserializeBatch(data: Uint8Array): Sensor[] {
    const reader = new Sia(data);
    return reader.readArray32(deserializeSensor);
  }
}

Pattern 3: Generic Codec Helper

A reusable codec type that works with any data shape:

interface Codec<T> {
  serialize: (sia: Sia, value: T) => void;
  deserialize: (sia: Sia) => T;
}

const createCodec = <T>(
  serialize: (sia: Sia, value: T) => void,
  deserialize: (sia: Sia) => T,
): Codec<T> => ({ serialize, deserialize });

const sensorCodec = createCodec<Sensor>(serializeSensor, deserializeSensor);

// Generic encode/decode
const encode = <T>(value: T, codec: Codec<T>): Uint8Array => {
  const sia = new Sia();
  codec.serialize(sia, value);
  return sia.toUint8Array();
};

const decode = <T>(data: Uint8Array, codec: Codec<T>): T => {
  const sia = new Sia(data);
  return codec.deserialize(sia);
};

Complex Data Structures

Nested Objects

Serialize nested objects by calling inner serializers from outer ones:

interface Address {
  street: string;
  city: string;
  zip: string;
  country: string;
}

interface Company {
  name: string;
  address: Address;
  employeeCount: number;
}

interface Employee {
  id: number;
  name: string;
  company: Company;
  skills: string[];
}

// Layer serializers from inner to outer
const serializeAddress = (sia: Sia, addr: Address): void => {
  sia
    .addString8(addr.street)
    .addString8(addr.city)
    .addAscii8(addr.zip)
    .addAscii8(addr.country);
};

const deserializeAddress = (sia: Sia): Address => ({
  street: sia.readString8(),
  city: sia.readString8(),
  zip: sia.readAscii8(),
  country: sia.readAscii8(),
});

const serializeCompany = (sia: Sia, co: Company): void => {
  sia.addString8(co.name);
  serializeAddress(sia, co.address);
  sia.addUInt32(co.employeeCount);
};

const deserializeCompany = (sia: Sia): Company => ({
  name: sia.readString8(),
  address: deserializeAddress(sia),
  employeeCount: sia.readUInt32(),
});

const serializeEmployee = (sia: Sia, emp: Employee): void => {
  sia.addUInt32(emp.id).addString8(emp.name);
  serializeCompany(sia, emp.company);
  sia.addArray8(emp.skills, (s, skill) => s.addString8(skill));
};

const deserializeEmployee = (sia: Sia): Employee => ({
  id: sia.readUInt32(),
  name: sia.readString8(),
  company: deserializeCompany(sia),
  skills: sia.readArray8((s) => s.readString8()),
});

Optional Fields

Use a boolean flag to indicate the presence of optional data:

interface UserProfile {
  name: string;
  email: string;
  bio: string | null;
  avatarUrl: string | null;
  phoneNumber: string | null;
}

const serializeProfile = (sia: Sia, profile: UserProfile): void => {
  sia.addString8(profile.name).addString8(profile.email);

  // Each optional field gets a presence flag
  sia.addBool(profile.bio !== null);
  if (profile.bio !== null) {
    sia.addString16(profile.bio);
  }

  sia.addBool(profile.avatarUrl !== null);
  if (profile.avatarUrl !== null) {
    sia.addString8(profile.avatarUrl);
  }

  sia.addBool(profile.phoneNumber !== null);
  if (profile.phoneNumber !== null) {
    sia.addAscii8(profile.phoneNumber);
  }
};

const deserializeProfile = (sia: Sia): UserProfile => {
  const name = sia.readString8();
  const email = sia.readString8();

  const hasBio = sia.readBool();
  const bio = hasBio ? sia.readString16() : null;

  const hasAvatar = sia.readBool();
  const avatarUrl = hasAvatar ? sia.readString8() : null;

  const hasPhone = sia.readBool();
  const phoneNumber = hasPhone ? sia.readAscii8() : null;

  return { name, email, bio, avatarUrl, phoneNumber };
};

For structures with many optional fields, you can pack the presence flags into a single byte using bitwise operations. For example, 8 optional fields can be represented with a single UInt8 bitmask.

Discriminated Unions

Use a type tag byte to distinguish between variants:

interface Circle {
  type: "circle";
  radius: number;
}

interface Rectangle {
  type: "rectangle";
  width: number;
  height: number;
}

interface Triangle {
  type: "triangle";
  a: number;
  b: number;
  c: number;
}

type Shape = Circle | Rectangle | Triangle;

const SHAPE_TAGS = { circle: 0, rectangle: 1, triangle: 2 } as const;
const SHAPE_NAMES = ["circle", "rectangle", "triangle"] as const;

const serializeShape = (sia: Sia, shape: Shape): void => {
  sia.addUInt8(SHAPE_TAGS[shape.type]);

  switch (shape.type) {
    case "circle":
      sia.addUInt32(shape.radius);
      break;
    case "rectangle":
      sia.addUInt32(shape.width).addUInt32(shape.height);
      break;
    case "triangle":
      sia.addUInt32(shape.a).addUInt32(shape.b).addUInt32(shape.c);
      break;
  }
};

const deserializeShape = (sia: Sia): Shape => {
  const tag = sia.readUInt8();
  const type = SHAPE_NAMES[tag];

  switch (type) {
    case "circle":
      return { type, radius: sia.readUInt32() };
    case "rectangle":
      return { type, width: sia.readUInt32(), height: sia.readUInt32() };
    case "triangle":
      return {
        type,
        a: sia.readUInt32(),
        b: sia.readUInt32(),
        c: sia.readUInt32(),
      };
    default:
      throw new Error(`Unknown shape tag: ${tag}`);
  }
};

Maps and Dictionaries

Serialize key-value maps by converting to an array of entries:

const serializeMap = (sia: Sia, map: Map<string, number>): void => {
  const entries = Array.from(map.entries());
  sia.addArray16(entries, (s, [key, value]) => {
    s.addAscii8(key).addInt32(value);
  });
};

const deserializeMap = (sia: Sia): Map<string, number> => {
  const entries = sia.readArray16((s) => {
    const key = s.readAscii8();
    const value = s.readInt32();
    return [key, value] as [string, number];
  });
  return new Map(entries);
};

// Usage
const scores = new Map<string, number>();
scores.set("alice", 100);
scores.set("bob", 85);
scores.set("charlie", 92);

const sia = new Sia();
serializeMap(sia, scores);

sia.seek(0);
const decoded = deserializeMap(sia);
// Map { "alice" => 100, "bob" => 85, "charlie" => 92 }

For object dictionaries (Record<string, T>), convert to entries first:

const serializeRecord = (sia: Sia, record: Record<string, number>): void => {
  const entries = Object.entries(record);
  sia.addArray16(entries, (s, [key, value]) => {
    s.addAscii8(key).addInt32(value);
  });
};

const deserializeRecord = (sia: Sia): Record<string, number> => {
  const entries = sia.readArray16((s) => {
    const key = s.readAscii8();
    const value = s.readInt32();
    return [key, value] as [string, number];
  });
  return Object.fromEntries(entries);
};

Versioning and Evolution

Schema Versioning

Prefix serialized data with a version byte to support schema evolution:

const CURRENT_VERSION = 3;

interface Config {
  name: string;
  maxRetries: number;
  timeout: number;
  // Added in v2
  enableLogging: boolean;
  // Added in v3
  logLevel: number;
  tags: string[];
}

const serializeConfig = (sia: Sia, config: Config): void => {
  sia.addUInt8(CURRENT_VERSION);
  sia.addString8(config.name);
  sia.addUInt8(config.maxRetries);
  sia.addUInt32(config.timeout);
  // v2 fields
  sia.addBool(config.enableLogging);
  // v3 fields
  sia.addUInt8(config.logLevel);
  sia.addArray8(config.tags, (s, tag) => s.addString8(tag));
};

const deserializeConfig = (sia: Sia): Config => {
  const version = sia.readUInt8();

  const name = sia.readString8();
  const maxRetries = sia.readUInt8();
  const timeout = sia.readUInt32();

  // v2 fields with defaults for v1 data
  const enableLogging = version >= 2 ? sia.readBool() : false;

  // v3 fields with defaults for v1/v2 data
  const logLevel = version >= 3 ? sia.readUInt8() : 0;
  const tags = version >= 3 ? sia.readArray8((s) => s.readString8()) : [];

  return { name, maxRetries, timeout, enableLogging, logLevel, tags };
};

Forward Compatibility

To support forward compatibility (old readers handling new data), include a size prefix so unknown fields can be skipped:

const serializeWithSize = (sia: Sia, config: Config): void => {
  // Reserve space for the total size
  const sizeOffset = sia.offset;
  sia.addUInt32(0); // placeholder

  const dataStart = sia.offset;
  sia.addUInt8(CURRENT_VERSION);
  serializeConfig(sia, config);
  const dataEnd = sia.offset;

  // Write the actual size back
  const totalSize = dataEnd - dataStart;
  const savedOffset = sia.offset;
  sia.seek(sizeOffset);
  sia.addUInt32(totalSize);
  sia.seek(savedOffset);
};

const deserializeWithSize = (sia: Sia): Config => {
  const totalSize = sia.readUInt32();
  const startOffset = sia.offset;

  const config = deserializeConfig(sia);

  // Skip any unread bytes from a newer version
  sia.seek(startOffset + totalSize);

  return config;
};

Performance Optimization

Reusing Instances

For high-throughput scenarios, avoid creating new Sia instances:

class HighThroughputSerializer {
  private writer = new Sia();
  private reader = new Sia(new Uint8Array(0));

  serialize(sensor: Sensor): Uint8Array {
    this.writer.seek(0);
    serializeSensor(this.writer, sensor);
    return this.writer.toUint8Array();
  }

  deserialize(data: Uint8Array): Sensor {
    this.reader.setContent(data);
    return deserializeSensor(this.reader);
  }
}

Pre-Allocating Buffers

When you know the maximum size of your data, allocate once:

// Sensor: 2 (id) + 1 + 32 (name max) + 4 (value) + 8 (timestamp) = 47 bytes max
const SENSOR_MAX_SIZE = 64; // round up for safety

const serializeSensorFast = (sensor: Sensor): Uint8Array => {
  const sia = Sia.allocUnsafe(SENSOR_MAX_SIZE);
  serializeSensor(sia, sensor);
  return sia.toUint8Array();
};

Batch Processing

Process arrays of items efficiently:

const serializeSensorBatch = (sensors: Sensor[]): Uint8Array => {
  const sia = new Sia();
  sia.addArray32(sensors, serializeSensor);
  return sia.toUint8Array();
};

const deserializeSensorBatch = (data: Uint8Array): Sensor[] => {
  const sia = new Sia(data);
  return sia.readArray32(deserializeSensor);
};

Error Handling

Validation Before Serialization

Validate data before serializing to catch issues early:

const validateSensor = (sensor: Sensor): void => {
  if (sensor.id < 0 || sensor.id > 65535) {
    throw new Error(`Sensor ID out of UInt16 range: ${sensor.id}`);
  }
  if (sensor.name.length > 255) {
    throw new Error(`Sensor name too long for ASCII: ${sensor.name.length}`);
  }
};

const safeSerializeSensor = (sia: Sia, sensor: Sensor): void => {
  validateSensor(sensor);
  serializeSensor(sia, sensor);
};

Try-Catch Wrapper

Wrap deserialization with error context:

const safeDeserialize = <T>(
  data: Uint8Array,
  deserializer: (sia: Sia) => T,
  label: string,
): T => {
  try {
    const sia = new Sia(data);
    return deserializer(sia);
  } catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    throw new Error(
      `Failed to deserialize ${label} (${data.length} bytes): ${message}`,
    );
  }
};

// Usage
const sensor = safeDeserialize(bytes, deserializeSensor, "Sensor");

Summary

Embedding

Use embedSia and embedBytes to compose complex packets from separately built components.

Type-Safe Functions

Define serialize/deserialize function pairs for each type. Use with addArray* for collections.

Complex Structures

Handle nesting by composing serializers. Use boolean flags for optional fields and type tags for unions.

Versioning

Prefix data with a version byte. Use conditional reads with defaults for backward compatibility.

Edit this pageorReport an issue

Unsafe Buffer Allocation

How Sia's allocUnsafe works, when to use it, and how to avoid common pitfalls with shared buffer allocation.

Performance Benchmarks

Detailed performance comparisons of Sia against JSON, Protobuf, and MessagePack across serialization, deserialization, and output size.