Shrinking Message Queue Records at the Edge

Queues Are Full of Small Records

Message queues — SQS, Kafka, Cloudflare Queues, Redis streams — are built for high volumes of small messages. A queue record is typically a structured event under 1KB: a job, a notification, a state change. That puts queue payloads squarely in smoltext's target band.

Why Compress at the Edge

Compressing before a message enters the queue has compounding benefits:

• Throughput: smaller messages mean more fit per batch and per partition.
• Storage: queues that retain messages bill on byte volume.
• Egress: cross-region or cross-cloud queue traffic is billed per byte.

Doing the compression at the Cloudflare edge in sub-millisecond time means it does not slow down the producer.

A Job Record Example

{"job":"resize_image","id":"a91","w":800,"h":600,"fmt":"webp","q":82}

Around 68 bytes. The keys and the enumerated values (resize_image, webp) recur across the whole job stream, so smoltext's trained codebook and dictionary deflate reference them rather than re-encoding. Records like this typically drop to the low-30-byte range.

The Producer-Consumer Pattern

1. Producer builds the record.
2. Producer compresses via https://api.smoltext.sprapp.com/v1/compress.
3. Compressed bytes go onto the queue.
4. Consumer decompresses on receipt.

Keep a one-byte version marker on each record so consumers can handle both compressed and legacy formats during rollout.

Batching to Amortize Overhead

If your producer emits many records, batch them into a single compress request rather than one call per record. This amortizes request overhead and improves throughput. smoltext supports compressing arrays of small payloads in one round trip.

What Not to Compress

Large queue payloads — say, a record carrying an embedded document or image — are better handled by a general compressor or left raw. smoltext is for the small structured records that dominate most queue traffic, not the occasional large outlier.

Idempotency and Determinism

smoltext compression is deterministic and lossless, so a record always decompresses to exactly what was enqueued. This matters for idempotent consumers that hash payloads — the decompressed bytes match the original exactly.

Measuring the Win

Sample a few thousand real queue records, compress them, and compute the aggregate ratio. Uniform job and event streams are among the best-fit workloads because they are small and endlessly repetitive. Once you confirm the ratio, the savings scale linearly with your message volume.