Batching Small Payloads: Amortizing Request Overhead

The Hidden Cost of Tiny Records

smoltext compresses small payloads beautifully, but if you make one API call per 40-byte record, the network round trip dwarfs the work. The fix is batching: send many small records in one request.

Why Batching Helps

Each HTTP request carries fixed overhead — connection handling, headers, TLS. For a tiny payload, that overhead can exceed the payload itself. By packing hundreds of records into a single call to https://api.smoltext.sprapp.com/v1/compress, you amortize that overhead across the whole batch.

The Batch Pattern

Instead of:

for record in records:
    compress(record)   # one round trip each — slow

Do:

compress_batch(records)   # one round trip for all

smoltext accepts an array of payloads and returns an array of compressed results in the same order, so mapping results back to records is trivial.

Cross-Record Savings

Batching has a second benefit beyond round-trip amortization. When records share keys and value vocabulary — which small structured records almost always do — the trained codebook and shared dictionary apply uniformly across the batch. You get consistent ratios because the same patterns recur in every record.

Choosing a Batch Size

There is a tradeoff. Larger batches amortize overhead better but increase latency before the batch is ready and the memory you hold. A few hundred to a few thousand records per batch is a common sweet spot for event and log pipelines. Tune it against your latency budget.

Streaming Pipelines

In a streaming ingestion path, accumulate records into a buffer and flush either when the buffer reaches a size threshold or after a short time window — whichever comes first. This caps both batch size and the latency any single record waits.

Decompression in Batches

The same logic applies on read. If you need many records at once — say, hydrating a page of analytics — decompress them in a single batch call rather than one at a time.

When Single-Call Is Fine

If you compress on the rare write rather than in a hot loop, single calls are perfectly fine and simpler. Batching matters specifically for high-throughput paths where request overhead would otherwise dominate. Profile your path before adding batching complexity.

Measuring the Improvement

Compare end-to-end throughput with single calls versus batches on your real volume. For high-frequency small-record workloads, batching commonly delivers an order-of-magnitude throughput improvement while keeping the same per-record compression ratio.