Zstd cache transformer

Cache value compression transformer implementing cache.TransformerPort using Zstandard. It compresses on write and decompresses on read, so payloads in Redis, Valkey, or any other backend take less memory and less wire bandwidth.

Overview

Zstandard (zstd) is a general-purpose compression algorithm. It produces a better compression ratio than gzip at similar speeds and decompresses faster. Cache traffic favours read-heavy workloads. Callers fetch each entry more times than they store it. That asymmetric read and write speed profile suits any payload large enough that compression matters.

Reach for the zstd transformer when cached values are large. Rendered HTML fragments, JSON blobs over a kilobyte, and binary payloads with redundant structure all qualify. For small values, under a hundred bytes or so, compression overhead can dominate. Measure before assuming. Pair with the crypto transformer by ordering compression first (lower priority number) and encryption second, so the ciphertext entropy does not defeat compression.

The transformer default priority is 100, earlier in the write pipeline than the default crypto transformer at 250. The pipeline sorts on this priority field, so compression and encryption compose without manual ordering code. Compression level defaults to zstd.SpeedDefault (level 3), the standard fast and small balance. Raise to SpeedBetterCompression for archival-style entries, or lower to SpeedFastest when CPU is scarce.

The transformer ships hardened defaults. Reverse caps the decompressed output at DefaultMaxDecompressedCacheBytes (64 MiB) so a crafted entry cannot exhaust caller memory. A payload that exceeds the cap makes Reverse return ErrDecompressedCacheTooLarge, which you detect with errors.Is. Tune the cap with the MaxDecompressedBytes field or the WithMaxDecompressedCacheBytes option. A non-positive value disables the cap, which is only safe for fully trusted cache contents. The transformer is safe for concurrent use and satisfies io.Closer through a once-guarded Close, so it slots into long-lived services.

Configuration

import (
    "github.com/klauspost/compress/zstd"

    "piko.sh/piko/wdk/cache/cache_transformer_zstd"
)

transformer, err := cache_transformer_zstd.New(cache_transformer_zstd.Config{
    Level:                zstd.SpeedBetterCompression, // compression level; default level 3
    MaxDecompressedBytes: 128 * 1024 * 1024,           // cap on read; default 64 MiB
})
if err != nil {
    return err
}

Every field is optional. An empty Config resolves the transformer name to zstd, priority to 100, level to zstd.SpeedDefault, and the decompression cap to 64 MiB. cache_transformer_zstd.DefaultConfig() returns those same defaults as a value you can adjust.

Bootstrap

Cache transformers do not have a piko.With* option. Attach them through the cache builder when creating a namespace. NewCacheBuilder returns a builder and an error, so check the error before chaining:

import (
    "piko.sh/piko/wdk/cache"
)

builder, err := cache.NewCacheBuilder[string, User](service)
if err != nil {
    return err
}

userCache, err := builder.
    Provider("redis").
    Namespace("users").
    Compression(). // shorthand: defaults-only zstd
    Build(ctx)

Compression() resolves the built-in zstd transformer with default settings. The adapter registers that transformer during package initialisation, so you skip any manual registration step. The shorthand always uses defaults and cannot carry a custom level.

To attach a configured instance, pass a Config to the Transformer method. The builder resolves the same built-in zstd blueprint and applies your settings:

import (
    "github.com/klauspost/compress/zstd"

    "piko.sh/piko/wdk/cache"
    "piko.sh/piko/wdk/cache/cache_transformer_zstd"
)

userCache, err := builder.
    Provider("redis").
    Namespace("users").
    Transformer("zstd", cache_transformer_zstd.Config{
        Level: zstd.SpeedBetterCompression,
    }).
    Build(ctx)

See also

Other cache transformers and encoders:

Cache providers that benefit from compression:

Framework docs:

External: