Amazon S3 storage provider

AWS S3 provider implementing storage.ProviderPort against the AWS SDK for Go v2, with multipart uploads, server-side copy, presigned URLs, and batch deletes.

Overview

The provider satisfies storage.ProviderPort with a compile-time assertion, so it drops into the storage service like any other provider. It is interchangeable with the disk, GCS, and R2 providers. The same S3 engine backs the Cloudflare R2 provider, which builds a storage_provider_s3.Config and calls NewS3Provider under the hood.

The provider drives the AWS SDK for Go v2 directly. It uses the transfermanager for streaming multipart uploads and the s3.PresignClient for client-direct uploads and downloads. A Put uses multipart when the content exceeds 100 MB, when the size is unknown, or when the caller supplies an explicit multipart config. Smaller, known-size uploads use a single PutObject request. Reads and writes flow through a per-provider rate limiter (default 100 requests per second, burst 200), reusing piko's shared rate-limiter machinery; batch operations run through a per-provider worker pool that returns piko's shared batch-result types.

The provider maps piko repositories (logical names like uploads or avatars) to S3 buckets through RepositoryMappings. This indirection lets the same code target different buckets per environment. Set EndpointURL and UsePathStyle to point at any S3-compatible service (MinIO, LocalStack, Wasabi, Backblaze B2). All public methods are safe for concurrent use.

The provider lives in its own Go module under wdk/storage/storage_provider_s3. Register it in your compiled main.go through the bootstrap options below. Storage providers have no interpreted-mode symbols, so they run in compiled user code.

Requirements

  • An AWS account with at least one S3 bucket created beforehand, the provider does not create buckets.
  • IAM permissions on each mapped bucket: s3:PutObject, s3:GetObject, s3:DeleteObject, s3:ListBucket. Add s3:PutObjectAcl only if you set object ACLs, and s3:AbortMultipartUpload for multipart-upload cleanup.
  • Credentials via the standard AWS chain (environment, shared config, EC2/ECS/Lambda role) or via AccessKey/SecretKey for static credentials.
  • Network egress to the region's S3 endpoint (for example s3.eu-west-1.amazonaws.com).

Configuration

import (
    "context"

    "piko.sh/piko/wdk/storage"
    "piko.sh/piko/wdk/storage/storage_provider_s3"
)

provider, err := storage_provider_s3.NewS3Provider(ctx, &storage_provider_s3.Config{
    RepositoryMappings: map[string]string{ // required; logical repo -> bucket
        "uploads": "myapp-uploads-prod",
        "avatars": "myapp-avatars-prod",
    },
    Region:          "eu-west-1", // required for AWS; use "auto" for some S3-compatibles
    AccessKey:       "",          // optional; empty uses the AWS credential chain
    SecretKey:       "",          // optional; pair with AccessKey for static creds
    EndpointURL:     "",          // optional; set for MinIO, LocalStack, Wasabi, etc.
    UsePathStyle:    false,       // set true for path-style endpoints (MinIO/LocalStack)
    DisableChecksum: false,       // set true for S3-compatibles that reject checksums
})
if err != nil {
    return err
}

DisableChecksum turns off both request checksum calculation and response checksum validation. Leave it false against AWS S3. Set it true only for an S3-compatible service that rejects the checksum headers, and weigh the loss of integrity validation.

Pass storage.ProviderOption values as variadic arguments after the config to set the rate limiter. The default applied by the constructor is 100 requests per second with a burst of 200. An option that sets RateLimitConfig to zero values disables the limit.

The SDK retries each request, but the provider caps retries at two attempts. This cap is not configurable through Config. Handle S3 errors as recoverable in your action code instead of relying on deep SDK retry.

Bootstrap

ssr := piko.New(
    piko.WithStorageProvider("s3", provider),
    piko.WithDefaultStorageProvider("s3"),
)

One NewS3Provider call plus these two options is the entire wiring. WithDefaultStorageProvider selects the provider that handles operations when the caller names no provider. Drop the second option when you register a single provider.

See also

Other storage providers:

  • Google Cloud Storage, GCP-native object storage with strong consistency.
  • Cloudflare R2, S3-compatible storage with zero egress fees.
  • Disk, local filesystem storage for development and single-node deployments.

Storage transformers:

Framework docs:

External: