Google Cloud Storage provider

Google Cloud Storage provider implementing storage.ProviderPort against the official cloud.google.com/go/storage SDK.

Overview

GCS is Google Cloud's object store. It offers strong read-after-write consistency and regional, dual-region, or multi-region buckets that replicate without a separate replication policy.

The provider maps Piko repositories to GCS buckets via RepositoryMappings, then plugs into the storage service through the standard provider options. It implements the same storage.ProviderPort as every other backend. Swapping S3 or R2 for GCS is a one-line change in the bootstrap. The provider uses the official Go SDK. It auto-enables chunked uploads for files over 100 MiB, does server-side copy via CopierFrom, and issues V4-signed presigned URLs for direct client uploads and downloads.

Batch PutMany and RemoveMany run a bounded concurrent worker pool instead of a sequential loop, which is why SupportsBatchOperations reports true. The default pool size is 5 concurrent uploads and 10 concurrent deletes, and each call honours the ContinueOnError flag and returns per-key success and failure results. The provider emits a full OpenTelemetry metric set with no extra wiring. The instruments cover operation duration, operation and error counts, bytes transferred, and batch and multipart counters.

Authentication uses Google's Application Default Credentials by default. Pass an explicit service account JSON via CredentialsJSON only when ADC is not available, for example running outside GCP without GOOGLE_APPLICATION_CREDENTIALS. All public methods are safe for concurrent use.

The provider applies a default rate limit of 100 calls per second with a burst of 200. Put, Get, Remove, Exists, and the batch operations wait on this limiter, so calls block once they pass the rate. The public wdk/storage facade does not re-export the rate-limit options, so most callers cannot change this default.

Requirements

A GCP project with the Cloud Storage API enabled and at least one bucket created beforehand.
IAM permissions on each mapped bucket: storage.objects.create, storage.objects.get, storage.objects.delete, storage.objects.list (the roles/storage.objectAdmin predefined role covers all four).
Authentication via Application Default Credentials (recommended on GCP, workload identity, GCE/GKE service account, or gcloud auth application-default login for local dev) or an explicit service account JSON passed via CredentialsJSON.
The signing permission for presigned URLs. The provider signs URLs with the V4 scheme. Under ADC or workload identity (no private key in the credentials), V4 signing calls the IAM signBlob API, which needs iam.serviceAccounts.signBlob (the roles/iam.serviceAccountTokenCreator role on the service account). Without it, PresignURL and PresignDownloadURL fail even though object reads and writes work.
Network egress to storage.googleapis.com.

Configuration

import (
    "context"
    "os"

    "piko.sh/piko/wdk/storage"
    "piko.sh/piko/wdk/storage/storage_provider_gcs"
)

provider, err := storage_provider_gcs.NewGCSProvider(ctx, storage_provider_gcs.Config{
    RepositoryMappings: map[string]string{ // required; logical repo -> bucket
        "uploads": "myapp-uploads-prod",
        "avatars": "myapp-avatars-prod",
    },
    CredentialsJSON: nil, // optional; nil = Application Default Credentials
})
if err != nil {
    return err
}

For local development outside GCP, load the service account JSON from disk and pass it as CredentialsJSON. On GCE, GKE, or Cloud Run, leave it nil and rely on the attached service account.

Bootstrap

ssr := piko.New(
    piko.WithStorageProvider("gcs", provider),
    piko.WithDefaultStorageProvider("gcs"),
)

Both options take any storage.ProviderPort, so the bootstrap is identical to S3, R2, or disk. Only the constructed provider value changes.

Hashing behaviour

GetHash returns the object's MD5 hash when GCS reports one. GCS omits MD5 for composite objects and for objects uploaded in chunks, including the large files this provider chunks automatically. When MD5 is absent, the provider streams the whole object and returns a SHA-256 hash instead. Code that relies on GetHash for integrity checks should expect a possible full-object download and a different algorithm for those objects.