Goblin StoreTechnical documentation

Field notes from the cave

Goblin Store command-line reference

This page documents every option accepted by the goblin-store server in version 0.0.4. Defaults shown here are the executable defaults; startup validation may impose additional relationships between options.

Goblin Store manual: Getting started · Installation · Command-line reference · Ubuntu service

Syntax

goblin-store [options]

SIZE values are integer bytes with an optional case-insensitive binary suffix: K = KiB, M = MiB, and G = GiB. Durations are integer milliseconds. Options described as repeatable may be supplied more than once.

At least one --ssd-dir is required. SSD, HDD, and persistent-head paths must first be prepared with goblin-store-path-prep.

Help and execution

Option Default Description
-h, --help — Print the built-in option summary and exit.
--cores N 0 Protocol workers. Zero uses every CPU allowed on the selected NUMA node.
--net MODE async Network backend: async, blocking, or exasock. ExaSock requires an opt-in build and an exact non-wildcard listening address.

The production async backend requires a Linux build with liburing. blocking is primarily a portable/fallback backend. ExaSock is Linux-only, must already be installed, and is never vendored.

Memory, heads, and NUMA

Option Default Description
--memory SIZE 1G Fixed-head RAM on the preferred/local NUMA node. Must be at least one --block and a multiple of it. In legacy mode, packed small objects share this pool.
--sub-memory SIZE none Fixed-head RAM on each non-local NUMA node. Requires explicit --numa; must be nonzero and a multiple of --block.
--small-memory SIZE omitted Opt into a physically separate local pool for packed objects smaller than --ram-head. Must be nonzero and a multiple of --block. Omission preserves the shared-pool layout.
--small-sub-memory SIZE none Separate small-object RAM on each non-local node. Requires --small-memory and explicit --numa; must be nonzero and a multiple of --block.
--block SIZE 2M x86; 32M Arm/LoongArch Allocation and NUMA-promotion block. Must be a power of two and a multiple of the platform HugeTLB page size.
--small-min-alloc SIZE 16 Alignment/minimum allocation granule for packed small objects. Power of two from 8 bytes through --block.
--ram-head SIZE 256K Maximum resident prefix per fixed-head object. Power of two, at least 4 KiB, no larger than --block or --ssd-prefix, and must divide --block.
--no-mlock off Do not lock ordinary RAM mappings. Intended for development; production should raise RLIMIT_MEMLOCK. HugeTLB mappings remain inherently unswappable.
--numa NODE listener NIC node Bind workers to this Linux NUMA node and make it the serving node. Overrides automatic listener-interface selection.
--no-numa off Disable CPU affinity, NUMA allocation policy, and promotion. Incompatible with --numa, --sub-memory, and --small-sub-memory.
--perverse off Benchmark-only: keep workers on the serving node but prefer head memory on the farthest NUMA node. Requires NUMA mode.
--no-numa-promotion off Keep NUMA placement and score maintenance but disable hot/cold full-block swaps.
--increment FLOAT 1.0 Positive finite score added after each successful logical object read.
--decay FLOAT 0.5 Once-per-minute popularity multiplier; finite and strictly between 0 and 1.

Without --numa, Goblin maps exact listener addresses to Linux interfaces and selects their NUMA node. If active listeners map to different nodes, startup reports every interface/address/node and asks for an explicit choice. HugeTLB allocation is best effort per node and falls back to ordinary memory without changing allocation-block geometry.

Disk tiers, persistence, and streaming I/O

Option Default Description
--ssd-dir DIR required One prepared SSD pool directory. Repeat for multiple physical SSD paths; objects are striped across them.
--hdd-dir DIR none One prepared HDD pool directory. Repeatable. Supplying any HDD directory enables the RAM/SSD/HDD layout.
--heads-path DIR none Prepared, separate persistent-head catalog. Enables restart recovery instead of wipe-on-start for the configured pools.
--ssd-prefix SIZE 32M Bytes of a three-tier object assigned to SSD, including the head. Must be 4 KiB-aligned and at least --ram-head.
--io-chunk SIZE 256K Aligned maximum tail-read/callback chunk. Power of two and at least 4 KiB. Does not change object layout.
--write-io-chunk SIZE 256K; mirror default 1M Write-staging quantum. Power of two and at least 4 KiB. Mirror mode raises an unspecified value to 1 MiB; an explicit value always wins.
--file-handle-cache N 128 Process-wide sharded CLOCK cache of immutable read descriptors. Must be a nonzero power of two; raise RLIMIT_NOFILE for large values.
--io-buffers N 64 Streaming buffers per worker; must be at least one.
--no-read-ahead off Disable the normal four-buffer GET read-ahead and issue tail reads serially. Intended for A/B testing.
--max-object-size SIZE 4G Reject larger objects before body admission. Range is 1 byte through the 4 GiB hard limit.
--source DIR none Preload a directory tree before listeners start. Repeatable. Relative file paths become object keys under the selected HTTP key rules.

In a three-tier configuration, --ssd-prefix - --ram-head must be at least one HDD stripe unit. The current SSD/HDD stripe unit is 64 KiB. Persistent heads, SSD pools, and HDD pools must use different roots because their published components share digest-derived filenames.

Capacity, eviction, and storage health

Option Default Description
--eviction NAME s3fifo Head/whole-object eviction policy. s3fifo is currently the only implemented selection.
--max-objects N 0 Bound disk-backed object count. Zero is unbounded; admission evicts whole objects above a nonzero bound.
--disk-high-watermark FLOAT 0.90 Begin proactive filesystem reclamation at this used-capacity fraction.
--disk-low-watermark FLOAT 0.80 Reclaim down to this used-capacity fraction. Must satisfy 0 < low < high <= 1.
--disk-reclaim-interval MS 1000 Filesystem health and background-reclamation cadence. Zero disables the periodic pass.

Disk-read corruption quarantines the failed generation. Storage readiness, capacity, inode availability, tier errors, reclamation, and quarantine state are visible through memcache stats and GET /__goblin/ready.

TCP, Unix, and overload controls

Option Default Description
--listen-address ADDRESS 127.0.0.1 Numeric IPv4 or IPv6 address shared by TCP listeners. Use 0.0.0.0 or :: explicitly for wildcard exposure.
--memcache-port N 11211 Memcache TCP port.
--http-port N 8080 Plain HTTP port.
--https-port N 8443 HTTPS port when HTTPS is enabled.
--no-memcache off Disable memcache over TCP. Unix-socket and native-RDMA memcache are independent.
--no-http off Disable only the plaintext HTTP listener. HTTPS is independent.
--max-connections N 1024 Aggregate connection ceiling for TCP/Unix stream listeners; must be at least one.
--listen-backlog N 1024 TCP completed-connection backlog; range 1 through INT_MAX. The kernel may clamp it.
--idle-timeout MS 300000 Close idle TCP keepalives after this interval. Zero disables the timeout.
--io-timeout MS 30000 Drop a stalled in-flight transfer, such as a slow reader. Zero disables the timeout.
--queue-timeout MS 30000 Maximum wait for a GET read buffer or SET write buffer. Zero disables the deadline.
--max-get-waiters N 64 Maximum queued GET buffer waiters per worker.
--max-set-waiters N 64 Maximum queued SET buffer waiters per worker.

At least one TCP, Unix, or native-RDMA memcache listener, or one HTTP/HTTPS listener, must remain enabled.

TLS, authentication, and Unix memcache

Option Default Description
--memcache-tls off Replace plaintext on --memcache-port with TLS 1.3. Requires the async backend and at least one certificate/key pair; incompatible with ExaSock.
--auth-file FILE none Require memcached ASCII authentication using user:password records. The regular file must have no group/other permission bits. Protects TCP and Unix memcache; currently incompatible with native RDMA.
--memcache-socket PATH none Also serve plaintext memcache on an AF_UNIX socket. A stale socket is replaced; a live socket or non-socket is never removed.
--memcache-socket-mode OCTAL 0600 Unix-socket permissions, from 0000 through 0777.
--tls-cert FILE none PEM certificate chain. Repeat once per SNI domain and pair by position with --tls-key. The first pair is the no-SNI/default memcache certificate.
--tls-key FILE none PEM private key paired with the corresponding --tls-cert.
--https inferred for HTTPS-only certs Explicitly enable HTTPS. Required when reusing memcache-TLS certificates for HTTPS.
--no-https off Suppress HTTPS inference from certificate/key options. Mutually exclusive with --https.

Supplying certificate/key pairs normally enables HTTPS. When --memcache-tls is set, the certificates enable only memcache TLS unless --https is also supplied. TLS requires an OpenSSL-enabled build; the async memcache path also requires Linux kTLS at runtime.

HTTP keys and reverse-cache mode

Option Default Description
--mirror URL none Act as a streaming cache for an absolute http:// or https:// origin/base path. The configured URL may not contain a query or fragment. Incompatible with virtual-host key mode.
--mirror-client MODE curl Upstream client: curl or uring. The native io_uring client accepts only well-behaved HTTP/1.1 over an http:// origin.
--http-vhost off Derive ordinary HTTP keys as lowercase Host + path rather than URI path alone.
--virtual-host off Alias for --http-vhost.
--key-on-query off Include the query string in ordinary HTTP keys. Mirror mode always uses query-aware cache keys.
--key-strip-slash off Drop the leading slash from path-mode keys so memcache key foo matches HTTP GET /foo. Does not alter virtual-host keys.
--http-index NAME index.html Map an HTTP path ending in / to this index filename.
--no-http-index off Disable HTTP directory-index mapping.

On a mirror miss, Goblin joins the requested path to the configured base URL, streams the origin response to the client and storage together, and applies HTTP cache-control semantics. An early client disconnect does not cancel a healthy cache fill.

Native InfiniBand/RoCE memcache

Option Default Description
--rdma ADDRESS disabled Enable the native reliable-connected RDMA memcache endpoint on a numeric IPv4/IPv6 address. This is not TCP over IPoIB.
--rdma-port N 11211 RDMA CM port; must be from 1 through 65535.
--rdma-ring SIZE 64K Per-connection control-ring budget. It must hold at least two fixed control slots. Object bodies do not travel in this ring.
--rdma-window SIZE 256K Registered bulk-window size. Power of two and at least 4 KiB.
--rdma-windows N 4 Bulk windows in each direction. At least two are required so tail I/O can overlap the resident head.

--rdma requires a Linux build with libibverbs and librdmacm. Without explicit --numa, its exact address participates in HCA/interface locality selection. ASCII --auth-file does not currently cover the native RDMA protocol, so the two options are rejected together.

Common configurations

Local development:

goblin-store --ssd-dir /var/tmp/goblin/ssd \
    --memory 256M --no-numa --no-mlock

External TLS memcache with authentication:

goblin-store --ssd-dir /mnt/nvme/goblin \
    --memory 4G \
    --listen-address 10.0.0.20 \
    --memcache-tls \
    --auth-file /etc/goblin-store/users \
    --tls-cert /etc/goblin-store/server.crt \
    --tls-key /etc/goblin-store/server.key \
    --no-http

Streaming HTTP reverse cache:

goblin-store --ssd-dir /mnt/nvme/goblin \
    --memory 8G \
    --mirror https://origin.example/assets \
    --no-memcache

For a supervised deployment, continue with Run as an Ubuntu service.