Running Goblin Store as an Ubuntu service

This guide runs Goblin Store under systemd with a dedicated account, bounded privileges, raised file and locked-memory limits, automatic restart after failure, graceful shutdown, and journald logs.

Goblin Store manual: Getting started · Installation · Command-line reference · Ubuntu service

Complete Installation first. The examples expect:

/usr/local/bin/goblin-store
/usr/local/bin/goblin-store-path-prep
a 4 GiB resident-memory budget
persistent heads and SSD data under /var/lib/goblin-store
local-only memcache and HTTP listeners

Replace the example storage paths with dedicated mounted filesystems where appropriate.

Create the service account

sudo useradd --system \
    --home-dir /var/lib/goblin-store \
    --create-home \
    --shell /usr/sbin/nologin \
    goblin-store

sudo install -d \
    -o goblin-store \
    -g goblin-store \
    -m 0750 \
    /var/lib/goblin-store

The service never needs a login shell or ownership of its executable.

Prepare the storage directories

Run the marker-gated preparation helper as the service account:

sudo -u goblin-store /usr/local/bin/goblin-store-path-prep \
    /var/lib/goblin-store/heads \
    /var/lib/goblin-store/ssd

Each target must be empty the first time it is prepared. The example enables --heads-path, so valid objects can be recovered after an ordinary restart. For an ephemeral cache, omit the heads directory and --heads-path; the prepared SSD pool will be cleared at startup.

For a three-tier deployment, prepare and add one or more HDD paths:

sudo -u goblin-store /usr/local/bin/goblin-store-path-prep \
    /mnt/goblin-hdd/pool

Make sure every /mnt/... filesystem is mounted before starting the service. Add RequiresMountsFor=/mnt/goblin-hdd/pool to the unit when systemd manages the mount dependency.

Install the systemd unit

Create /etc/systemd/system/goblin-store.service:

[Unit]
Description=Goblin Store large-object cache
Documentation=https://goblin-store.dev/docs/getting-started.html
Wants=network-online.target
After=network-online.target local-fs.target

[Service]
Type=simple
User=goblin-store
Group=goblin-store
UMask=0027

ExecStart=/usr/local/bin/goblin-store \
    --heads-path /var/lib/goblin-store/heads \
    --ssd-dir /var/lib/goblin-store/ssd \
    --memory 4G \
    --listen-address 127.0.0.1

Restart=on-failure
RestartSec=2s
TimeoutStopSec=30s

# Raise these with the configured RAM and descriptor-cache budgets.
LimitMEMLOCK=5G
LimitNOFILE=262144

# CAP_IPC_LOCK permits locked/HugeTLB memory; no other capability is retained.
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_IPC_LOCK
NoNewPrivileges=true

PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/goblin-store
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true
RestrictSUIDSGID=true
LockPersonality=true

[Install]
WantedBy=multi-user.target

LimitMEMLOCK must cover the locked pools selected on the command line; do not copy the example 5 GiB limit unchanged when increasing --memory or adding --small-memory. LimitNOFILE must cover listeners, active connections, cached descriptors, pool directories, and normal process headroom.

The unit grants only CAP_IPC_LOCK. If your deployment deliberately uses --no-mlock and does not need explicit HugeTLB allocation, remove both capability lines and lower LimitMEMLOCK.

ProtectSystem=strict makes the filesystem read-only except for ReadWritePaths. Add every SSD, HDD, and head root outside /var/lib/goblin-store to that directive:

ReadWritePaths=/var/lib/goblin-store /mnt/goblin-ssd/pool /mnt/goblin-hdd/pool

Native RDMA deployments also need access to the installed RDMA userspace stack and /dev/infiniband; test additional hardening changes with the exact HCA and driver in use.

Start and verify

sudo systemctl daemon-reload
sudo systemctl enable --now goblin-store

systemctl status goblin-store
curl --fail --silent --show-error \
    http://127.0.0.1:8080/__goblin/ready

A healthy readiness response uses HTTP 200 and includes "ready":true. A degraded or read-only storage state returns HTTP 503 with tier details.

Follow the service log:

sudo journalctl -u goblin-store -f

Goblin handles SIGTERM, stops accepting work, drains in-flight transfers, and then exits. systemctl stop goblin-store therefore performs a graceful shutdown.

Change the configuration

Goblin Store has no reloadable configuration file; the command line is the configuration. Edit ExecStart, validate the resulting arguments, then restart:

/usr/local/bin/goblin-store --help
sudo systemctl daemon-reload
sudo systemctl restart goblin-store
sudo systemctl status goblin-store

Useful production changes include:

an exact external --listen-address, which also lets Goblin discover the serving NIC’s NUMA node;
explicit --numa NODE on a multi-interface or deliberately pinned host;
--hdd-dir and --ssd-prefix for a three-tier layout;
--file-handle-cache together with a suitable LimitNOFILE;
--small-memory to keep packed small objects from consuming the fixed-head pool;
TLS and authentication before exposing memcache outside a trusted network.

See the complete command-line reference before changing combinations.

Add memcache TLS and authentication

Create a credentials file owned by the service with no group/other permissions:

sudo install -d -o root -g goblin-store -m 0750 /etc/goblin-store
sudo install -o goblin-store -g goblin-store -m 0600 \
    ./goblin-users \
    /etc/goblin-store/users

The file format is one user:password record per line. Install the certificate and private key with similarly restrictive ownership, then extend ExecStart:

    --listen-address 10.0.0.20 \
    --memcache-tls \
    --auth-file /etc/goblin-store/users \
    --tls-cert /etc/goblin-store/server.crt \
    --tls-key /etc/goblin-store/server.key \
    --no-http

The default remains loopback-only. Never change it to 0.0.0.0 or :: without also deciding how the service will be authenticated and firewalled.

NUMA and HugeTLB checks

At startup Goblin reports its serving node, selected CPUs, memory regions, allocation block, and head geometry. Inspect the host before choosing explicit placement:

lscpu
numactl --hardware
grep -E 'HugePages|Hugepagesize' /proc/meminfo

HugeTLB allocation is best effort on every selected NUMA node. If the required HugeTLB pool is not available, Goblin falls back to ordinary locked memory and keeps the same logical --block geometry. Reserve HugeTLB pages according to the host’s page size and NUMA policy; do not assume every architecture uses 2 MiB pages.

Troubleshooting

The service exits immediately

Read the configuration error:

sudo journalctl -u goblin-store -n 100 --no-pager

Common causes are an unprepared pool, a head path that aliases an SSD/HDD path, insufficient LimitMEMLOCK, missing liburing, ambiguous NUMA locality, or an unavailable TLS/RDMA build feature.

Permission denied in a pool

Confirm every configured root and marker belongs to the service:

sudo chown -R goblin-store:goblin-store /var/lib/goblin-store
sudo -u goblin-store test -w /var/lib/goblin-store/ssd

Also add external mount paths to ReadWritePaths.

Too many open files

Compare --file-handle-cache, connection limits, and the effective service limit:

systemctl show goblin-store -p LimitNOFILE

Raise LimitNOFILE, run systemctl daemon-reload, and restart.