The Goblin Store silkie mascot

Large-object cache + HTTP server/proxy

First byte from RAM. The rest is already on its way.

Goblin Store keeps each object’s latency-sensitive head in RAM, streams the middle from SSD, and hoards the cold tail on cheap HDD. It speaks memcache over TCP or native InfiniBand RDMA, and HTTP as either a server or a caching proxy, but large objects no longer require an all-RAM bill.

Get started Read the full benchmarks Feed the goblin bugs →
New · preliminary 43× faster median first byte. 59% higher median request bandwidth. Read the report →

Open source · Apache-2.0 · C++23 · NUMA-aware · HugeTLB · Linux io_uring · memcache/TCP · native InfiniBand/RoCE · HTTP/HTTPS

7.4× less RAM than memcached at the same large-object latency
+68% throughput over memcached extstore on the same HDD
~25% lower illustrative monthly cost than extstore
100% of the disk-sized set retained; extstore shed 41.5% under write load

The mission

Beat memcached on large objects.

Match memcached’s hit latency with a fraction of its RAM. Beat extstore when cold reads reach disk. Store more bytes per dollar by placing every part of an object on the tier fast enough for when the wire needs it.

  • Binary artifacts and packages
  • CDN and edge-cache objects
  • Media fragments and model blobs
  • Values from 256 KiB into the multi-MiB range

How it works

The network becomes the latency budget.

As a request lands, Goblin Store launches the storage pipeline. The RAM head starts leaving immediately while SSD and HDD reads run ahead of the client’s current byte position.

Position, not temperature

Placement follows byte offset: head, middle, tail. There is no hotness tracker moving whole objects between tiers.

Bounded all the way down

Fixed head and I/O pools, explicit backpressure, O_DIRECT, and chunked streaming keep RAM independent of object size.

Measured, not imagined

More object. Less memory.

A 2-vCPU server faced a 16-vCPU load box across a real NIC with a right-skewed mix of 256 KiB–8 MiB objects. One run fit in memory. The other forced cold reads onto the same slow HDD used by extstore.

Full rig, methodology, tables, and caveats →

Goblin vs RocksDB on 8 GiB: 31% faster fills, 72% faster local reads, 47% higher baseline-rate delivery →

Optane small-object concurrency: 54.1 Gbit/s and tighter delivery times →

NUMA interconnect-bandwidth report →

68 million requests: local vs remote first-byte latency →

Native InfiniBand: 38.2 Gbit/s and 7.7 us median header TTFB →

Measured results · lower RAM is better · higher throughput is better
Memory-sized working set · resident RAM
memcached1,974 MiB
Goblin Store267 MiB
Disk-sized working set · throughput
memcached + extstore70 QPS
Goblin Store · three tiers118 QPS
Disk-sized run QPS Avg latency RAM SSD HDD bulk ~$/mo
memcached + extstore 70 226 ms 2,188 MiB — 9.5 GiB $9.0
Goblin Store · 2-tier 93 170 ms 1,520 MiB 10.7 GiB — $6.9
Goblin Store · 3-tier 118 135 ms 1,520 MiB 2.84 GiB 8.9 GiB $6.6

Illustrative cost model: RAM $4, gp3 SSD $0.08, st1 HDD $0.045 per GiB-month. Point-in-time numbers on modest AWS hardware; reproduce with your own object mix.

Choose by object shape

Big tails belong here. Tiny objects have another goblin.

Goblin Store’s advantage appears when objects are large enough to tier. Sub-2 KiB values do not have a useful tail to put on disk, so a large-object allocator is the wrong tool.

Operational shape

Familiar at the edge. Purpose-built underneath.

01 / protocol

TCP and native RDMA

Classic memcache over TCP, plus a native InfiniBand/RoCE v3 path: small commands stay inline while object bodies use registered, credit-controlled bulk windows. HTTP/1.1 adds GET, HEAD, ranges, ETags, optional HTTPS, and a query-aware streaming origin cache with --mirror.

02 / runtime

NUMA-aware data paths

TCP/HTTP use core-local loops; each native RDMA connection owns its QP and progress thread. Exact HCA addresses select their NUMA node, heads pack into HugeTLB-backed blocks, and hot full blocks move off remote memory. FPGA NICs and direct userspace networking are on the development path.

03 / safety

Ephemeral on purpose

Pool directories must be explicitly blessed before startup can wipe them. Atomic publish, bounded memory, and backpressure keep partial objects and surprise OOMs out of the serving path.

Build + run

Put the cave under load.

Goblin Store targets Linux. Prepare dedicated pool directories once, then choose RAM, SSD, and optional HDD budgets at launch.

Linux ≥ 5.19 C++23 CMake ≥ 3.28 liburing OpenSSL libcurl (--mirror) libibverbs + librdmacm (optional) 
# Build and test
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure

# Bless empty, dedicated pool directories once
./build/goblin-store-path-prep /mnt/ssd/pool /mnt/hdd/pool

# Serve memcache on 11211 and HTTP on 8080
./build/goblin-store --memory 4G \
  --ssd-dir /mnt/ssd/pool --hdd-dir /mnt/hdd/pool \
  --memcache-port 11211 --http-port 8080

# Or fill HTTP misses from an origin while streaming them
./build/goblin-store --memory 4G --ssd-dir /mnt/ssd/pool \
  --mirror https://origin.example --http-port 8080
Getting started → Installation → Command-line flags → Run as an Ubuntu service → Repository README → 22 architecture decisions → HTTP mirror-cache design → Native RDMA bulk-window design → Native InfiniBand performance report → Goblin versus RocksDB object-cache report → NUMA interconnect-bandwidth report → Direct-link NUMA latency study → Source + issues →

Feed the goblin

Chickens eat bugs. This goblin eats software bugs.

If Goblin Store saves you RAM, feed it through Buy Me a Coffee and keep the open-source chicken pecking through the bug pile.

Buy the goblin some bugs →