Goblin Store versus RocksDB on an 8 GiB object-cache host

Goblin Store populated a 62.1 GiB object corpus 31.3% faster than RocksDB and replayed the 266.1 GiB request stream 72.0% faster through the local C++ APIs. When the recorded chunk-ready times are passed through an idealized 0.937 Gbit/s network—the instance’s sustainable baseline— Goblin delivers the complete schedule at 46.7% higher throughput: 98.28 MiB/s versus 66.99 MiB/s.

This is a direct-I/O, single-request-at-a-time storage-engine comparison on an 8 GiB AWS instance. It is not a network-server benchmark. The network result is a deterministic simulation intended to answer a narrower question: if both engines sat behind an otherwise perfect baseline-rate server, could Goblin’s ordered, incremental delivery hide storage time behind transmission?

Results

Both engines populated the same 373,165 objects and then retrieved the same 6,141,630 requests in the same order. Population covered 66,693,472,653 bytes (62.113 GiB); the read schedule returned 285,757,364,203 bytes (266.132 GiB).

The aggregate completion rate is computed as total returned bytes divided by the sum of each serial request’s measured time. It is not the mean of per-request bandwidths.

Goblin’s resident heads also change when the first useful bytes become visible:

RocksDB’s DB::Get() materializes the whole value before returning, so its observable time to first byte equals its completion time. Goblin calls back with the resident head and then with ordered tail chunks as they become available. For the disk-backed portion of the workload, Goblin and RocksDB had nearly identical mean complete-value latency—320.908 µs and 320.988 µs respectively—but Goblin exposed the first bytes in 12.916 µs on average. That is the intended architectural difference: byte zero can be useful while byte n is still in flight.

Machine and storage layout

The benchmark ran on an AWS m8id.large: 2 vCPUs, 8 GiB of RAM, an Intel Xeon 6 (Granite Rapids) processor, and one 118 GB local NVMe SSD. AWS lists the device at up to 33,542 read IOPS and 16,771 write IOPS. The source corpus and schedules lived on an attached EBS volume; each disposable cache database lived on the local instance-store NVMe. The host had deliberately little RAM relative to the 62.1 GiB corpus, so neither implementation could turn the complete data set into an accidental page-cache benchmark. See the AWS general-purpose instance specifications.

Goblin received 1,120 MiB for packed small objects and 4,100 MiB for fixed resident heads. It used 16 KiB heads, 256 KiB tail-read chunks, four reads in flight, four population workers, and enough cached file descriptors to retain every disk-backed object. Persistent heads were enabled in a directory separate from the SSD-tail directory.

RocksDB received the same 5,220 MiB application-memory ceiling through a shared WriteBufferManager. This is a match between the engines’ explicit cache/write-buffer controls, not a claim that their total process RSS or metadata overhead is identical. RocksDB used:

rocksdb::BlockBasedTableOptions table_options;
table_options.no_block_cache = true;

rocksdb::Options options;
options.use_direct_reads = true;
options.use_direct_io_for_flush_and_compaction = true;

Compression, mmap reads, the WAL, and BlobDB were disabled. Population used 512 MiB memtables, up to eight write buffers, two background jobs, normal leveled compaction, one final flush, and a barrier for background work. Goblin used its ordinary direct-I/O SSD path and persisted the hashed key plus resident head with normal open/write/close head-file writes. Both engines therefore finished population in a restartable state before reads began.

The source cache was advised away before population and again after each source read. Reads were strictly serial. The timer stopped before each result row was written, and the CSV stream was flushed between requests, so a crash could not discard completed measurements and result logging was outside the timed interval.

The RocksDB measurements came from an earlier completed pass on the same instance, corpus, schedule, storage layout, and engine configuration. Goblin was rerun after its bounded population pipeline and persistent-head write path were optimized. This was not an interleaved A/B run, so small changes in host or device state remain a caveat; the size of the observed read and simulated delivery differences is much larger than that caveat is likely to explain.

The 0.937 Gbit/s idealized network

The raw CSV records every Goblin callback as size_in_bytes:ready_time_ns, in callback order. For each request the simulator computes:

chunk_start = max(chunk_ready, previous_chunk_end)
chunk_end   = chunk_start + ceil(chunk_size * 8 / 937,000,000)

The final chunk end is the request’s simulated last-byte arrival time. Requests remain serial, just as they were in the storage benchmark. RocksDB contributes one chunk at DB::Get() completion because it exposes no earlier bytes.

At 0.937 Gbit/s the payload itself requires 2,439.8 seconds of wire time. Goblin overlaps 613.3 seconds, or 25.1%, of that transmission with storage:

Goblin:  946.484 s storage + 2,439.768 s wire - 613.333 s overlap = 2,772.919 s
RocksDB: 1,628.405 s storage + 2,439.767 s wire -   0.000 s overlap = 4,068.172 s

The AWS network specification table for m8id.large lists 0.937 Gbit/s baseline and 12.5 Gbit/s burst bandwidth. This study uses the baseline, not the additional credit-based burst capacity that may be temporarily available. AWS explains that instances of this size earn network I/O credits below baseline, spend them while bursting, and return to baseline after the credits are exhausted. A cache is expected to remain under constant use, so the sustainable baseline is the appropriate long-running model. See Amazon EC2 instance network bandwidth.

The model has no TCP/IP headers, packetization, syscall or kernel-stack cost, congestion, retransmission, TLS, or receiver delay. Its value is comparative: it applies the same perfect link to both recorded storage traces and preserves the storage/wire overlap made possible by Goblin’s chunk callback.

The simulator is bench/simulate_chunk_network.py; the benchmark driver and analysis tools are in bench/.

Relation to Netflix’s RAM-to-SSD cache work

Netflix’s 2018 article “Evolution of Application Data Caching: From RAM to SSD” describes the same economic pressure behind this experiment: keeping large caches entirely in RAM is expensive, while NVMe can meet useful cache latency targets at much lower cost. Their EVCache Moneta design stored the complete data set in RocksDB on SSD and active data in Memcached. Netflix reported that this reduced some cluster footprints substantially, but overwrite compaction caused throughput and latency problems for larger online workloads. They ultimately favored memcached extstore, which retained metadata in RAM and put values on flash, avoided duplicate-record capacity pressure, and made many clusters network-bound rather than storage-bound.

This benchmark is related, but it does not reproduce Netflix’s production experiment. We do not model EVCache replication, Memcached’s hot-object tier, their 1 KiB/200K-IOPS test, cluster operations, or a production SLA. Instead, it isolates the local object-store decision on an actual 8 GiB host: ordinary RocksDB with direct I/O and no block cache versus Goblin’s application-owned RAM heads and direct-I/O tails, under matched explicit memory budgets and a trace-derived large-object workload.

The result supports the architectural direction Netflix described. A general LSM tree can be an excellent database, but its compaction and complete-value API are not free when the job is a replaceable object cache. Goblin’s layout is closer in spirit to extstore—small resident routing state plus flash-backed bytes—but retains an initial value prefix, not merely metadata, in RAM. That prefix is what makes ordered storage/network overlap possible. The simulation suggests that the benefit survives wrapping the local engine in a network server: Goblin’s 287.9 MiB/s local completion rate becomes 98.28 MiB/s on the idealized baseline link, while RocksDB’s 167.4 MiB/s becomes 66.99 MiB/s.

This is evidence for Goblin as a blob-cache engine, not a universal RocksDB replacement. A future networked rerun should use an actual peer, hold the instance above its baseline long enough to observe credit behavior, and measure server CPU, TTFB, and last-byte arrival directly.

Corpus provenance

The corpus was derived from the Encrypted Web Traffic Dataset: Event Logs and Packet Traces, published by Science Data Bank under CC BY 4.0. The event log’s hostnames and file sizes were mapped to opaque numeric keys and incompressible random payloads; the benchmark preserves the event-log access order. The generated corpus contains 373,165 unique objects with a mean size of 174.5 KiB, a 26.3 KiB median, a 2.495 MiB p99, and a 150.1 MiB maximum.