Goblin Store versus RocksDB object-cache benchmark

bench_goblin_and_rocksdb compares Goblin Store’s embedded C++ storage API with ordinary RocksDB on the same machine and backing filesystem. It does not enable RocksDB’s integrated BlobDB, and neither side crosses a socket or parses a network protocol.

The default Goblin budget is 1120 MiB for packed small objects plus 4100 MiB for fixed heads. RocksDB receives their 5220 MiB sum as its write-buffer ceiling. These settings fit the fixed corpus on an 8 GiB host while leaving memory for Goblin’s canonical index and policies, the benchmark catalog and schedule, file-descriptor and filesystem metadata, the kernel, and normal daemons. Its block cache is disabled. The benchmark does not impose a host-memory or cgroup limit; the host must have enough RLIMIT_MEMLOCK for Goblin’s pools.

The benchmark is opt-in because RocksDB is not a Goblin Store dependency:

sudo apt install librocksdb-dev
cmake -S . -B build-rocks-bench -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DGOBLIN_BUILD_ROCKSDB_BENCHMARK=ON
cmake --build build-rocks-bench --target bench_goblin_and_rocksdb

Run it with a source tree, a parent for disposable databases, a unique list of keys to preload, and the ordered list of keys to retrieve:

./build-rocks-bench/bench_goblin_and_rocksdb \
    --data /mnt/ebs/source \
    --scratch /mnt/local/benchmark-scratch \
    --prefetch /mnt/ebs/prefetch.txt \
    --schedule /mnt/ebs/order.txt \
    --goblin-small-memory 1120M \
    --goblin-large-memory 4100M \
    --goblin-head-size 16K \
    --goblin-read-chunk 256K \
    --goblin-read-depth 4 \
    --goblin-file-handles 262144 \
    --goblin-population-workers 4 \
    --rocksdb-buffer 5220M

--source aliases --data, and --keys aliases --schedule. A leading / in either key-list row is removed. Every listed key must name a regular file under --data, every scheduled key must occur in --prefetch, and repeated schedule keys retain their exact order. Duplicate prefetch keys are rejected so population counts and timing remain unambiguous.

--goblin-head-size controls the resident head used for this run. --goblin-read-chunk controls the embedded reader’s aligned disk-read quantum and defaults to 256 KiB. --goblin-read-depth controls the number of queued, independently buffered blocks and defaults to four. Callbacks remain serial and ordered regardless of I/O completion order. --goblin-file-handles controls Goblin’s shared immutable-generation descriptor cache. Before the Goblin case, the harness raises its soft RLIMIT_NOFILE to the cache capacity plus 4,096 descriptors and verifies that every disk-backed prefetched object can retain a descriptor. --goblin-population-workers is a bounded source/store pipeline. Each worker owns one 1 MiB source copy buffer and at most one Goblin write staging buffer; the default four workers overlap source reads, direct backing writes, and head-file publication without materializing the corpus.

What one invocation does

The harness inventories all regular source files and gives both engines the exact population and request orders supplied by the input files. It produces two datasets:

1. Goblin Store using its normal O_DIRECT backing-file path.
2. Plain RocksDB using direct reads and direct I/O for flushes and compactions.

Each engine begins with a fresh database. The harness loads every object named by --prefetch itself, and population time is measured rather than inferred. Goblin uses four bounded streaming writers by default with persistent heads enabled. Workers claim input positions monotonically, although differently sized objects may finish out of order. Its digest-named head files live under scratch/goblin/heads, while tail files live under the distinct scratch/goblin/ssd directory; combining those directories is invalid because both formats use the same digest filenames. Population therefore measures the head-catalog persistence work as well as tail writes. The harness does not restart either engine between population and reads.

RocksDB uses DB::Put with the WAL disabled because the database is a disposable cache. It receives the prefetch workload through 512 MiB memtables, up to eight write buffers, and two background jobs. A WriteBufferManager enforces --rocksdb-buffer across memtables and stalls writers when necessary. Normal leveled compaction runs during ingest. After the final Put, the harness performs one blocking flush and waits for background work and obsolete-file purging before reads begin. There is no per-object flush and no full-range rewrite that would require a second copy of the database on the 110 GiB benchmark SSD.

RocksDB is configured as follows:

rocksdb::BlockBasedTableOptions table_options;
table_options.no_block_cache = true;

rocksdb::Options options;
options.use_direct_reads = true;
options.use_direct_io_for_flush_and_compaction = true;

The harness disables compression and mmap reads and does not enable blob files. It permits up to 4,096 open files so the finished SST set does not churn a tiny descriptor table. summary.csv records ingest, terminal flush, and final background-settlement time separately; their sum is the reported population time.

After population, one reusable Goblin reader or one RocksDB handle retrieves schedule keys strictly one at a time. Goblin invokes a callback first with the pinned RAM head and then with each disk-read block in object order. Four independently buffered reads are queued by default and may complete out of order, but callbacks run serially on the benchmark thread. Goblin never allocates or populates a whole-object destination buffer.

RocksDB’s DB::Get is synchronous and materializes the complete value before returning. Its observable first-byte time is therefore the same as its complete-value time. That distinction is intentional: Goblin can expose byte zero while RocksDB is still waiting for the final byte.

For both engines, the request clock stops before result bookkeeping begins. The completed row is then appended to reads.csv and the userspace stream is flushed before the next request starts. Only the current request’s chunk list is retained in RAM. read_phase_ns is the sum of timed storage calls, so buffered CSV writes are excluded from latency and aggregate storage throughput. The harness does not fsync() every row because doing so would turn the workload into a result-metadata benchmark.

Population checkpoints follow the same incremental rule. Each completed ingest, final-flush, and background-settlement phase is appended to phases.csv immediately. A later read failure therefore cannot erase the completed population measurement.

RocksDB’s point-write and point-read APIs require a complete value string, so the harness materializes one source object during population and one result during retrieval. That transient memory is separate from --rocksdb-buffer and must be included when sizing a host for very large individual values.

By default the harness applies POSIX_FADV_DONTNEED to each selected source file before every population phase and again after consuming it. This prevents either engine from inheriting an accidentally warm source corpus. Use --keep-source-cache to disable this behavior. Both database read paths use direct I/O; there is no buffered RocksDB case.

Artifacts

The default output is a timestamped directory in the current working directory. Override it with --output DIR. It contains:

• reads.csv: engine, I/O mode, sequence, key, expected and returned bytes, time to first byte, time to completion, ordered chunk_latencies, and status. chunk_latencies is one CSV field: size_in_bytes:latency_in_ns;size_in_bytes:latency_in_ns;.... RocksDB has one entry per request because DB::Get exposes only the completed value.
• phases.csv: incrementally flushed population ingest, final-flush, settlement, and completion checkpoints with phase and cumulative nanoseconds.
• summary.csv: total population time plus ingest, final-flush, and background-settlement sub-phases; object and byte counts; read-phase time; requests; and bytes.
• metadata.txt: resolved budgets, exact RocksDB I/O/cache/flush settings, paths, clock, library version, and source/schedule dimensions.

Disposable databases live only in fixed goblin and rocksdb children of --scratch. Goblin’s child contains separate heads and ssd pools. All selected database children are removed before the first case starts so an interrupted old run cannot consume capacity during the next one, and they are removed again after their case. --keep-databases retains the newly populated databases for inspection.

--case goblin and --case rocksdb run one fresh case in isolation. The default --case all runs both.

The launcher accepts the same selection as an environment variable, for example CASE=goblin bench/run_goblin_rocksdb_benchmark.sh. Its other environment variables and defaults are listed at the top of the script.

Replaying chunks through an idealized network

simulate_chunk_network.py turns the ordered chunk_latencies field into last-byte delivery times for a serial, idealized link:

python3 bench/simulate_chunk_network.py results/reads.csv \
    --engine goblin \
    --line-rate-gbps 0.937 \
    --output results/simulated-network-0.937gbps.csv

--engine NAME is repeatable and defaults to all successful engines in the input. --line-rate-gbps RATE is a positive decimal-Gbit/s line rate and defaults to 0.937, the sustained network baseline of the m8id.large used for the published comparison. --output FILE writes CSV instead of standard output. The simulator validates that callbacks are ordered and that their byte counts equal returned_bytes; it streams the input without retaining the request set in memory.

The measured report is Goblin Store versus RocksDB on an 8 GiB object-cache host.