Embedded C++ storage API

Goblin Store can be used as an in-process C++23 storage engine without starting a listener or speaking memcache/HTTP. Add the repository with add_subdirectory, include <goblin/store.hpp>, and link GoblinStore::storage. The concrete archive is libgoblin-store-storage.a.

The boundary retains the properties that matter to the server data path:

StoreWriter admits a known-size object and accepts bounded sequential pieces, then publishes the new generation atomically on commit().
StoreReader is reusable and defaults to four aligned 256 KiB read-ahead buffers plus a per-reader io_uring when the build and kernel support one. Queue creation failure falls back to one-buffer aligned pread.
StoreReader::stream() never allocates or populates a whole-object buffer. It invokes the caller’s (pointer, length) callback for the resident head and then for each disk-read block.
Tail reads may complete out of order, but callbacks run serially on the calling thread and strictly in object order. A second callback cannot begin until the first returns. Each pointer is borrowed and remains valid only for that callback.
Before delivering a resident head, the reader submits up to four tail blocks. After a callback returns, its slot is immediately reused to keep the queue populated.
Store is shared and thread-safe. A StoreReader is single-threaded by design; create one per application worker so rings and scratch buffers are never contended.

Concurrent disk-backed writers share the configured bounded staging-buffer pool. If every buffer is busy, begin_put() returns Errc::would_block; an embedded scheduler can retry or queue that write without allowing admission memory to grow without bound. Destroying an uncommitted StoreWriter aborts its scratch generation and leaves the previously published value untouched.

Minimal use

#include <goblin/store.hpp>

using namespace goblin;

Store::prepare_directory("/mnt/ssd/my-cache"); // one time; directory must be empty

StoreOptions options;
options.ssd.dirs = {"/mnt/ssd/my-cache"};
options.memory.total_bytes = 4 * GiB;
options.tiers.ram_head = 256 * KiB;
options.read_chunk_bytes = 256 * KiB;
options.file_handle_cache = 8192;

auto opened = Store::open(options); // opens an empty store; wipes only marked pool directories
if (!opened) return 1;
Store store = std::move(*opened);

auto writer = store.begin_put("large-object", object_size);
while (auto piece = next_piece()) {
    if (!writer->write(piece)) return 1;
}
if (!writer->commit()) return 1;

auto made_reader = store.make_reader();
if (!made_reader) return 1;
StoreReader reader = std::move(*made_reader);
auto read = reader.stream("large-object",
                          [](const std::byte* data, std::size_t length) -> Status {
    consume(data, length);
    return {};
});

StoreOptions exposes the same tier, memory, eviction, access-score, read/write I/O-chunk, file-handle-cache, write-buffer, maximum-object, and O_DIRECT choices used by the server. The descriptor-cache capacity defaults to 128 and must be a nonzero power of two. Applications using a larger cache must raise RLIMIT_NOFILE before Store::open(); allow additional descriptors for application sockets and files. small_total_bytes selects distinct packed-small-object and fixed-head pools. An empty HDD directory list creates the common two-layer RAM-head/SSD store; adding HDD directories enables the cold tail.

Pool safety is unchanged. Store::prepare_directory() only marks an empty directory, and Store::open() refuses to clear a directory without that marker. Opening is destructive by default. Setting StoreOptions::heads_path opts into the server’s restart format instead: the marked SSD/HDD pools are not wiped, two-hex-digit-sharded digest-named heads rebuild the index, and incomplete lower-tier components without a head are reclaimed at startup. Legacy flat digest files remain readable. The heads directory must already exist and be separate from all pool directories; prepare it once with Store::prepare_directory() as well.

StoreReader::stream() is the only retrieval path. The resident head is passed directly from its pinned RAM allocation without a copy. Disk-backed bytes arrive through four aligned slots by default, each with the configured read-quantum capacity. The pipeline keeps I/O concurrent but never invokes callbacks concurrently: completion ordering is reduced to object ordering on the calling thread. A slot is reused only after its callback returns. make_reader() accepts the io_uring entry count and in-flight read count when an application needs non-default geometry.

A caller that needs a complete value may assemble the callbacks into its own container, but Goblin Store does not impose that allocation or delay on every embedded caller. The same ordered, multi-buffer lifetime rule drives Goblin Store’s socket egress: network sends retain a lane until their completion before that lane is refilled.

The Goblin Store versus RocksDB benchmark uses this interface on both population and retrieval, making the comparison storage-engine-to-storage-engine rather than memcache-over-loopback-to-in-process.