Storage

Columnar file I/O, splayed tables, date-partitioned storage, symbol table persistence, and CSV import/export — everything for getting data in and out of Rayforce.

Columnar Files

Rayforce's native binary format stores a single vector per file. Each file contains the 32-byte ray_t header followed by raw element data. Null state is stored in the payload as type-correct sentinels; there is no separate null bitmap region. The format is designed for direct memory mapping — no deserialization needed.

The header carries a single-byte format generation. The current generation is 0, and data written by any prior engine that produced the same layout (also generation 0) loads unchanged — no migration required. A file whose generation the reader does not recognize is refused with a version error rather than mis-read; a future breaking layout change bumps the generation so old files are rejected for migration instead of silently misinterpreted.

File Structure

/*
 * Column file layout:
 *
 * Bytes  0-31:   ray_t header (type, attrs, len, etc.)
 * Bytes 32-N:    element data (len * elem_size bytes)
 *                nulls use type-correct sentinels in this payload
 *
 * String columns (RAY_STR) continue:
 * Bytes 32-N:    16-byte descriptors; a string longer than 12 bytes
 *                lives in the pool and its descriptor carries the
 *                pool offset and a 32-bit content hash
 * Bytes N+1..:   32-byte pool header, then the pool bytes
 */

A string column file written by the current engine stores the content hash of every pooled string in its descriptor and marks the header to say so. Loading or mapping such a file checks descriptor bounds only: no string bytes are read and no descriptor page is written, so the mapping stays shared with the page cache. Files written before the hash was stored carry padding in that field; the loaders detect the missing mark, recompute the hashes and repair them in the private mapping, without touching the file.

C API

Function	Description
ray_col_save(vec, path)	Write a vector to a column file. Handles slices, sentinel nulls, and string pools transparently.
ray_col_load(path)	Read a column file into a heap-allocated vector. The file is read entirely into memory.
ray_col_mmap(path)	Memory-map a column file. A generation-0 file is zero-copy: the returned vector points directly into the mapped file pages. Ideal for large datasets that exceed available RAM. A compressed (generation-2) file is mapped too, but the column decodes into heap memory the first time it is read.

// Save a vector
ray_t* prices = ray_vec_from_raw(RAY_F64, data, 1000000);
ray_err_t err = ray_col_save(prices, "db/trades/price");

// Load into memory
ray_t* loaded = ray_col_load("db/trades/price");

// Memory-map for zero-copy access
ray_t* mapped = ray_col_mmap("db/trades/price");
// mapped->data points into file pages — no allocation

Memory-mapped vectors

Memory-mapped generation-0 vectors have mmod = 1 in their header, distinguishing them from heap-allocated vectors; a decoded compressed column is heap memory. The buddy allocator skips them during free. Slices of mmap'd vectors retain a reference to the parent mapping.

Splayed Tables

A splayed table stores each column as a separate file in a directory. This is the standard on-disk representation for a Rayforce table. The schema (column names) is stored in a .d file alongside the data files.

Directory Layout

db/trades/
  .d                 — I64 vector of column name symbol IDs
  .sym               — symbol table (domain for SYM columns; dotfile)
  sym                — SYM column (stock tickers)
  price              — F64 column (trade prices)
  qty                — I64 column (quantities)
  time               — TIMESTAMP column

The symbol table is a dotfile (.sym, lock .sym.lk), so it never collides with a user column — a table may have an ordinary column named sym (the canonical ticker column), as shown above.

Atomic table replacement

When an existing splayed table is replaced, Rayforce writes the new schema and all columns into a fresh directory below .generations/. After the files are complete, it atomically replaces the small .current manifest. Readers follow that manifest, so they see either the previous complete generation or the new complete generation — never a mixture of columns from both. A failed write before publication leaves the previous generation selected. A cancelled write reports cancel and changes nothing: the previous generation stays selected, and a cancelled first write leaves no table behind, while files that were already in the directory before the write are kept. Superseded generations, and what a failed first write created, are removed afterwards; a symbolic link found among them is unlinked, never followed, so nothing outside the table directory is touched. Tables written by older Rayforce versions remain readable; the first replacement upgrades the directory to the generation layout.

The table, partition and CSV entry points follow the same publication protocol. Indexes are built in the new generation before it is published. The CSV and Parquet converters build each column's chunk zone or string dictionary from the rows as they are streamed and write it with the column, so a conversion does not read its own output back. The converters add no hash index of their own: a hash index is persisted only when the column carries one built with .idx.hash when the table is saved (a table resaved from an older store keeps a hash it already has), and a parted conversion builds none for its segments. An index that cannot be written (for example, the disk fills) is dropped and the column is published without it. Writers to the same table serialize on .write.lock; readers resolve the manifest once and continue using that generation. The symbol vocabulary stays at its original location and grows append-only.

Publication keeps only the selected generation and removes the others best-effort, so a replaced table holds one copy of itself on disk. A reader that resolved a generation removed before it opened every file resolves the manifest again and reads the selected one; a table already loaded stays readable. A failed unpublished generation is removed immediately where the platform allows it; on Windows an open mapped file can defer that cleanup until a later publish. At the first manifest publish, the legacy root .d is retired so older binaries fail loudly, and the column files it named are removed from the table directory; nothing else there is touched.

The .write.lock writer lock is local-filesystem coordination. Do not rely on it for NFS-backed shared writers.

The first write of a table directory, one without .d or .current, writes in place. When it fails, the writer removes what it produced there and, once the directory holds nothing else, the directory itself and every directory it created above it, so no schema-less partition is left behind. A directory that existed before the write is handled the same way, whether an operator pre-made it or an earlier first write was killed: files that were there before the lock was taken and that the write did not rewrite stay, since a directory without a schema is not always a dead table. A partition directory left empty this way is still a partition; .db.parted.fill gives it the missing tables.

ray_splay_save (including .db.splayed.set) syncs primary data and directory entries before acknowledging publication. Inline indexes are rebuildable accelerators written with a marker-last protocol. The bulk C API and CSV import retain their no-fsync contract: publication is atomic for readers, but is not a power-loss durability guarantee. An I/O error after the manifest rename (during the final directory sync) has an uncertain commit outcome; reopen the table to inspect the selected generation.

This protects publication of one splayed table. Coordinating a consistent snapshot across multiple tables or date partitions still belongs to the caller.

C API

Function	Description
ray_splay_save(tbl, dir, sym_path)	Save a table as a splayed directory. Each column becomes a file named after its column symbol. Pass sym_path to also save the symbol table.
ray_splay_load(dir, sym_path)	Load a splayed table from a directory. Columns are memory-mapped by default. Pass sym_path to load the associated symbol table.

// Save table to disk
ray_err_t err = ray_splay_save(table, "db/trades", "db/.sym");

// Load table (columns are mmap'd)
ray_t* trades = ray_splay_load("db/trades", "db/.sym");

Rayfall builtins

Use .db.splayed.set and .db.splayed.get from Rayfall to save and load splayed tables without writing C code. See the Rayfall Storage Builtins section below.

Date-Partitioned Tables

For large time-series datasets, Rayforce supports date-partitioned storage. Data is split into directories named by date, each containing a splayed table for that day's data.

Directory Layout

db/
  .sym                  — shared symbol table (domain for all partitions)
  2024.01.15/
    trades/
      sym
      price
      qty
      time
  2024.01.16/
    trades/
      sym
      price
      qty
      time
  2024.01.17/
    ...

The shared symbol table lives in the parted root (db/) as the dotfile .sym; each partition is a date-named directory holding one splayed table per table name (db/<date>/<table>/cols). Partitions have no own symfile and encode their SYM columns (e.g. sym) against the shared .sym.

Loading Partitioned Data

The ray_read_parted() function scans all partition directories, memory-maps every column file, and assembles them into a single logical table with parted columns and a virtual MAPCOMMON date column.

// C API: load all partitions
ray_t* trades = ray_read_parted("db", "trades");

// The result is a single table with:
// - A MAPCOMMON 'date' column derived from directory names
// - Parted columns (RAY_PARTED_BASE + base_type) for each data column
// - All segments are memory-mapped — no data copy

Rayfall builtin

Use .db.parted.get from Rayfall to load partitioned tables: (.db.parted.get "db" 'trades). See the Rayfall Storage Builtins section below.

Mounting the same root again on the same thread reuses unchanged partitions while the earlier view is held. Cache entries record the resolved generation directory and the device, inode, size, modification time and change time of the schema and every column file. A changed partition is loaded afresh; a disappeared partition is dropped. Symbol columns must match the root's current domain, so a missing .sym is still an error.

The cache holds at most 1024 partition tables per thread. GC, symbol-domain opens and the next mount discard entries whose columns have no external references; heap abandonment, heap teardown and runtime shutdown clear the thread's entries. To release a dropped view's mappings and copied LIST values immediately, call (.sys.gc). C callers can also use ray_parted_cache_clear() from store/part.h to clear their thread's cache.

Partitions without a column (ragged tables)

A partition may hold fewer columns than the first one, as long as the columns it has come first in the same order with the same names and types; a column it lacks is a NULL segment for that partition. Such a ragged table reads as the flat table with typed nulls in the missing rows, wherever its rows are read: a select over it (projections, where:, asc:/desc:, take:, by: and aggregates), (at table 'col), the standalone take verb, printing and show, and ser, whose bytes equal those of the equivalent flat table. The partition-key column's row counts give the missing rows; BOOL and U8, which have no null, read as false and 0. A parted column met away from its table (through get on a column value rather than a query) still contributes no rows for a partition that lacks it.

update over a partitioned table materializes in memory

update on a partitioned table flattens the whole table into memory first — the parted/MAPCOMMON columns cannot be mutated in place — so the result is an ordinary in-memory table. It is not written back to the store: re-reading the root returns the original values, and the returned table loses its parted / memory-mapped identity.

Partition Pruning

The query optimizer recognizes predicates on the MAPCOMMON column and eliminates entire partitions from the scan plan. This means a query filtering on a single date in a year of data only touches 1/365^th of the files on disk — with zero per-row cost for the pruned partitions.

Symbol Table Persistence

The global symbol intern table maps strings to integer IDs. When saving data to disk, the symbol table must be persisted so that symbol vectors can be correctly interpreted when reloaded.

Append-Only .sym Files

A table's symbol file, which .db.splayed.set and the importers write, is append-only: new symbols are appended to the end of the file without rewriting existing entries, so a symbol's position never changes and updates are incremental. One process writes a database at a time: a writer whose view of the file is out of date fails with RAY_ERR_CORRUPT instead of overwriting another writer's symbols (see Running Several Instances).

The C functions below work on the runtime's global symbol table instead. ray_sym_save writes the whole table to a temporary file and renames it over the target, and ray_sym_load merges a file into the running table.

Function	Description
ray_sym_save(path)	Persist the current global symbol table to a .sym file
ray_sym_load(path)	Load a symbol table from disk, merging with any existing entries
ray_sym_intern(str, len)	Intern a string, returning its integer ID
ray_sym_find(str, len)	Look up a string without interning (returns -1 if absent)
ray_sym_str(id)	Resolve an ID back to its string

// Save symbol table alongside data
ray_sym_save("db/.sym");

// On startup, load symbols before loading data
ray_sym_load("db/.sym");

Concurrency and Integrity

• File locking — flock() on POSIX, LockFileEx() on Windows. Multiple processes can safely read the symbol file; writes acquire an exclusive lock.
• Corruption detection — symbol files carry no checksum. The loader checks the magic, the entry count and every entry's length; a truncated or malformed file makes ray_sym_load() return RAY_ERR_CORRUPT.
• Arena backing — interned strings are allocated from a dedicated arena (ray_arena_t), making bulk allocation fast and ensuring all strings are freed together when the symbol table is destroyed.

CSV Import and Export

Rayforce includes a high-performance CSV loader with parallel parsing, automatic type inference, and null handling. No external libraries are used — the parser operates directly on memory-mapped file contents.

Reading CSV Files

; Basic CSV load — auto-detect types, comma delimiter, header row
ray> (set data (.csv.read "trades.csv"))
sym  price   qty  date
----------------------------
AAPL 150.25  100  2024.01.15
GOOG 140.50  200  2024.01.15
MSFT 380.00   50  2024.01.15

C API

Function	Description
ray_read_csv(path)	Load a CSV file with default options: comma delimiter, first row as header, and automatic type inference. Empty fields are treated as null.
ray_read_csv_opts(path, delim, header, col_types, n_types)	Load with custom options: delimiter character, whether first row is a header, explicit column type array (int8_t*), and number of type entries. Pass NULL, 0 for automatic type inference.
ray_write_csv(table, path)	Write a table to a CSV file with header row and comma delimiter.

// Default options
ray_t* data = ray_read_csv("trades.csv");

// Tab-delimited, no header, auto type inference
ray_t* tsv = ray_read_csv_opts("data.tsv", '\t', false, NULL, 0);

// Write results back
ray_write_csv(result, "output.csv");

Type Inference

The CSV loader samples values in each column and infers types in priority order:

1. BOOL — true/false, 1/0
2. I64 — integer values within 64-bit range
3. F64 — floating-point values
4. DATE — YYYY-MM-DD format (hyphen-separated)
5. TIMESTAMP — date + time with nanosecond precision
6. SYM — short repeated strings (auto-interned as symbols)
7. STR — fallback for everything else

Parallel Parsing

The CSV file is memory-mapped and split into chunks. Multiple threads parse chunks in parallel, with a merge step that reconciles column types and combines partial results. For large files (100 MB+), this delivers near-linear speedup with core count.

Null Handling

Empty fields and fields matching the null string (default: "") are recognized as null values. The loader writes the appropriate type-correct null sentinels into the resulting vectors and marks them with RAY_ATTR_HAS_NULLS.

Symbol Merge

When loading CSV data with symbol columns, the loader interns all unique strings into the global symbol table. If a symbol table was previously loaded from disk, existing IDs are preserved and new symbols are appended.

Cross-Platform File I/O

All file operations go through a portable abstraction layer in src/store/fileio.{h,c} that handles platform differences:

Feature	POSIX	Windows
File locking	flock()	LockFileEx()
Sync to disk	fsync()	FlushFileBuffers()
Atomic rename	rename()	MoveFileEx(MOVEFILE_REPLACE_EXISTING)
Memory mapping	mmap()	CreateFileMapping() + MapViewOfFile()

Atomic writes

When saving column files, Rayforce writes to a temporary file first, calls fsync(), then atomically renames it to the target path. This prevents data corruption if the process is interrupted during a write.

Rayfall Storage Builtins

Rayfall provides three builtins for working with on-disk storage directly from the REPL or scripts.

.db.splayed.set — Save a Splayed Table

Writes a table as column files to a directory. A symbol table file is saved automatically alongside the data.

(set t (table [City Temp Rain]
  (list [London Paris Tokyo]
        [15 22 28]
        [120.5 60.3 200.1])))

; Save to /tmp/weather
(.db.splayed.set "/tmp/weather" t)

An optional third argument specifies a custom symbol table path: (.db.splayed.set "/tmp/weather" t "/tmp/my_sym"). Without it, a sym file is created in the table directory.

A trailing options dict selects the column file format: (.db.splayed.set "/tmp/weather" t {compress: true}), also after the symbol path. true writes generation-2 (compressed) files where eligible, false generation 0; when absent each file inherits (generation 2 when its source is an encoded column or the destination's current file is generation 2). Unknown keys or a non-bool value are a domain error. Linked and nested/list columns are always written uncompressed. Compressed files record their indexes in the file header: those you built, and the dictionary or chunk zone a save builds on its own.

A save also builds indexes of its own for columns of at least 65 536 rows: a dictionary for a string column and a chunk zone (per-chunk min/max) for a numeric or temporal one. {index: false} builds neither, in either generation, for a table that is never queried by value (read by partition, by position or whole); the write is faster and the files are smaller. It governs only what the save builds. An index the column already carries is written as it always is: one you built (.idx.hash, .idx.sort, .idx.bloom), or a chunk zone a loaded column brought from its file, as when a table read from one splayed store is saved to another. A dictionary is only ever built, never carried over from the source, so {index: false} leaves a loaded string column without one. A column without an index reads as it does after a save of a short one. true, the default, builds the automatic indexes; a non-bool value is a domain error. compress, symfile and index combine in one dict: {compress: true index: false}.

Compressed floating-point environment

Generation-2 compressed F32/F64 I/O requires default floating-point controls on every calling/executing thread: round-to-nearest (FE_TONEAREST), masked FP exceptions, and gradual underflow (FTZ and DAZ disabled). Embedders must maintain those controls during calls. Rayforce does not silently change or normalize them; sticky exception status flags need not be clear and arithmetic may set them. Detected unsupported controls produce fpenv (RAY_ERR_FPENV in C), not a corruption error. Restore supported controls before retrying the same mapped column; an environment failure is transient, not permanently cached.

The refusal applies conservatively to compressed float writes and whole-column materialization, including decoded-cache hits and non-ALP encodings. ALP range decoding also checks the executing thread. It does not change the file format or require every non-ALP range read to reject nondefault controls.

Detection is deliberately limited: standard rounding is checked, plus SSE rounding, FTZ/DAZ and exception masks on supported x86 SSE builds. This is not a complete portable FP-environment validator: x87 trap masks are not checked, and on other native architectures only standard rounding is checked. WebAssembly has fixed nearest rounding. Maintaining other required controls remains the embedder's responsibility; passing a check is not proof that every control is supported.

.db.splayed.get — Load a Splayed Table

Loads a splayed table from a directory. Columns are memory-mapped: generation-0 files for zero-copy access, while a compressed (generation-2) column decodes into heap memory the first time a query reads it.

(.db.splayed.get "/tmp/weather")

┌─────────┬──────┬────────────────────┐
│  City   │ Temp │        Rain        │
│   SYM   │ I64  │        F64         │
├─────────┼──────┼────────────────────┤
│ 'London │ 15   │ 120.5              │
│ 'Paris  │ 22   │ 60.3               │
│ 'Tokyo  │ 28   │ 200.1              │
└─────────┴──────┴────────────────────┘

An optional second argument specifies the symbol table path: (.db.splayed.get "/tmp/weather" "/tmp/my_sym").

.db.parted.get — Load a Partitioned Table

Loads a date-partitioned table. The first argument is the database root directory, the second is the table name as a quoted symbol.

; Load the "trades" partitioned table from db/
(set dbroot "/tmp/rayforce-storage-db")
(set symfile (format "%/.sym" dbroot))
(set jan15 (table [sym price qty] (list [AAPL GOOG] [150.5 2800.0] [100 50])))
(set jan16 (table [sym price qty] (list [MSFT] [410.0] [75])))
(.db.splayed.set (format "%/2024.01.15/trades" dbroot) jan15 symfile)
(.db.splayed.set (format "%/2024.01.16/trades" dbroot) jan16 symfile)
(.db.parted.get dbroot 'trades)

This scans all date-named subdirectories under db/ (loading db/<date>/trades/ from each), memory-maps every column, and returns a single table with a virtual date column. Partition pruning applies to subsequent queries.

.db.parted.tables — List a Root's Table Names

Returns a sorted sym vector of the table names available under a partitioned database root — the table subdirectories of the most recent (last, sorted) partition, which reflects the current table set. Each name can be passed straight to .db.parted.get; nothing is loaded by this call.

; Discover which tables live under db/, then load each one
(.db.parted.tables dbroot)                ; => [`trades]
(map (fn [t] (.db.parted.get dbroot t)) (.db.parted.tables dbroot))

.db.parted.fill — Backfill Missing Tables Across Partitions

For every table present in any partition, ensures every partition has it: a partition missing the table gets an empty copy whose schema is taken from the most recent partition that has it. This keeps queries that span partitions from failing on a partition where a table is absent — typically a table added partway through the database's life. Returns a sorted sym vector of the partition names it filled (empty when nothing needed fixing, so a repeat call is a no-op). Requires write permission on the root.

; `news` was only added from 2024.01.10 onward; backfill the earlier days.
(.db.parted.fill "db")                  ; => [`2024.01.01 … `2024.01.09]
; now every partition has every table — the empty copies add no rows.
(count (.db.parted.get "db" 'news))     ; unchanged

Symbol Table Management

Symbol tables are persisted automatically when you use .db.splayed.set. A .sym file is written into the table directory containing all interned symbol strings. When loading with .db.splayed.get or .db.parted.get, the symbol table is loaded first so that symbol columns decode correctly.

At the C API level, ray_sym_save(path) and ray_sym_load(path) handle persistence directly. ray_sym_save writes the whole runtime table to a temporary file and renames it over path; one process writes a given file.

Working Example: Round-Trip

A complete workflow: create a table, save it to disk, load it back, and verify the data.

; 1. Create a table
(set t (table [City Temp Rain]
  (list [London Paris Tokyo]
        [15 22 28]
        [120.5 60.3 200.1])))

; 2. Save as splayed table
(.db.splayed.set "/tmp/weather" t)

; 3. Load it back
(set loaded (.db.splayed.get "/tmp/weather"))

; 4. Query the loaded table
(select {from:loaded where: (> Temp 20)})

┌────────┬──────┬────────────────────┐
│  City  │ Temp │        Rain        │
│  SYM   │ I64  │        F64         │
├────────┼──────┼────────────────────┤
│ 'Paris │ 22   │ 60.3               │
│ 'Tokyo │ 28   │ 200.1              │
└────────┴──────┴────────────────────┘

The loaded table is fully functional — you can run select, update, joins, and aggregations on it just like an in-memory table. Symbol columns resolve correctly because the symbol table was saved and loaded alongside the data.

Duration column compatibility

TIMESPAN uses the concrete type tag 16 and a signed 64-bit nanosecond payload. Serialization, generation-0 and generation-2 splayed columns, and partitioned columns preserve the tag and 0Nn nulls. Tags 14 and 15 remain internal selection and encoded-column tags.

Readers built before TIMESPAN support reject the new tag, including when it appears in a table schema. Upgrade readers before writing duration columns to shared storage. Existing column tags and format generations retain their meanings. Parquet logical time and duration mappings are separate from this column format.