Core C API Reference
Headers and include paths
Include <rayforce.h> for the bulk of the public API. A few internal helpers documented here live in their own headers — in particular ray_cow and the ray_heap_init / ray_heap_destroy lifecycle calls are declared in "mem/heap.h", not in <rayforce.h>. The Makefile builds the static archive librayforce.a; it does not provide a shared-library target. On Linux, link the archive with -Wl,--gc-sections -lm -lpthread so the linker can discard unused function and data sections.
Building against a source checkout uses two include roots, cc -Iinclude -Isrc .... Building against a staged install (make install-headers) uses the staged pair instead: cc -Iinstall/include -Iinstall/include/rayforce/src ... - the quoted, src/-relative includes of ops.h, parquet.h and heap.h do not resolve from a flat directory. The target fails loudly on a failed copy, so a successful run leaves a complete staged tree. Staging is a convenience, not an SDK contract: it promises the header contents at the commit that staged them, no ABI stability and no API stability, so stage from the same engine revision you built the archive against.
Floating-point environment for compressed columns
Generation-2 compressed F32/F64 I/O requires default floating-point controls on every calling/executing thread: round-to-nearest (FE_TONEAREST), masked FP exceptions, and gradual underflow (FTZ and DAZ disabled). Embedders must maintain those controls during calls. Rayforce does not silently change or normalize them; sticky exception status flags need not be clear and arithmetic may set them. Detected unsupported controls produce fpenv (RAY_ERR_FPENV in C), not a corruption error. Restore supported controls before retrying the same mapped column; an environment failure is transient, not permanently cached.
The refusal applies conservatively to compressed float writes and whole-column materialization, including decoded-cache hits and non-ALP encodings. ALP range decoding also checks the executing thread. It does not change the file format or require every non-ALP range read to reject nondefault controls.
Detection is deliberately limited: standard rounding is checked, plus SSE rounding, FTZ/DAZ and exception masks on supported x86 SSE builds. This is not a complete portable FP-environment validator: x87 trap masks are not checked, and on other native architectures only standard rounding is checked. WebAssembly has fixed nearest rounding. Maintaining other required controls remains the embedder's responsibility; passing a check is not proof that every control is supported.
Use ray_table_col / ray_table_col_idx to obtain an owned RAY_ERROR on a failed materialization (free it with ray_error_free). The older ray_table_get_col / ray_table_get_col_idx getters collapse decode errors to NULL. Successful plain columns returned by these getters are borrowed from the table. Enum-returning save APIs report RAY_ERR_FPENV; reconstructing it with ray_error(ray_err_code_str(err), NULL) supplies the default explanation.
Lifecycle
Every program must initialize the heap allocator and symbol table before using any other API. Destroy them in reverse order when done.
ray_heap_init
Initializes the thread-local buddy allocator, slab cache, and memory subsystem. Must be called once per thread before any allocation. The main thread must call this before any other Rayforce function.
ray_heap_destroy
Tears down the thread-local heap, releasing all buddy blocks and slab caches back to the OS. Call once per thread at shutdown, after releasing all ray_t objects.
ray_sym_init
Initializes the global symbol intern table. Must be called after ray_heap_init() and before interning any symbols, loading CSV files, or creating tables with symbol columns. Returns RAY_OK on success.
ray_sym_destroy
Frees the global symbol intern table and its arena-backed storage. Call at shutdown, before ray_heap_destroy().
Typical lifecycle pattern
#include <rayforce.h>
int main(int argc, char** argv) {
ray_runtime_t* runtime = ray_runtime_create(argc, argv);
if (!runtime) return 1;
/* ... use Rayforce API ... */
ray_runtime_destroy(runtime);
return 0;
}
Memory Management
Rayforce uses its own buddy allocator with slab cache. Never use malloc/free for ray_t objects. All objects use COW (copy-on-write) ref counting for safe sharing.
ray_alloc
Allocates a ray_t block with at least data_size bytes of data space (beyond the 32-byte header). Uses the slab cache for common sizes, buddy allocator for larger blocks. The returned block has rc = 1. Returns NULL on OOM.
ray_free
Returns a ray_t block to the allocator. Supports cross-thread free: blocks freed from a non-owning thread are enqueued to a lock-free LIFO and reclaimed when the owning heap flushes foreign blocks. Do not call directly on shared objects — use ray_release() instead.
ray_retain
Increments the reference count of a ray_t object. Call when storing an additional reference to an object (e.g., adding a column to a table while keeping the original pointer).
ray_release
Decrements the reference count. When rc reaches zero, the object and any owned children are freed. This is the primary way to dispose of ray_t objects. Safe to call on NULL.
ray_shallow_bytes / ray_retained_bytes
How many bytes of native memory a value stands for, so an embedding runtime (an OCaml or Python binding that wraps a ray_t* in a finalized object) can charge its garbage collector for the wrapper. Neither call allocates on the Rayforce heap.
ray_shallow_bytes is what this value's own release gives back: the block the allocator committed for it (header plus capacity; a buddy block is a power of two, a large direct block its exact page-rounded mapping) plus the private storage that no public API hands out on its own and that only this value releases:
- a long string atom's payload and a GUID atom's payload;
- a string vector's pool;
- an attached accelerator index with its tables;
- a file-mapped column's mapping (a splayed string column's region is charged to the column; the pool block inside it reports 0).
Values that are reachable but are values in their own right are not included: list elements, table and dict slots, lambda parts, a slice's parent. A table and its separately wrapped columns are therefore never charged twice; a slice reports only its header block. Non-owning values report 0: the null and OOM singletons, arena-allocated values, borrowed blocks, an index that is a passenger in a column's mapping. A string pool shared by copy-on-write copies of one vector is charged to each copy. Symbol domains, lazy DAGs and opaque native handles (HNSW, graph) are outside the accounting.
ray_retained_bytes is the shallow figure of the value plus that of everything reachable through ownership edges (list elements, table and dict slots, lambda parts, slice parents, parted segments), each distinct block counted once. Sharing inside the walked graph is not double counted, but two calls on two values that share children both report the shared part. Use it for diagnostics; drive GC pressure with the shallow figure.
ray_t* col = ray_vec_new(RAY_I64, 1000000);
/* ... */
caml_alloc_custom_mem(&ops, sizeof(ray_t*), ray_shallow_bytes(col)); /* OCaml: charge the GC */
ray_heap_set_anon_watermark / ray_heap_anon_watermark / ray_heap_anon_peak
The spill threshold the engine's -m SIZE sets, for a program that embeds the library. The heap counts the anonymous memory it maps itself: pools, direct blocks and sys blocks. A new pool, direct block or large sys block that would push the count past the anon watermark is backed by a spill file (under RAY_HEAP_SWAP, else TMPDIR) instead; where no spill file can be made it stays in anonymous RAM rather than fail. It is a threshold the engine manages, not a cap on the process's RAM nor a guarantee against the OOM killer: mapped inputs and the page cache, and what the host allocates itself, are outside the count (see Memory). The default is ray_sys_ram_limit() — physical RAM, or a smaller container/cgroup limit — less a quarter of it, at least 3 GiB and at most half, left to the page cache.
void ray_heap_set_anon_watermark(int64_t bytes); /* 0 restores the default */
int64_t ray_heap_anon_watermark(void); /* the threshold in force */
int64_t ray_heap_anon_peak(void); /* peak of the heap's anonymous mappings */
ray_heap_anon_watermark resolves 0 to the default. ray_heap_anon_peak is the high-water mark, since process start, of the anonymous mappings the heap tracks, counted at their mapped size, spill excluded. It is not the process's RSS, nor all the RAM the embedding process uses. They are what .sys.mem reports as anon-watermark and anon-peak.
On Linux the heap also locks the anonymous memory it maps under the watermark against swap-out, a page at a time as it is faulted in (MLOCK_ONFAULT), and best effort. The smaller of the watermark and half the RAM limit is the budget for new locks: a mapping that would take the locked bytes past it is left unlocked. Setting the watermark lower unlocks nothing at once: pools and cached blocks are unlocked later, as the heap trims them, and live large blocks stay locked until they are freed, so the locked bytes can stay above the new bound for a while. An embedding keeps its host's RLIMIT_MEMLOCK, and what does not fit stays unlocked (see Memory); RAY_HEAP_MLOCK=0 turns locking off.
ray_cow
For a valid non-arena heap value, reuses a sole-owner header (rc == 1); otherwise copies its allocation and returns a header with rc = 1. A successful copy consumes one input reference; other owners retain the original. If allocation fails, the input reference remains owned by the caller. Arena headers retain their existing special lifetime policy.
This is not a recursive deep copy. In particular, copying a slice header retains its backing parent rather than materializing independent writable data. Header uniqueness alone does not authorize mutation through a backing pointer.
COW ownership pattern
/* v is an owned, valid non-arena, non-slice vector. */
ray_t* alias = ray_retain(v);
ray_t* writable = ray_cow(v);
if (!writable || RAY_IS_ERR(writable)) {
if (writable && RAY_IS_ERR(writable)) ray_error_free(writable);
ray_release(v); /* failed COW did not consume this reference */
ray_release(alias);
return;
}
/* COW consumed v's input reference; alias still owns the old value. */
/* Use the appropriate mutator for writable, then release each owner once. */
ray_release(writable);
ray_release(alias);
Atom Constructors
Atoms are scalar values stored in the ray_t header (no trailing data). Each constructor returns a heap-allocated ray_t* with rc = 1 and a negative type field indicating an atom.
ray_bool
Creates a boolean atom. The value is stored in the b8 field of the header.
ray_u8
Creates an unsigned 8-bit integer atom.
ray_i16
Creates a signed 16-bit integer atom.
ray_i32
Creates a signed 32-bit integer atom.
ray_i64
Creates a signed 64-bit integer atom. Also used for symbol IDs, dates, times, and timestamps internally.
ray_f64
Creates a 64-bit floating-point atom.
ray_str
Creates a string atom. Strings of 7 bytes or fewer are stored inline (SSO in the header). Longer strings allocate a child block. The string is copied; the caller retains ownership of s.
ray_sym
Creates a symbol atom from an intern table ID. Use ray_sym_intern() to get the ID first. The symbol references the global intern table; the atom itself stores only the integer ID.
ray_t* ray_sym(int64_t id);
/* Usage: */
int64_t id = ray_sym_intern("hello", 5);
ray_t* s = ray_sym(id);
ray_date
Creates a date atom. The value is an integer representing days since epoch (2000-01-01).
ray_time
Creates a time atom. The value is milliseconds since midnight.
ray_timestamp
Creates a timestamp atom. The value is nanoseconds since epoch.
ray_guid
Creates a 128-bit GUID atom from a 16-byte array.
Vector API
Vectors are the fundamental columnar container. Each vector has a typed element array following the 32-byte header. All vector operations process data in 1024-element morsels.
ray_vec_new
Creates an empty vector with the given element type and initial capacity. The vector length is 0; capacity determines pre-allocated space to avoid reallocation during appends.
ray_t* ray_vec_new(int8_t type, int64_t capacity);
/* Create a 64-bit integer vector with room for 1000 elements */
ray_t* v = ray_vec_new(RAY_I64, 1000);
ray_vec_append
Appends a single element to the end of the vector. The elem pointer must point to a value of the correct type and size. May reallocate if capacity is exceeded, returning a new pointer. The old pointer may be invalidated.
ray_t* ray_vec_append(ray_t* vec, const void* elem);
int64_t val = 42;
ray_t* result = ray_vec_append(v, &val);
/* Check for NULL/error; on success, result replaces the owned input. */
For a supported non-STR slice, append, raw append and single-element insertion construct an independent result from the visible range. The parent, other views and their indexes/attributes remain unchanged; SYM width and domain are preserved. The source pointer may refer into the old backing allocation: the slice keeps that allocation alive until all copies finish.
ray_t* ray_vec_append_raw(ray_t* vec, const void* src, int64_t count);
ray_t* ray_vec_insert_at(ray_t* vec, int64_t idx, const void* elem);
A successful slice operation consumes one input reference. If slice-result allocation fails, the caller still owns the original input: keep its pointer until success, and free any returned owned error with ray_error_free. This slice-specific preservation does not redefine ordinary vector growth-failure ownership. Raw append of zero elements remains an identity/no-op and permits a NULL source; raw callers remain responsible for their existing null-hint obligations.
ray_vec_set
Sets the element at index idx. The index must be in range [0, len). Uses COW internally: if the vector is shared (rc > 1), a copy is made first. Returns the (possibly new) vector pointer.
For a supported non-STR slice, this copies only the visible range into an independent vector, even if the slice header is unique. The parent and other views remain unchanged. Successful mutation consumes one input reference; if detachment allocation fails, the original input reference remains owned by the caller. Keep the original pointer until a successful return.
ray_vec_get
Returns a pointer to the element at index idx. The returned pointer is valid until the vector is reallocated or freed. Returns NULL if idx is out of range.
void* ray_vec_get(ray_t* vec, int64_t idx);
int64_t* p = (int64_t*)ray_vec_get(v, 0);
if (p) printf("%lld\n", *p);
ray_vec_slice
Creates a zero-copy slice referencing the original vector's data from offset to offset + len. The slice retains the parent (incrementing its rc); the caller may release its own parent reference while the slice exists. Mutations require COW. A direct ray_cow copies a slice header, not its backing payload; use the supported slice-aware ray_vec_set, ray_vec_append, ray_vec_append_raw or ray_vec_insert_at operations rather than treating that header copy as independent writable data.
ray_t* ray_vec_slice(ray_t* vec, int64_t offset, int64_t len);
/* Zero-copy view of elements [10..20) */
ray_t* s = ray_vec_slice(v, 10, 10);
ray_vec_concat
Creates a new vector containing all elements of a followed by all elements of b. Both vectors must have the same type. Returns a new vector with rc = 1.
ray_vec_from_raw
Creates a vector by copying count elements from a raw C array. The data is copied; the caller retains ownership of the source array.
ray_t* ray_vec_from_raw(int8_t type, const void* data, int64_t count);
int64_t vals[] = {10, 20, 30};
ray_t* v = ray_vec_from_raw(RAY_I64, vals, 3);
Null bitmap operations
/* Set element at index 5 as null */
ray_vec_set_null(vec, 5, true);
/* Check if element is null */
if (ray_vec_is_null(vec, 5)) { /* ... */ }
/* Checked version returns RAY_ERR_RANGE on out-of-bounds */
ray_err_t err = ray_vec_set_null_checked(vec, idx, true);
String Vector API
RAY_STR vectors have a specialized API because elements are variable-length. These functions handle the SSO/pool layout transparently.
ray_str_vec_append
Appends a string to a RAY_STR vector. Strings of 12 bytes or fewer are stored inline; longer strings are written to the per-vector pool. May reallocate the vector and/or pool. Always reassign the return value.
ray_t* ray_str_vec_append(ray_t* vec, const char* s, size_t len);
ray_t* v = ray_vec_new(RAY_STR, 100);
v = ray_str_vec_append(v, "hello", 5);
v = ray_str_vec_append(v, "a longer string here", 20);
ray_str_vec_get
Returns a pointer to the string data at index idx and writes the length to *out_len. The returned pointer is valid until the vector is modified. Returns NULL if the element is null or index is out of range. The string is not null-terminated.
const char* ray_str_vec_get(ray_t* vec, int64_t idx, size_t* out_len);
size_t len;
const char* s = ray_str_vec_get(v, 0, &len);
if (s) printf("%.*s\n", (int)len, s);
ray_str_vec_set
Replaces the string at index idx. Uses COW if the vector is shared. The old pool data is not reclaimed immediately; use ray_str_vec_compact() to reclaim unused pool space.
ray_str_vec_compact
Rebuilds the string pool, removing unreferenced data and compacting live strings. Use after many ray_str_vec_set() calls that leave dead pool entries.
List API
Lists are heterogeneous containers holding ray_t* pointers. Used for table column lists, function argument lists, and mixed-type collections.
ray_list_new
Creates an empty list with initial capacity for capacity items.
ray_list_append
Appends an item to the list. The item is retained (rc incremented). May reallocate, so always reassign the return value.
ray_list_get
Returns the item at index idx. Does not increment the reference count. Returns NULL if out of range.
ray_list_set
Replaces the item at index idx. The old item is released and the new item is retained.
Table API
Tables are the primary data structure: a list of named columns (vectors) with a shared row count. Column names are symbol IDs from the global intern table.
ray_table_new
Creates an empty table with capacity for ncols columns. Columns are added with ray_table_add_col().
ray_table_add_col
Adds a named column to the table. The name_id is a symbol ID (from ray_sym_intern()). The column vector is retained. Returns the table pointer (may reallocate).
ray_t* ray_table_add_col(ray_t* tbl, int64_t name_id, ray_t* col_vec);
int64_t name = ray_sym_intern("price", 5);
tbl = ray_table_add_col(tbl, name, price_vec);
ray_table_get_col
Looks up a column by symbol ID. Returns the column vector or NULL if not found. Does not increment the reference count.
ray_table_ncols
Returns the number of columns in the table.
ray_table_nrows
Returns the number of rows (length of the first column). Returns 0 if the table has no columns.
Complete table example
ray_heap_init();
ray_sym_init();
/* Create a 3-column table */
ray_t* ids = ray_vec_from_raw(RAY_I64, (int64_t[]){1, 2, 3}, 3);
ray_t* prices = ray_vec_from_raw(RAY_F64, (double[]){9.99, 19.99, 29.99}, 3);
ray_t* names = ray_vec_new(RAY_STR, 3);
names = ray_str_vec_append(names, "Widget", 6);
names = ray_str_vec_append(names, "Gadget", 6);
names = ray_str_vec_append(names, "Doohickey", 9);
ray_t* tbl = ray_table_new(3);
tbl = ray_table_add_col(tbl, ray_sym_intern("id", 2), ids);
tbl = ray_table_add_col(tbl, ray_sym_intern("price", 5), prices);
tbl = ray_table_add_col(tbl, ray_sym_intern("name", 4), names);
printf("cols: %lld, rows: %lld\n",
ray_table_ncols(tbl), ray_table_nrows(tbl));
/* Output: cols: 3, rows: 3 */
ray_release(ids);
ray_release(prices);
ray_release(names);
ray_release(tbl);
ray_sym_destroy();
ray_heap_destroy();
Error Handling
Rayforce uses two error mechanisms: ray_err_t enum return codes for non-pointer functions, and error objects (RAY_ERROR) for pointer-returning functions.
ray_error
Creates an error object with an 8-byte packed ASCII code and a formatted message. The error object is a ray_t with type == RAY_ERROR. Use RAY_IS_ERR() to test return values.
ray_t* ray_error(const char* code, const char* fmt, ...);
/* Create a type error */
return ray_error("type", "expected I64, got F64");
RAY_IS_ERR(p)
Macro. Tests whether a ray_t* is an error object. Safe to call on NULL. Returns true if p is non-null and has type == RAY_ERROR.
ray_t* result = ray_execute(g, root);
if (RAY_IS_ERR(result)) {
printf("Error: %s\n", ray_err_code(result));
ray_error_free(result);
return 1;
}
Error codes
| Code | Enum | Description |
|---|---|---|
RAY_OK |
0 | Success |
RAY_ERR_OOM |
1 | Out of memory |
RAY_ERR_TYPE |
2 | Type mismatch |
RAY_ERR_RANGE |
3 | Index out of range |
RAY_ERR_LENGTH |
4 | Length mismatch |
RAY_ERR_RANK |
5 | Rank error |
RAY_ERR_DOMAIN |
6 | Domain error (invalid value) |
RAY_ERR_NYI |
7 | Not yet implemented |
RAY_ERR_IO |
8 | I/O error |
RAY_ERR_SCHEMA |
9 | Schema mismatch |
RAY_ERR_CORRUPT |
10 | Data corruption |
RAY_ERR_CANCEL |
11 | Operation cancelled |
RAY_ERR_PARSE |
12 | Parse error |
RAY_ERR_NAME |
13 | Name not found |
RAY_ERR_LIMIT |
14 | Limit exceeded |
RAY_ERR_RESERVED |
15 | Reserved name |
RAY_ERR_VERSION |
16 | Version mismatch |