Syntax & Types
Rayfall is a Lisp-like query language with prefix notation, rich scalar types, columnar vectors, and first-class tables. The parser produces ray_t objects directly with no separate AST.
Atoms
Atoms are scalar values. Rayfall supports a wide range of types, each with a distinct literal syntax.
Integers
64-bit signed integers by default. Suffixed variants available for narrower types.
Floats
64-bit IEEE 754 double-precision floating point.
Booleans
Symbols
Symbols are interned identifiers used for column names, dictionary keys, and categorical data. Prefix with a single quote to create a literal symbol.
A symbol literal runs over letters, digits, _, . and -. A symbol, keyword, name or number cannot be directly followed by a quote, a colon, or another name or number character: ['a:1] is a parse error rather than the two symbols a and 1, and 0Na is not 0 followed by Na. A symbol that must contain a colon is built from a string, (as 'symbol "a:1").
Strings
Double-quoted character sequences. Two internal representations: short strings (up to 12 bytes) stored inline, longer strings in a per-vector pool.
"hello" ; inline short string
"hello world!" ; still inline (12 bytes)
"a longer string" ; pool-allocated
Dates
Date literals use dot-separated year.month.day format. Stored as days since 2000-01-01 (i32).
The month must be 1-12 and the day 1-31; anything outside those
ranges is a parse error (e.g. 2024.13.01 or 2024.01.00). A day past the
end of its month is normalized by the calendar rather than rejected, so
2024.02.30 parses as 2024.03.01.
Times
Time-of-day literals in HH:MM:SS.mmm format. Stored as milliseconds since midnight (i32). Values past 24 hours and negative values are valid; for the range, and what an out-of-range result becomes, see Date/Time types.
A time literal has two hour digits, 00-99, and two digits each for
minutes and seconds. Minutes and seconds past 59 carry into the next unit,
the way a day past the end of its month does for dates, so 01:60:00.000
parses as 02:00:00.000. A leading - makes the time negative
(-01:00:00.000). The fractional part is optional (09:30:00 is
09:30:00.000) and is a decimal fraction of a second: one to three digits are
padded (09:30:00.5 is 09:30:00.500), and digits past the third are
dropped. A timestamp literal, the string cast (as 'time "...") and the CSV
reader read it the same way. Times beyond 99:59:59.999 come from arithmetic
or from (as 'TIME <ms>): (as 'TIME 2147483647) is 596:31:23.647.
Timestamps
Full date+time with nanosecond precision. Stored as nanoseconds since 2000-01-01 (i64). Literal form uses D as the date/time separator and requires a fractional-seconds suffix:
The date part follows the date rules above. The time part is stricter than a
time literal: hours 00-23 and minutes and seconds 00-59, anything else
being a parse error; no leading -; and the fractional part is required, with
one to nine digits read as a decimal fraction (.5 is half a second; digits
past the ninth are dropped). Because the value is a 64-bit nanosecond count
from the epoch, timestamps only span roughly ±292 years around 2000 (about
1708 to 2292); a year outside that range is a parse error.
GUIDs
128-bit globally unique identifiers.
Null Values
Nulls are sentinel-encoded directly in the column payload — INT64_MIN
for i64, NaN for f64, and the type-correct reserved value for each
other type (there is no separate null bitmap). The HAS_NULLS attribute
is a fast "may contain nulls" hint. Typed null literals produce the
sentinel for their type:
0Nl ; i64 null
0Ni ; i32 null
0Nh ; i16 null
0Nf ; f64 null
0Ne ; f32 null
0Ng ; guid null (the all-zero GUID)
0Nd ; date null
0Nt ; time null
0Np ; timestamp null
(nil? 0Nl) ; true
Symbols have no 0N null literal (there is no 0Ns). Their null is the
empty symbol, written as a lone quote ', and it works wherever a symbol
does: (set s '), (take ' 3) is [' ' '], [a ' b], {k: '}. It prints as
' too, so a printed vector reads back as the same value. The symbol null
is an ordinary symbol, not the null.
nil? is element-wise: given a vector or a list it returns a B8 vector with
one entry per element, so (where (nil? v)) and (sum (as 'I64 (nil? v)))
work without a map. An atom answers for itself, and so does any other
container (a table or a dict is never null). Inside select and update
expressions, (nil? col) lowers to the DAG null-check opcode, which gives the
same per-row answer and can be used directly in where: predicates and
projections.
(nil? [1 0N 3]) ; [false true false]
(nil? [a ' b]) ; [false true false] — the empty symbol is SYM's null
Vectors
Vectors are homogeneous, typed, columnar arrays. Created with square brackets. The type is inferred from the first element.
[1 2 3 4 5] ; i64 vector
[1.5 2.7 3.9] ; f64 vector
[true false true] ; boolean vector
[AAPL GOOG MSFT] ; symbol vector
["hello" "world"] ; string vector
Vector operations are morsel-driven, processing 1024 elements at a time for cache efficiency.
Vector Arithmetic
All arithmetic operators auto-map over vectors (marked FN_ATOMIC):
Lists
Lists are heterogeneous collections of vectors. Created with the list function. Used as the data component of tables.
Nesting depth
The parser caps recursive nesting of container literals and calls at
1024 levels. A pathologically deep input (e.g. thousands of nested
[ or () is rejected with parse: nesting too deep, limit 1024
rather than overflowing the stack. Real queries never approach this.
Tables
Tables are the core data structure in Rayforce. A table is a vector of column names paired with a list of column vectors. All column vectors must have the same length.
; Create a table with explicit column names
(set trades (table
[sym price size]
(list
[AAPL GOOG MSFT]
[150.5 2800.0 300.2]
[100 50 200])))
; Access column names
(key trades) ; [sym price size]
; Access column data
(value trades) ; list of 3 vectors
Dictionaries
Dictionaries map keys to values. Created with the dict function or with {key: value} syntax in query contexts.
Function Calls
Rayfall uses prefix (Polish) notation. Every expression is either an atom or a parenthesized list where the first element is the function:
Function Types
Built-in functions fall into three arity categories:
| Type | Arguments | Examples |
|---|---|---|
| Unary | Exactly 1 | sum, count, not, neg, type |
| Binary | Exactly 2 | +, -, set, take, at |
| Variadic | 1 or more | if, do, fn, select, list |
Function Flags
| Flag | Behavior |
|---|---|
FN_ATOMIC |
Auto-maps element-wise over vectors. (+ [1 2] [3 4]) yields [4 6]. |
FN_AGGR |
Aggregation function. Reduces a vector to a scalar. (sum [1 2 3]) yields 6. |
FN_SPECIAL_FORM |
Arguments are not evaluated before being passed. Used by set, if, fn, select. |
Quoting
The single quote ' prevents evaluation, creating a symbol atom. Useful for column references and dictionary keys:
'price ; symbol, not a variable lookup
(quote (+ 1 2)) ; returns the unevaluated list (+ 1 2)
(quote price) ; a bare name becomes a literal symbol, equal to 'price
quote returns its argument unevaluated. When the argument is a bare name, the result is a literal symbol — (quote name) is equal to the tick form 'name. Such a literal symbol, when it names a from-table column, resolves to that column during query evaluation. This is how the dynamic-query idiom works: a query assembled as data evaluates (quote name) to the literal symbol 'name during construction, and that symbol resolves to the column when the query runs (see IPC). Inside a query, the tick form 'name and a bare name resolve the same way.
An inline (quote name) node written directly inside a query expression is
different: it keeps a literal symbol even when a column has that name (see
Select). In the dynamic-query idiom above, quote
was evaluated while constructing the query as data, before query evaluation.
Comments
Line comments start with a semicolon and extend to the end of the line:
Control Flow
Conditional: if
Evaluates the condition and returns the true or false branch. Supports if/then/else chaining:
(set x 75)
(if (> x 0) "positive" "non-positive")
; Multi-branch
(if (> x 100) "high"
(> x 50) "mid"
"low")
Sequential Execution: do
Evaluates multiple expressions in order, returning the last result:
Variable Binding: set and let
Dotted Namespaces
A symbol whose name contains one or more . is a dotted symbol — it names a path through nested dictionaries rather than a single global binding. set auto-creates the intermediate dicts on write, and read/delete traverse them with copy-on-write semantics so other references to the enclosing dict see their old value.
; Write a nested namespace — intermediate dicts are created automatically
(set math.pi 3.14159)
(set math.e 2.71828)
(set cfg.db.host "localhost")
(set cfg.db.port 5432)
; Read walks the path segment by segment
math.pi ; 3.14159
cfg.db.host ; "localhost"
; Deleting a leaf cascades: cfg.db becomes empty, then cfg.db is itself removed
(del cfg.db.host)
(del cfg.db.port)
(nil? (resolve 'cfg.db)) ; true
When a dotted path lands on a temporal value (DATE / TIME / TIMESTAMP atom or vector), the trailing segment dispatches through the temporal extraction API instead of looking for a dict key. This lets you reach calendar fields uniformly:
(set d 2024.03.15)
d.yyyy ; 2024
d.mm ; 3
d.dd ; 15
d.dow ; 5 (ISO: Mon=1..Sun=7)
; Vector of dates — the whole column is lifted
(set ds [2024.01.01 2024.06.15 2024.12.31])
ds.yyyy ; [2024 2024 2024]
ds.doy ; [1 167 366]
; Inside a table, col.yyyy resolves against the column vector
(set ts_table (table [Ts]
(list [2024.01.01D10:00:00.000 2024.01.01D11:00:00.000])))
(select {from: ts_table by: Ts.date}) ; group by day
(select {from: ts_table by: Ts.hh}) ; group by hour of day
Recognised temporal segments: yyyy, mm, dd, hh, minute, ss, dow, doy, plus the two truncations date (drop time-of-day) and time (drop date). Null input rows propagate to null output rows — the null sentinel bit pattern is not decoded as a bogus calendar value.
Reserved .* Namespace
Symbols whose name starts with . are a reserved system namespace populated at startup by builtin registration. Typing one of these at the REPL returns the namespace dict for introspection:
.sys ; {gc:<.sys.gc> exec:<.sys.exec> info:<.sys.info> mem:<.sys.mem> build:<.sys.build>}
.os ; {getenv:<.os.getenv> setenv:<.os.setenv>}
.ipc ; {open:<.ipc.open> close:<.ipc.close> send:<.ipc.send>}
.csv ; {read:<.csv.read> write:<.csv.write>}
Ordinary .-prefixed names are protected: set, let, lambda parameters,
and del refuse them with error: reserve. The five designated .ipc.on.*
connection-hook slots are an exception: set can
install a hook and del can clear it. Parameters still cannot use reserved
names.
(set .os.foo 1) ; error: reserve
(let .sys.gc 99) ; error: reserve
((fn [.sys.gc] .sys.gc) 7) ; error: reserve (lambda parameter name)
(del .sys.gc) ; error: reserve: cannot delete reserved binding
Error Handling: try / raise
(try
(raise "oops") ; throws an error
(fn [e] "caught")) ; handler receives error → "caught"
; Unhandled raise: the error code, then the raised value
; (raise "custom error") ; error: domain: custom error
Lambdas & the VM
User-defined functions are created with fn. Lambdas compile lazily to bytecode and run in a stack-based computed-goto VM (ray_vm_t) with a 1024-slot program stack and return stack.
; Named function
(set square (fn [x] (* x x)))
(square 5) ; 25
; Multi-expression body
(set clamp (fn [x lo hi]
(if (< x lo) lo
(> x hi) hi
x)))
; Anonymous lambda passed to map
(map (fn [x] (* x 2)) [1 2 3]) ; [2 4 6]
The VM supports trap frames for try/raise error handling, ensuring exceptions unwind cleanly through compiled code.
A lambda made inside another function is a closure over the locals visible where it was made. A fn literal inside a compiled body is compiled once, as a template, when the enclosing function compiles; every closure made from it shares that bytecode and captures only the enclosing locals its body mentions (all of them when the body uses eval or resolve), so making a closure costs no compilation and its recursion depth does not shrink with the bindings in scope. Its captures are locals of its own body: a let of a captured name rebinds it for that call, a capture (or a parameter or let) named like a builtin or a special form is the callee, a parameter shadows a capture of its name, self inside a closure is the closure itself, and a caller's local of the same name never stands in for a capture. A let that has not run leaves its name unbound, in the body and in a closure made after it, so a read can find a visible interpreted caller's local, the global of that name, or raise a name error. Capture is lexical: a closure captures the bindings of the function it is written in, never a caller's locals. A closure body resolves the names it does not capture when it runs, so a global rebound after the closure was made is seen, and an ordinary local in an interpreted caller's frame can also be seen; a lambda made with nothing captured keeps the builtins it names as they were when it compiled. eval and resolve inside a compiled body see its locals and captures; their arguments run as part of the body. A lazy value is materialized when captured. A body nested deeper than 2048 levels (built at run time; parsed text stops at 1024) is not compiled and runs on the interpreter, whose own depth limit of 512 nested evaluations applies. A closure made by interpreted code (at the top level, by eval, or inside a special form the compiler hands to the tree walker) shares a template too, kept per literal and capture set in a per-thread table of 256 entries, so it is born compiled as well; one written inside a query stays on the tree walker and keeps the query's rule for quoted symbols.
Select & Update
The select and update builtins bridge to the Rayforce DAG executor. They accept a dictionary of options:
select
; Basic filter
(select {from: trades where: (> price 100)})
; Project specific columns with expressions
(select {from: trades
sym: sym notional: (* price size)})
; Group by with aggregation
(select {from: trades
by: {sym: sym}
avg_price: (avg price)
total_size: (sum size)})
update
insert / upsert
insert takes the target table and a table of new rows, and returns the
combined table (the two-argument form does not mutate its argument):
; Append new rows — returns a table with the extra rows
(insert trades
(table [sym price size]
(list [TSLA] [250.0] [300])))
For named mutation, (insert 'trades newrows) appends to the innermost
visible local or parameter named trades, falling back to a global only
when no such local exists. It returns the target symbol. An error never
redirects a local write into a same-named global. The same resolution applies
to alter, including dictionary paths rooted in a local:
(set collect (fn [rows]
(let records (table [ts qty] (list (as 'I64 []) (as 'I64 []))))
(let weights (til 3))
(insert 'records rows)
(alter 'weights set 0 99)
records))
A unique ordinary local table can use the same amortized append path as a
unique global table. Unique writable vectors can be amended by alter set
without copying. Actual aliases, captured values, slices, mapped storage,
and shared dictionary paths retain copy-on-write isolation. Other operations
retain their existing allocation behavior; named mutation does not promise
that every operation or representation avoids allocation.
This applies when the call's head is the literal alter or insert, or a
global alias of one, such as (set ins insert), bound when the function is
compiled. A mutator reached through any other head (a parameter, a let, a
dotted head, a computed head, or a name unbound at that time) is an ordinary
callee: a quoted target resolves as a global, as before, and a computed
target publishes nothing, whether the function is compiled or left to the
tree walker.
Under a literal head or an alias, a closure with a computed alter or
insert target retains the enclosing bindings that the target could name,
just as dynamic eval or resolve does. Constructing the target symbol at
runtime does not redirect a captured local write into a same-named global.
Captures remain values: amending one inside the closure does not change the
enclosing function's binding.
A separately defined function does not gain access to its caller's locals.
Query source columns/aliases are not writable named targets, and a dotted
mutation path must traverse dictionaries rather than table columns or methods.
An unbound target is an error. These rules do not change upsert's existing
contract; for a global target it takes an explicit key count:
(upsert 'trades 1 newrows).
C API
Rayforce exposes a single public header: include/rayforce.h. The core abstraction is ray_t — a 32-byte block header. Every object (atom, vector, list, table) is a ray_t with data following at byte 32.
Key Types
| Type | Description |
|---|---|
ray_t |
32-byte universal block header for all objects |
ray_err_t |
Error code return type |
ray_str_t |
16-byte string element (inline or pooled) |
ray_csr_t |
CSR graph edge storage |
ray_rel_t |
Graph relationship (forward + reverse CSR) |
ray_arena_t |
Bump allocator for bulk allocations |
ray_vm_t |
Bytecode VM for compiled lambdas |
Error Handling
ray_t* result = ray_eval_str("(+ 1 2)");
if (RAY_IS_ERR(result)) {
// handle error
}
// RAY_ERR_PTR() to create error pointers
Memory Management
Never use malloc/free. Use the Rayforce allocator:
ray_t* obj = ray_alloc(size); // general allocation
ray_release(obj); // decrement refcount, free if zero
ray_retain(obj); // increment refcount
ray_t* copy = ray_cow(obj); // copy-on-write
DAG & Execution
The execution pipeline builds a lazy DAG, optimizes it, then executes with fused morsel-driven processing:
// 1. Build lazy DAG
ray_t* g = ray_graph_new(df);
ray_t* filtered = ray_filter(g, predicate);
ray_t* projected = ray_project(g, filtered, cols);
// 2. Execute (optimizer runs automatically)
ray_t* result = ray_execute(g, projected);
Optimizer Passes
- Type inference — propagate types through the DAG
- Constant folding — evaluate compile-time-known expressions
- SIP (Sideways Information Passing) — propagate selection bitmaps backward through expand chains
- Factorize — avoid materializing cross-products with factorized vectors
- Predicate pushdown — move filters closer to data sources
- Filter reorder — rank eligible row-local filters by static selectivity and cost without crossing row-domain barriers
- Fusion — merge adjacent operations into single morsel loops
- DCE (Dead Code Elimination) — remove unused DAG nodes
CSR Storage
Rayforce stores graph edges in double-indexed Compressed Sparse Row (CSR) format: one forward index (source to destination) and one reverse index (destination to source). Both indices are built simultaneously.
// Build CSR from edge list
ray_csr_t csr;
ray_csr_build(&csr, src_ids, dst_ids, n_edges);
// Persist to disk
ray_csr_save(&csr, "edges");
// Memory-map for zero-copy access
ray_csr_mmap(&csr, "edges");
Graph Algorithms
Available as DAG opcodes, all integrated into the same morsel-driven pipeline:
| Opcode | Algorithm | Description |
|---|---|---|
OP_EXPAND |
1-Hop Expand | Follow edges one step from source nodes |
OP_VAR_EXPAND |
BFS | Variable-length path expansion (breadth-first) |
OP_SHORTEST_PATH |
Shortest Path | Single-source shortest paths |
OP_ASTAR |
A* | Heuristic-guided shortest path |
OP_K_SHORTEST |
Yen's K-Shortest | K shortest loopless paths |
OP_WCO_JOIN |
LFTJ | Worst-case optimal join (Leapfrog Triejoin) |
OP_BETWEENNESS |
Brandes | Betweenness centrality |
OP_CLOSENESS |
Closeness | Closeness centrality |
OP_CLUSTER_COEFF |
Clustering | Local clustering coefficients |
OP_RANDOM_WALK |
Random Walk | Random walks on graph |
OP_MST |
Kruskal | Minimum spanning tree |
Pipeline & Optimizer
The full execution pipeline:
Rayfall source
| parse (ASCII dispatch table, recursive descent)
v
ray_t objects (no separate AST)
| ray_eval() / bytecode VM
v
Lazy DAG construction
| ray_graph_new() -> ray_scan/ray_add/ray_filter/...
v
Optimizer (8 passes)
| type inference -> constant fold -> SIP -> factorize
| -> predicate pushdown -> filter reorder -> fusion -> DCE
v
Fused morsel-driven executor
| bytecode over register slots, 1024 elements per morsel
v
Result (ray_t)
Memory Model
- Buddy allocator with thread-local arenas for contention-free allocation
- Slab cache for small, frequently-allocated objects
- COW ref counting —
ray_cow()returns a private copy only when the refcount exceeds 1 - Arena (bump) allocator (
ray_arena_t) for bulk short-lived allocations; blocks carryRAY_ATTR_ARENA, making retain/release no-ops - Per-VM heaps — each heap carries a
heap_id(u16); cross-heap frees enqueue to a lock-free LIFO, reclaimed viaray_heap_flush_foreign()
Files & Partitions
- Column files — native binary format for vectors and CSR graphs, supports mmap
- Sym table — global string intern table, arena-backed, append-only persistence with file locking
- CSV loader — mmap-based, parallel parse, automatic type inference, null handling, sym merge
- File I/O — cross-platform locking (flock/LockFileEx), fsync, atomic rename