From 520907173f2c6b4e09a796b3c187440a53dbe8ea Mon Sep 17 00:00:00 2001 From: Bob Jansen Date: Wed, 16 Sep 2026 22:37:43 +0200 Subject: [PATCH] seq: add an arithmetic ramp generator, and correct rep's length_out docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `rep` tiles a pattern but never advances, so no form of it produces 0,1,2,...,N-1. Until now a row index cost two statements and a junk column: transforms require a materialized column name, so `cumsum(rep(1))` inline fails with "cumsum: argument must be a column name" and you had to bind `one = rep(1)` first. Table(1000)[update { i = seq(), t = seq(0.0, 0.5) }] `seq(from, by)` fills that gap as a Generator, with no parser work: a FnKind::Generator row, a GeneratorExec entry, REPL completion and help. Length is always the frame's row count. Row i is computed as from + by*i rather than accumulated, so the float form does not drift. Two things are deliberate and easy to get wrong later: `from` and `by` are positional. `infer` receives only positional argument types, so a named `from=0.0` would type the column Int64 and then fill it with Doubles; the kernel refuses named arguments rather than mistype. The names are also reserved keywords (KeywordFrom/KeywordBy), so `seq(from=0)` is a parse error and the named form cannot be written at all — the tests assert that through parser::parse, since require_ir REQUIREs a lower. `seq` gets no effects.cpp entry despite sitting beside rand_* in every other registry: it is deterministic, and kEffNondet would wrongly block folding and CSE. `rep` is absent there for the same reason. Also corrects SPEC 12.8: `length_out` was documented as if it set the output length, but N is the only value it ever accepts — anything else errors with "generates X rows but the frame has Y", in select and update alike. It is retained only for R fidelity. The docs no longer advertise the -1 sentinel or imply the length is overridable. repl.cpp's completion array is reflowed by clang-format, since inserting one element re-pairs every subsequent line. Co-Authored-By: Claude Opus 5 --- SPEC.md | 76 ++++++++++++++++-- docs/functions.html | 3 +- docs/language-details.html | 23 ++++-- src/ir/expr_predicates.cpp | 1 + src/repl/repl.cpp | 38 +++++---- src/runtime/expr.cpp | 112 +++++++++++++++++++++++++++ src/runtime/interpreter_internal.hpp | 2 + tests/test_interpreter.cpp | 92 ++++++++++++++++++++++ 8 files changed, 317 insertions(+), 30 deletions(-) diff --git a/SPEC.md b/SPEC.md index 90579299..c5af337b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -2801,7 +2801,7 @@ with columns sized to `n` rows, so RNG, `rep`, and sequence generators have a row count to emit against: ``` -Table(4000)[update { x = rand_normal(0.0, 1.0), g = rep(0, length_out=4000) }] +Table(4000)[update { i = seq(), x = rand_normal(0.0, 1.0), g = rep([0, 1, 2]) }] ``` A bare `Table(n)` displays as `rows: n` with no columns; once any column is @@ -4024,7 +4024,7 @@ exactly one value per row of the current table. ### 12.8 `rep` — Repeat and Fill ``` -rep(x, times=1, each=1, length_out=-1) +rep(x, times=1, each=1, length_out=) ``` Produces a column by repeating a scalar literal or an existing column. Mirrors @@ -4037,13 +4037,28 @@ R's `rep()` semantics within the columnar context. | `x` | *(required)* | Scalar literal (`Int`, `Float`, `Bool`, `String`) or column reference | | `times` | `1` | Repeat the whole sequence this many times | | `each` | `1` | Repeat each individual element this many times before advancing | -| `length_out` | *(row count)* | Final output length; shorter sequences are cycled, longer ones truncated | +| `length_out` | *(row count)* | Output length; **must equal the current table's row count** | `times`, `each`, and `length_out` must be positive integer literals and are passed as **named arguments**. When `length_out` is omitted, the output length equals the number of rows in the current table (the normal case for `update`/`select`). +**`length_out` cannot change the output length.** A field must produce exactly +one value per row of the frame it is added to, so a `length_out` that differs +from the row count is an error — it neither grows nor truncates the frame: + +``` +let t = Table { a = [1,2,3,4] }; +t[select { r = rep(1, length_out=10) }] // error: rep: generates 10 rows but the frame has 4 +t[select { r = rep(1, length_out=2) }] // error: rep: generates 2 rows but the frame has 4 +``` + +The parameter is therefore redundant wherever it is legal, and is retained only +for `rep`'s fidelity to R. Prefer omitting it; `rep(0)` and `rep([1,2,3])` +already fill the frame. The `times`/`each` pattern is still fitted to that +length: a short pattern is cycled and a long one truncated. + **Scalar `x` — constant-fill column:** ``` @@ -4057,9 +4072,8 @@ df[update { mask = rep(true) }] df[update { source = rep("live") }] ``` -When `x` is a scalar the value of `times` and `each` are redundant (all -repetitions of a scalar produce the same value); only `length_out` affects -the output size. +When `x` is a scalar, `times` and `each` are redundant — every repetition of a +scalar produces the same value, and `length_out` cannot change the output size. **Column `x` — element-wise repetition:** @@ -4082,6 +4096,56 @@ df[update { flag = rep(flag_col, times=50) }] **Constraint.** `rep` is not an aggregate function; it must not appear inside aggregate function calls (Section 7.3). +### 12.9 `seq` — Arithmetic Sequence + +``` +seq(from, by) // both optional: seq(), seq(from), seq(from, by) +``` + +Produces an arithmetic ramp with one value per row of the current table: +`from`, `from + by`, `from + 2·by`, … Where `rep` **tiles** a pattern, `seq` +**advances**, which is the one shape no `rep` form can produce. + +**Parameters:** + +| Parameter | Default | Meaning | +|---|---|---| +| `from` | `0` | First value | +| `by` | `1` | Step between consecutive rows; may be negative or zero | + +Both are **positional** numeric literals. Unlike `rep`'s `times` / `each`, +`from` and `by` are parameter names for documentation only and cannot be passed +as named arguments — they are **reserved keywords** (§2), so `seq(from=0)` and +`seq(0, by=1)` are parse errors. A named argument that does lex, such as +`seq(0, step=2)`, is refused by `seq` itself. The positional form is also what +type inference requires: the argument types decide whether the column is +`Int64` or `Float64`, and named arguments are not visible to that pass. + +There is no `length_out`: the output length is always the frame's row count +(§12.8 — the only length `rep`'s `length_out` accepts either). + +``` +// Row index 0,1,2,... over a generated frame +Table(1000)[update { i = seq() }] + +// Start and step: 10,15,20,25,... +Table(1000)[update { id = seq(10, 5) }] + +// An evenly spaced Float axis: 0.0, 0.5, 1.0, ... +Table(1000)[update { t = seq(0.0, 0.5) }] + +// Descending +Table(1000)[update { countdown = seq(999, -1) }] +``` + +**Return type:** `Int64` when `from` and `by` are both integer literals, +`Float64` as soon as either is a float literal. Row *i* is computed as +`from + by·i` rather than accumulated, so the float form does not drift. + +**Constraint.** `seq` is a generator, like `rep` and the RNG functions: it must +not appear inside aggregate function calls (Section 7.3), and it is not +row-local, so it is not evaluated under a partial row range. + --- ## 13. Stream Runtime diff --git a/docs/functions.html b/docs/functions.html index 680b4e3f..6ce6b0fc 100644 --- a/docs/functions.html +++ b/docs/functions.html @@ -173,7 +173,8 @@

Null and sequence functions

null_if_nan(x)Turn NaN cells into null; existing nulls stay null. null_if_not_finite(x)Turn NaN/±Inf cells into null. is_nan(x)Bool: true for a valid NaN cell; null cells stay null. - rep(x, times=1, each=1, length_out=-1)Repeat or cycle a value/column to build a full output column. + rep(x, times=1, each=1)Repeat or cycle a value/column to build a full output column. + seq(from, by)Arithmetic ramp, one value per row; positional, defaulting to 0 and 1; Float64 if either argument is. diff --git a/docs/language-details.html b/docs/language-details.html index 6bcb64c0..e7cedb2c 100644 --- a/docs/language-details.html +++ b/docs/language-details.html @@ -968,13 +968,19 @@

Generate random columns in a single vectorized pass

- rep & Boolean Masks -

Fill, repeat, and cycle columns with named-argument syntax

+ rep, seq & Boolean Masks +

Fill, repeat, cycle, and count columns

rep(x) mirrors R’s rep(): it fills - a column by repeating a scalar literal or cycling an existing column. - Named arguments — times, each, and - length_out — control the exact repetition pattern. + a column by repeating a scalar literal, or by cycling a column or an + array literal. Named arguments — times and + each — control the repetition pattern. The output + is always one value per row of the frame. +

+

+ Where rep tiles a pattern, seq advances: + seq(from, by) produces an arithmetic ramp, which is the + idiomatic way to add a row index or an evenly spaced axis.

Passing a Bool literal produces a first-class @@ -992,8 +998,11 @@

Fill, repeat, and cycle columns with named-argument syntax

// Repeat each element of a column twice df[update { rep2 = rep(price, each=2) }]; -// Named args: times, each, length_out -df[update { flag = rep(flag_col, times=50) }]; +// Cycle an array literal across all rows +df[update { g = rep([0, 1, 2]) }]; + +// Row index, and an evenly spaced Float axis +df[update { i = seq(), t = seq(0.0, 0.5) }];
diff --git a/src/ir/expr_predicates.cpp b/src/ir/expr_predicates.cpp index d10743e4..8f07a780 100644 --- a/src/ir/expr_predicates.cpp +++ b/src/ir/expr_predicates.cpp @@ -57,6 +57,7 @@ constexpr auto kBuiltinFunctionInfo = std::to_array({ "rolling_skew", "rolling_std", "rolling_sum", "round", "scalar", "seed_rng", - "skew", "sqrt", - "std", "sum", - "summary", "trunc", - "filter", "select", - "update", "by", - "window", "order", - "rename", "distinct", - "head", "tail", - "top", "melt", - "dcast", "join", - "on", "sin", - "cos", "tan", - "asin", "acos", - "atan", "sinh", - "cosh", "tanh", - "log2", "log10", + "seq", "skew", + "sqrt", "std", + "sum", "summary", + "trunc", "filter", + "select", "update", + "by", "window", + "order", "rename", + "distinct", "head", + "tail", "top", + "melt", "dcast", + "join", "on", + "sin", "cos", + "tan", "asin", + "acos", "atan", + "sinh", "cosh", + "tanh", "log2", + "log10", }); CompletionContext g_completion_context; @@ -2314,6 +2315,11 @@ constexpr auto kBuiltinDocs = std::to_array({ .signature = "scalar(table, column) -> scalar", .summary = "Extract a scalar value from a one-row table.", .example = "scalar(summary, \"avg\")"}, + {.name = "seq", + .signature = "seq(from, by) -> Series", + .summary = "Arithmetic ramp the length of the frame; positional args default to 0 and 1, " + "and the column is Float64 if either is.", + .example = "Table(1000)[update { i = seq(), t = seq(0.0, 0.5) }]"}, {.name = "columns", .signature = "columns(table) -> DataFrame", .summary = "Return a metadata table of column names.", diff --git a/src/runtime/expr.cpp b/src/runtime/expr.cpp index 91fb7a70..5a08f953 100644 --- a/src/runtime/expr.cpp +++ b/src/runtime/expr.cpp @@ -1537,6 +1537,37 @@ const robin_hood::unordered_map& builtins() { }}, }); + // seq(from=0, by=1): an arithmetic ramp the length of the frame. Both + // parameters are positional because `infer` is handed positional types + // only — Int when both are Int, Float as soon as either one is. + m.emplace( + "seq", + BuiltinFn{ + .min_args = 0, + .max_args = 2, + .infer = [](std::string_view, const std::vector& a) -> IT { + for (const auto t : a) { + if (t == ExprType::Double) { + return ExprType::Double; + } + if (t != ExprType::Int) { + return std::unexpected( + std::string("seq: from and by must be Int or Float")); + } + } + return ExprType::Int; + }, + .exec = GeneratorExec{.column_eval = [](const ir::CallExpr& call, const Table&, + std::size_t rows, const ColumnEvalCtx&) + -> std::expected { + auto col = apply_seq_func(call, rows); + if (!col) { + return std::unexpected(col.error()); + } + return ComputedColumn{.column = std::move(*col), .validity = std::nullopt}; + }}, + }); + // ── Transforms (N→N, ordered/validity-aware): the output row i depends // on other rows (rolling/cum/lag/lead/fill_forward/fill_backward) or on // column validity (fill_null, null_if_*), so they evaluate at column @@ -3760,4 +3791,85 @@ auto apply_rep_func(const ir::CallExpr& call, const Table& input, std::size_t ro return std::unexpected("rep: first argument must be a scalar literal or column reference"); } +// seq(from=0, by=1) — an arithmetic ramp over the frame's rows. +// +// seq() – 0, 1, 2, ... +// seq(10) – 10, 11, 12, ... +// seq(10, 5) – 10, 15, 20, ... +// seq(0.0, 0.5) – 0.0, 0.5, 1.0, ... +// +// `from` and `by` are positional, not named: `infer` receives only positional +// argument types (see infer_expr_type), so a named `from=0.0` would type the +// column Int and then be filled with Doubles. Named args are refused rather +// than ignored, so that mismatch cannot be written. +// +// There is no `length_out`. The output is always the frame's row count, which +// is the only length `rep`'s `length_out` accepts either — evaluate_field's +// generator guard rejects a generated length that differs from the frame. +auto apply_seq_func(const ir::CallExpr& call, std::size_t rows) + -> std::expected { + if (call.args.size() > 2) { + return std::unexpected("seq: expected at most 2 arguments (from, by)"); + } + if (!call.named_args.empty()) { + return std::unexpected("seq: unknown named argument '" + call.named_args.front().name + + "'; from and by are positional — seq(from, by)"); + } + + // Whether either literal was written as a Float decides the column type: + // Int only when both are Int, which is what `infer` concludes from these + // same two literals. + std::int64_t from_i = 0; + std::int64_t by_i = 1; + double from_d = 0.0; + double by_d = 1.0; + bool is_double = false; + std::string err; + + const auto read = [&](std::size_t pos, std::int64_t& out_i, double& out_d) { + if (pos >= call.args.size() || !err.empty()) { + return; + } + const auto* lit = std::get_if(&call.args[pos]->node); + if (lit == nullptr) { + err = "seq: argument " + std::to_string(pos + 1) + " must be a numeric literal"; + return; + } + if (const auto* i = std::get_if(&lit->value)) { + out_i = *i; + out_d = static_cast(*i); + return; + } + if (const auto* d = std::get_if(&lit->value)) { + out_d = *d; + out_i = static_cast(*d); + is_double = true; + return; + } + err = "seq: argument " + std::to_string(pos + 1) + " must be numeric"; + }; + read(0, from_i, from_d); + read(1, by_i, by_d); + if (!err.empty()) { + return std::unexpected(err); + } + + // Row i is computed from i rather than accumulated: a running `+= by` would + // drift on the Float path. + if (is_double) { + Column col; + col.reserve(rows); + for (std::size_t i = 0; i < rows; ++i) { + col.push_back(from_d + (by_d * static_cast(i))); + } + return col; + } + Column col; + col.reserve(rows); + for (std::size_t i = 0; i < rows; ++i) { + col.push_back(from_i + (by_i * static_cast(i))); + } + return col; +} + } // namespace ibex::runtime diff --git a/src/runtime/interpreter_internal.hpp b/src/runtime/interpreter_internal.hpp index 37241fd9..b65c3fa7 100644 --- a/src/runtime/interpreter_internal.hpp +++ b/src/runtime/interpreter_internal.hpp @@ -1885,6 +1885,8 @@ enum class FloatCleanMode : std::uint8_t { -> std::expected; [[nodiscard]] auto apply_rep_func(const ir::CallExpr& call, const Table& input, std::size_t rows) -> std::expected; +[[nodiscard]] auto apply_seq_func(const ir::CallExpr& call, std::size_t rows) + -> std::expected; [[nodiscard]] auto expr_value_to_double(const ExprValue& v) -> std::optional; [[nodiscard]] auto expr_value_to_string(const ExprValue& v) -> std::string; diff --git a/tests/test_interpreter.cpp b/tests/test_interpreter.cpp index 97104d5f..8dacd76e 100644 --- a/tests/test_interpreter.cpp +++ b/tests/test_interpreter.cpp @@ -10162,6 +10162,98 @@ TEST_CASE("rep array literal string labels", "[rep]") { } } +TEST_CASE("seq with no arguments ramps from zero", "[seq]") { + runtime::Table table; + table.logical_rows = 5; + runtime::TableRegistry registry; + registry.emplace("t", table); + + auto ir = require_ir("t[update { i = seq() }];"); + auto result = runtime::interpret(*ir, registry); + REQUIRE(result.has_value()); + + const auto& i_col = std::get>(*result->find("i")); + REQUIRE(i_col.size() == 5); + for (std::size_t i = 0; i < 5; ++i) { + CHECK(i_col[i] == static_cast(i)); + } +} + +TEST_CASE("seq honours from and by", "[seq]") { + runtime::Table table; + table.logical_rows = 5; + runtime::TableRegistry registry; + registry.emplace("t", table); + + auto ir = require_ir("t[update { a = seq(10), b = seq(10, 5), c = seq(0, -2) }];"); + auto result = runtime::interpret(*ir, registry); + REQUIRE(result.has_value()); + + const auto& a = std::get>(*result->find("a")); + const auto& b = std::get>(*result->find("b")); + const auto& c = std::get>(*result->find("c")); + const std::int64_t expect_a[] = {10, 11, 12, 13, 14}; + const std::int64_t expect_b[] = {10, 15, 20, 25, 30}; + const std::int64_t expect_c[] = {0, -2, -4, -6, -8}; + for (std::size_t i = 0; i < 5; ++i) { + CHECK(a[i] == expect_a[i]); + CHECK(b[i] == expect_b[i]); + CHECK(c[i] == expect_c[i]); + } +} + +TEST_CASE("seq yields Float64 when either argument is a Float", "[seq]") { + runtime::Table table; + table.logical_rows = 4; + runtime::TableRegistry registry; + registry.emplace("t", table); + + auto ir = require_ir("t[update { t0 = seq(0.0, 0.5), t1 = seq(1, 0.25) }];"); + auto result = runtime::interpret(*ir, registry); + REQUIRE(result.has_value()); + + const auto& t0 = std::get>(*result->find("t0")); + const auto& t1 = std::get>(*result->find("t1")); + const double expect_t0[] = {0.0, 0.5, 1.0, 1.5}; + const double expect_t1[] = {1.0, 1.25, 1.5, 1.75}; + for (std::size_t i = 0; i < 4; ++i) { + CHECK(t0[i] == Catch::Approx(expect_t0[i])); + CHECK(t1[i] == Catch::Approx(expect_t1[i])); + } +} + +TEST_CASE("seq rejects named arguments and non-numeric literals", "[seq]") { + runtime::Table table; + table.logical_rows = 4; + runtime::TableRegistry registry; + registry.emplace("t", table); + + SECTION("the named form cannot be written: from and by are reserved keywords") { + // Not a seq diagnostic at all — the lexer takes `from` and `by` as + // keywords (KeywordFrom / KeywordBy), so these fail in the parser and + // never reach lowering. Asserted through parser::parse rather than + // require_ir, which REQUIREs a successful lower. + CHECK_FALSE(parser::parse("t[update { i = seq(from=0) }];").has_value()); + CHECK_FALSE(parser::parse("t[update { i = seq(0, by=1) }];").has_value()); + } + SECTION("a named argument that does lex is still refused") { + auto ir = require_ir("t[update { i = seq(0, step=2) }];"); + CHECK_FALSE(runtime::interpret(*ir, registry).has_value()); + } + SECTION("length_out is not a seq parameter") { + auto ir = require_ir("t[update { i = seq(0, length_out=4) }];"); + CHECK_FALSE(runtime::interpret(*ir, registry).has_value()); + } + SECTION("too many positional arguments") { + auto ir = require_ir("t[update { i = seq(0, 1, 2) }];"); + CHECK_FALSE(runtime::interpret(*ir, registry).has_value()); + } + SECTION("a string literal is not numeric") { + auto ir = require_ir("t[update { i = seq(\"a\") }];"); + CHECK_FALSE(runtime::interpret(*ir, registry).has_value()); + } +} + TEST_CASE("string interpolation builds a String column in update", "[interp]") { // `row ${id}: ${g}` desugars to __interp(...) and produces a String column // per row. This also exercises the per-row String-column builder.