Skip to content

seq: add an arithmetic ramp generator, and correct rep's length_out docs #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bobjansen wants to merge 1 commit into
base: main
Choose a base branch
from
Open

seq: add an arithmetic ramp generator, and correct rep's length_out docs #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 70 additions & 6 deletions SPEC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2801,7 +2801,7 @@ with columns sized to `n` rows, so RNG, `rep`, and sequence generators have a
row count to emit against:

```
Table(4000)[update { x = rand_normal(0.0, 1.0), g = rep(0, length_out=4000) }]
Table(4000)[update { i = seq(), x = rand_normal(0.0, 1.0), g = rep([0, 1, 2]) }]
```

A bare `Table(n)` displays as `rows: n` with no columns; once any column is
Expand Down Expand Up @@ -4024,7 +4024,7 @@ exactly one value per row of the current table.
### 12.8 `rep` — Repeat and Fill

```
rep(x, times=1, each=1, length_out=-1)
rep(x, times=1, each=1, length_out=<row count>)
```

Produces a column by repeating a scalar literal or an existing column. Mirrors
Expand All @@ -4037,13 +4037,28 @@ R's `rep()` semantics within the columnar context.
| `x` | *(required)* | Scalar literal (`Int`, `Float`, `Bool`, `String`) or column reference |
| `times` | `1` | Repeat the whole sequence this many times |
| `each` | `1` | Repeat each individual element this many times before advancing |
| `length_out` | *(row count)* | Final output length; shorter sequences are cycled, longer ones truncated |
| `length_out` | *(row count)* | Output length; **must equal the current table's row count** |

`times`, `each`, and `length_out` must be positive integer literals and are
passed as **named arguments**. When `length_out` is omitted, the output
length equals the number of rows in the current table (the normal case for
`update`/`select`).

**`length_out` cannot change the output length.** A field must produce exactly
one value per row of the frame it is added to, so a `length_out` that differs
from the row count is an error — it neither grows nor truncates the frame:

```
let t = Table { a = [1,2,3,4] };
t[select { r = rep(1, length_out=10) }] // error: rep: generates 10 rows but the frame has 4
t[select { r = rep(1, length_out=2) }] // error: rep: generates 2 rows but the frame has 4
```

The parameter is therefore redundant wherever it is legal, and is retained only
for `rep`'s fidelity to R. Prefer omitting it; `rep(0)` and `rep([1,2,3])`
already fill the frame. The `times`/`each` pattern is still fitted to that
length: a short pattern is cycled and a long one truncated.

**Scalar `x` — constant-fill column:**

```
Expand All @@ -4057,9 +4072,8 @@ df[update { mask = rep(true) }]
df[update { source = rep("live") }]
```

When `x` is a scalar the value of `times` and `each` are redundant (all
repetitions of a scalar produce the same value); only `length_out` affects
the output size.
When `x` is a scalar, `times` and `each` are redundant — every repetition of a
scalar produces the same value, and `length_out` cannot change the output size.

**Column `x` — element-wise repetition:**

Expand All @@ -4082,6 +4096,56 @@ df[update { flag = rep(flag_col, times=50) }]
**Constraint.** `rep` is not an aggregate function; it must not appear inside
aggregate function calls (Section 7.3).

### 12.9 `seq` — Arithmetic Sequence

```
seq(from, by) // both optional: seq(), seq(from), seq(from, by)
```

Produces an arithmetic ramp with one value per row of the current table:
`from`, `from + by`, `from + 2·by`, … Where `rep` **tiles** a pattern, `seq`
**advances**, which is the one shape no `rep` form can produce.

**Parameters:**

| Parameter | Default | Meaning |
|---|---|---|
| `from` | `0` | First value |
| `by` | `1` | Step between consecutive rows; may be negative or zero |

Both are **positional** numeric literals. Unlike `rep`'s `times` / `each`,
`from` and `by` are parameter names for documentation only and cannot be passed
as named arguments — they are **reserved keywords** (§2), so `seq(from=0)` and
`seq(0, by=1)` are parse errors. A named argument that does lex, such as
`seq(0, step=2)`, is refused by `seq` itself. The positional form is also what
type inference requires: the argument types decide whether the column is
`Int64` or `Float64`, and named arguments are not visible to that pass.

There is no `length_out`: the output length is always the frame's row count
(§12.8 — the only length `rep`'s `length_out` accepts either).

```
// Row index 0,1,2,... over a generated frame
Table(1000)[update { i = seq() }]

// Start and step: 10,15,20,25,...
Table(1000)[update { id = seq(10, 5) }]

// An evenly spaced Float axis: 0.0, 0.5, 1.0, ...
Table(1000)[update { t = seq(0.0, 0.5) }]

// Descending
Table(1000)[update { countdown = seq(999, -1) }]
```

**Return type:** `Int64` when `from` and `by` are both integer literals,
`Float64` as soon as either is a float literal. Row *i* is computed as
`from + by·i` rather than accumulated, so the float form does not drift.

**Constraint.** `seq` is a generator, like `rep` and the RNG functions: it must
not appear inside aggregate function calls (Section 7.3), and it is not
row-local, so it is not evaluated under a partial row range.

---

## 13. Stream Runtime
Expand Down
3 changes: 2 additions & 1 deletion docs/functions.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,8 @@ <h3>Null and sequence functions</h3>
<tr><td><code>null_if_nan(x)</code></td><td>Turn <code>NaN</code> cells into null; existing nulls stay null.</td></tr>
<tr><td><code>null_if_not_finite(x)</code></td><td>Turn <code>NaN</code>/<code>&plusmn;Inf</code> cells into null.</td></tr>
<tr><td><code>is_nan(x)</code></td><td><code>Bool</code>: true for a valid <code>NaN</code> cell; null cells stay null.</td></tr>
<tr><td><code>rep(x, times=1, each=1, length_out=-1)</code></td><td>Repeat or cycle a value/column to build a full output column.</td></tr>
<tr><td><code>rep(x, times=1, each=1)</code></td><td>Repeat or cycle a value/column to build a full output column.</td></tr>
<tr><td><code>seq(from, by)</code></td><td>Arithmetic ramp, one value per row; positional, defaulting to <code>0</code> and <code>1</code>; <code>Float64</code> if either argument is.</td></tr>
</tbody>
</table>
</article>
Expand Down
23 changes: 16 additions & 7 deletions docs/language-details.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -968,13 +968,19 @@ <h3>Generate random columns in a single vectorized pass</h3>
<!-- 8. rep() and boolean masks -->
<div class="tour-item">
<div class="tour-text">
<span class="tour-tag">rep &amp; Boolean Masks</span>
<h3>Fill, repeat, and cycle columns with named-argument syntax</h3>
<span class="tour-tag">rep, seq &amp; Boolean Masks</span>
<h3>Fill, repeat, cycle, and count columns</h3>
<p>
<code>rep(x)</code> mirrors R&rsquo;s <code>rep()</code>: it fills
a column by repeating a scalar literal or cycling an existing column.
Named arguments &mdash; <code>times</code>, <code>each</code>, and
<code>length_out</code> &mdash; control the exact repetition pattern.
a column by repeating a scalar literal, or by cycling a column or an
array literal. Named arguments &mdash; <code>times</code> and
<code>each</code> &mdash; control the repetition pattern. The output
is always one value per row of the frame.
</p>
<p>
Where <code>rep</code> tiles a pattern, <code>seq</code> advances:
<code>seq(from, by)</code> produces an arithmetic ramp, which is the
idiomatic way to add a row index or an evenly spaced axis.
</p>
<p>
Passing a <code>Bool</code> literal produces a first-class
Expand All @@ -992,8 +998,11 @@ <h3>Fill, repeat, and cycle columns with named-argument syntax</h3>
<span class="cm">// Repeat each element of a column twice</span>
df[<span class="kw">update</span> { rep2 = <span class="fn-name">rep</span>(price, each=<span class="num">2</span>) }];

<span class="cm">// Named args: times, each, length_out</span>
df[<span class="kw">update</span> { flag = <span class="fn-name">rep</span>(flag_col, times=<span class="num">50</span>) }];</code></pre>
<span class="cm">// Cycle an array literal across all rows</span>
df[<span class="kw">update</span> { g = <span class="fn-name">rep</span>([<span class="num">0</span>, <span class="num">1</span>, <span class="num">2</span>]) }];

<span class="cm">// Row index, and an evenly spaced Float axis</span>
df[<span class="kw">update</span> { i = <span class="fn-name">seq</span>(), t = <span class="fn-name">seq</span>(<span class="num">0.0</span>, <span class="num">0.5</span>) }];</code></pre>
</div>
</div>

Expand Down
1 change: 1 addition & 0 deletions src/ir/expr_predicates.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ constexpr auto kBuiltinFunctionInfo = std::to_array<std::pair<std::string_view,
{"rand_poisson", {.kind = FnKind::Generator}},
{"rand_int", {.kind = FnKind::Generator}},
{"rep", {.kind = FnKind::Generator}},
{"seq", {.kind = FnKind::Generator}},
{"sum", {.kind = FnKind::Aggregate}},
{"mean", {.kind = FnKind::Aggregate}},
{"min", {.kind = FnKind::Aggregate}},
Expand Down
38 changes: 22 additions & 16 deletions src/repl/repl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,22 +392,23 @@ constexpr auto kMoreCompletionBuiltins = std::to_array<std::string_view>({
"rolling_skew", "rolling_std",
"rolling_sum", "round",
"scalar", "seed_rng",
"skew", "sqrt",
"std", "sum",
"summary", "trunc",
"filter", "select",
"update", "by",
"window", "order",
"rename", "distinct",
"head", "tail",
"top", "melt",
"dcast", "join",
"on", "sin",
"cos", "tan",
"asin", "acos",
"atan", "sinh",
"cosh", "tanh",
"log2", "log10",
"seq", "skew",
"sqrt", "std",
"sum", "summary",
"trunc", "filter",
"select", "update",
"by", "window",
"order", "rename",
"distinct", "head",
"tail", "top",
"melt", "dcast",
"join", "on",
"sin", "cos",
"tan", "asin",
"acos", "atan",
"sinh", "cosh",
"tanh", "log2",
"log10",
});

CompletionContext g_completion_context;
Expand Down Expand Up @@ -2314,6 +2315,11 @@ constexpr auto kBuiltinDocs = std::to_array<BuiltinDoc>({
.signature = "scalar(table, column) -> scalar",
.summary = "Extract a scalar value from a one-row table.",
.example = "scalar(summary, \"avg\")"},
{.name = "seq",
.signature = "seq(from, by) -> Series<Int64|Float64>",
.summary = "Arithmetic ramp the length of the frame; positional args default to 0 and 1, "
"and the column is Float64 if either is.",
.example = "Table(1000)[update { i = seq(), t = seq(0.0, 0.5) }]"},
{.name = "columns",
.signature = "columns(table) -> DataFrame",
.summary = "Return a metadata table of column names.",
Expand Down
112 changes: 112 additions & 0 deletions src/runtime/expr.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1537,6 +1537,37 @@ const robin_hood::unordered_map<std::string_view, BuiltinFn>& builtins() {
}},
});

// seq(from=0, by=1): an arithmetic ramp the length of the frame. Both
// parameters are positional because `infer` is handed positional types
// only — Int when both are Int, Float as soon as either one is.
m.emplace(
"seq",
BuiltinFn{
.min_args = 0,
.max_args = 2,
.infer = [](std::string_view, const std::vector<ExprType>& a) -> IT {
for (const auto t : a) {
if (t == ExprType::Double) {
return ExprType::Double;
}
if (t != ExprType::Int) {
return std::unexpected(
std::string("seq: from and by must be Int or Float"));
}
}
return ExprType::Int;
},
.exec = GeneratorExec{.column_eval = [](const ir::CallExpr& call, const Table&,
std::size_t rows, const ColumnEvalCtx&)
-> std::expected<ComputedColumn, std::string> {
auto col = apply_seq_func(call, rows);
if (!col) {
return std::unexpected(col.error());
}
return ComputedColumn{.column = std::move(*col), .validity = std::nullopt};
}},
});

// ── Transforms (N→N, ordered/validity-aware): the output row i depends
// on other rows (rolling/cum/lag/lead/fill_forward/fill_backward) or on
// column validity (fill_null, null_if_*), so they evaluate at column
Expand Down Expand Up @@ -3760,4 +3791,85 @@ auto apply_rep_func(const ir::CallExpr& call, const Table& input, std::size_t ro
return std::unexpected("rep: first argument must be a scalar literal or column reference");
}

// seq(from=0, by=1) — an arithmetic ramp over the frame's rows.
//
// seq() – 0, 1, 2, ...
// seq(10) – 10, 11, 12, ...
// seq(10, 5) – 10, 15, 20, ...
// seq(0.0, 0.5) – 0.0, 0.5, 1.0, ...
//
// `from` and `by` are positional, not named: `infer` receives only positional
// argument types (see infer_expr_type), so a named `from=0.0` would type the
// column Int and then be filled with Doubles. Named args are refused rather
// than ignored, so that mismatch cannot be written.
//
// There is no `length_out`. The output is always the frame's row count, which
// is the only length `rep`'s `length_out` accepts either — evaluate_field's
// generator guard rejects a generated length that differs from the frame.
auto apply_seq_func(const ir::CallExpr& call, std::size_t rows)
-> std::expected<ColumnValue, std::string> {
if (call.args.size() > 2) {
return std::unexpected("seq: expected at most 2 arguments (from, by)");
}
if (!call.named_args.empty()) {
return std::unexpected("seq: unknown named argument '" + call.named_args.front().name +
"'; from and by are positional — seq(from, by)");
}

// Whether either literal was written as a Float decides the column type:
// Int only when both are Int, which is what `infer` concludes from these
// same two literals.
std::int64_t from_i = 0;
std::int64_t by_i = 1;
double from_d = 0.0;
double by_d = 1.0;
bool is_double = false;
std::string err;

const auto read = [&](std::size_t pos, std::int64_t& out_i, double& out_d) {
if (pos >= call.args.size() || !err.empty()) {
return;
}
const auto* lit = std::get_if<ir::Literal>(&call.args[pos]->node);
if (lit == nullptr) {
err = "seq: argument " + std::to_string(pos + 1) + " must be a numeric literal";
return;
}
if (const auto* i = std::get_if<std::int64_t>(&lit->value)) {
out_i = *i;
out_d = static_cast<double>(*i);
return;
}
if (const auto* d = std::get_if<double>(&lit->value)) {
out_d = *d;
out_i = static_cast<std::int64_t>(*d);
is_double = true;
return;
}
err = "seq: argument " + std::to_string(pos + 1) + " must be numeric";
};
read(0, from_i, from_d);
read(1, by_i, by_d);
if (!err.empty()) {
return std::unexpected(err);
}

// Row i is computed from i rather than accumulated: a running `+= by` would
// drift on the Float path.
if (is_double) {
Column<double> col;
col.reserve(rows);
for (std::size_t i = 0; i < rows; ++i) {
col.push_back(from_d + (by_d * static_cast<double>(i)));
}
return col;
}
Column<std::int64_t> col;
col.reserve(rows);
for (std::size_t i = 0; i < rows; ++i) {
col.push_back(from_i + (by_i * static_cast<std::int64_t>(i)));
}
return col;
}

} // namespace ibex::runtime
2 changes: 2 additions & 0 deletions src/runtime/interpreter_internal.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1885,6 +1885,8 @@ enum class FloatCleanMode : std::uint8_t {
-> std::expected<ColumnValue, std::string>;
[[nodiscard]] auto apply_rep_func(const ir::CallExpr& call, const Table& input, std::size_t rows)
-> std::expected<ColumnValue, std::string>;
[[nodiscard]] auto apply_seq_func(const ir::CallExpr& call, std::size_t rows)
-> std::expected<ColumnValue, std::string>;
[[nodiscard]] auto expr_value_to_double(const ExprValue& v) -> std::optional<double>;
[[nodiscard]] auto expr_value_to_string(const ExprValue& v) -> std::string;

Expand Down
Loading
Loading