Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
:sectnums:
:sectnumlevels: 5

= LISTAGG 聚合函数设计

== 目的

通过 Oracle 解析器将 `LISTAGG` 的排序拼接语法转换为已有的 `string_agg` 聚合表达式,并在结果外层增加 4000 字节检查。实现复用普通聚合的排序、分组和 NULL 处理,不单独注册新的 LISTAGG 聚合状态或转换函数。

使用方式见 xref:master/oracle_compatibility/compat_listagg_function.adoc[LISTAGG 聚合函数]。

== 实现说明

=== 代码组织

关键字和语法文件分别位于 `src/include/oracle_parser/` 和 `src/backend/oracle_parser/`。函数注册及实现文件位于 `contrib/ivorysql_ora/src/builtin_functions/`。

回归文件位于 `src/oracle_test/regress/`:测试 SQL 和期望结果分别在 `sql/`、`expected/` 子目录,调度文件在该目录根部。下表只列文件名。

[cols="3,4",options="header"]
|===
|文件 |职责
|`ora_kwlist.h` |将 `LISTAGG` 注册为 Oracle 解析器关键字
|`ora_gram.y` |解析 `LISTAGG`,生成聚合及外层校验函数调用
|`builtin_functions--1.0.sql` |注册 `sys.ora_listagg_check(text)`
|`character_datatype_functions.c` |实现 NULL 处理及聚合结果字节长度检查
|`ora_listagg.sql` |聚合、分隔符、NULL、过滤和异常语法的测试
|`ora_listagg.out` |对应查询结果及错误信息
|`parallel_schedule` |将 `ora_listagg` 加入 Oracle 回归调度
|===

=== 语法转换

Oracle 解析器为 `LISTAGG` 设置专门的 `func_expr` 产生式。解析完成后创建两个嵌套的 `FuncCall` 节点:内层为系统聚合 `string_agg`,外层为扩展校验函数 `sys.ora_listagg_check`。

[source,text]
----
LISTAGG(ename, ',') WITHIN GROUP (ORDER BY sal DESC)
→ sys.ora_listagg_check(
pg_catalog.string_agg(ename, ',' ORDER BY sal DESC)
)
----

转换过程中:

. 检查 `within_group_clause` 是否存在;缺少时抛出语法错误 `LISTAGG requires WITHIN GROUP (ORDER BY ...)`。
. 单参数调用补入空字符串作为分隔符,使内层聚合具有两个参数。
. 使用 `SystemFuncName("string_agg")` 创建内层函数调用,把 `WITHIN GROUP` 排序列表放入 `agg_order`,把 `FILTER` 条件放入 `agg_filter`。
. 使用 `OracleSystemFuncName("ora_listagg_check")` 创建外层调用,并将聚合表达式作为它的唯一参数。

排序列表由普通聚合执行路径处理,而不是将 `WITHIN GROUP` 标记保留为有序集聚合。分组仍由查询的 `GROUP BY` 决定。排序表达式可以与拼接表达式不同,多列及升降序规则沿用底层聚合机制。

`LISTAGG` 的参数部分采用 `func_arg_list`,因此组内排序必须写在 `WITHIN GROUP` 中;参数列表内部的 `ORDER BY` 不属于该产生式。

=== 聚合状态与结果类型

内层聚合复用 `string_agg` 的状态管理及结果构造。非 NULL 值按聚合排序顺序加入缓冲区,NULL 值跳过;分隔符只出现在参与聚合的值之间,NULL 分隔符不追加内容。没有有效输入值时,聚合返回 `NULL`。

长度校验在聚合结果生成后执行。因此 4000 字节限制不是对每个输入值单独校验,也不会在拼接过程中达到上限时提前停止。分组查询对每个分组的最终结果分别校验。

外层函数声明返回 `text`,因此 LISTAGG 表达式的结果类型为 `text`。未指定列别名时,结果列名可能显示为外层函数名 `ora_listagg_check`;使用别名可为应用提供稳定的列名。

=== 字节长度校验

`sys.ora_listagg_check(text)` 注册为 `IMMUTABLE`、`PARALLEL SAFE` 和 `CALLED ON NULL INPUT`。C 函数显式处理 NULL,并通过 `PG_GETARG_TEXT_PP(0)` 取得文本,再用 `VARSIZE_ANY_EXHDR()` 计算不含变长头的有效字节长度。

[source,c]
----
if (PG_ARGISNULL(0))
PG_RETURN_NULL();

text *value = PG_GETARG_TEXT_PP(0);
int len = VARSIZE_ANY_EXHDR(value);

if (len <= 4000)
PG_RETURN_TEXT_P(value);

elog(ERROR, "result of aggregation exceeds 4000 bytes");
----

上述代码展示核心判断逻辑。检查结果包括值之间的分隔符,不额外计算不存在的尾部分隔符;多字节字符按编码后的字节数计入。

[cols="3,4",options="header"]
|===
|输入到校验函数的结果 |行为
|`NULL` |返回 `NULL`
|有效字节长度不超过 4000 |原样返回文本
|有效字节长度超过 4000 |报 `result of aggregation exceeds 4000 bytes`
|===

== 测试验证

`ora_listagg` 回归用例覆盖按值排序、按其他列降序排序、分组聚合、默认分隔符、NULL 值、多列排序、空集、`FILTER`、长度检查和参数内排序语法错误。

长度边界用例直接调用校验函数,验证 4000 字节成功、4001 字节报错,期望错误文本为 `result of aggregation exceeds 4000 bytes`。

这些用例位于 Oracle 核心回归目录。在已配置、编译并安装的源码目录中,可运行:

[source,shell]
----
make -C src/oracle_test/regress oracle-installcheck-tests \
TESTS=ora_listagg EXTRA_REGRESS_OPTS='--port=1521'
----

目标实例需安装对应版本的 `ivorysql_ora`。端口应按实际环境调整。
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
:sectnums:
:sectnumlevels: 5

= LISTAGG 聚合函数

== 目的

`LISTAGG` 按指定顺序将多行非 NULL 字符串拼接为一个结果,适用于生成分组名单、标签列表等汇总信息。IvorySQL 在 Oracle 兼容模式下提供 `LISTAGG(...) WITHIN GROUP (ORDER BY ...)` 语法。

== 功能说明

* 不带 `GROUP BY` 时,将符合条件的输入行聚合为一个结果;带 `GROUP BY` 时,每个分组产生一个结果。
* `WITHIN GROUP (ORDER BY ...)` 指定组内拼接顺序,必须提供;可以使用多列排序及升降序。
* 分隔符可省略。省略或为 `NULL` 时,直接连接字符串;指定分隔符时,只在参与拼接的值之间插入。
* NULL 值不参与拼接,也不产生多余分隔符。没有输入行或全部输入值为 NULL 时,返回 `NULL`。
* 支持 `FILTER (WHERE ...)`,只聚合满足过滤条件的行。
* 返回类型为 `text`,结果最长为 4000 字节,恰好 4000 字节允许返回,超过则报错。

== 语法

[source,sql]
----
LISTAGG(measure_expr [, delimiter])
WITHIN GROUP (ORDER BY sort_expr [ASC | DESC] [, ...])
[FILTER (WHERE condition)]
----

[cols="2,4",options="header"]
|===
|参数 |说明
|`measure_expr` |参与拼接的字符表达式。其他数据类型可先显式转换为 `text`
|`delimiter` |可选的分隔字符串,例如 `','` 或 `', '`
|`sort_expr` |组内排序表达式,可以与拼接表达式不同
|`condition` |聚合输入行的过滤条件
|===

组内排序由 `WITHIN GROUP` 决定,查询末尾的 `ORDER BY` 只决定结果行的显示顺序。如果排序键存在并列值且对应字符串不同,应增加能够区分这些行的排序列,以获得确定的拼接顺序。建议为结果指定列别名。

== 使用示例

=== 准备数据

[source,sql]
----
CREATE TABLE listagg_test (
deptno integer,
ename varchar2(20),
sal numeric
);

INSERT INTO listagg_test VALUES
(10, 'CLARK', 2450),
(10, 'KING', 5000),
(10, 'MILLER', 1300),
(20, 'ADAMS', 1100),
(20, 'FORD', 3000),
(20, 'JONES', 2975),
(30, 'ALLEN', 1600),
(30, 'BLAKE', 2850),
(30, 'MARTIN', 1250);
----

=== 整组聚合与不同排序列

[source,sql]
----
SELECT LISTAGG(ename, ',') WITHIN GROUP (ORDER BY ename) AS employees
FROM listagg_test WHERE deptno = 10;
-- CLARK,KING,MILLER

SELECT LISTAGG(ename, ', ') WITHIN GROUP (ORDER BY sal DESC) AS employees
FROM listagg_test WHERE deptno = 10;
-- KING, CLARK, MILLER
----

=== 分组聚合

[source,sql]
----
SELECT deptno,
LISTAGG(ename, '|') WITHIN GROUP (ORDER BY ename) AS employees
FROM listagg_test
GROUP BY deptno
ORDER BY deptno;
----

[source,text]
----
10 CLARK|KING|MILLER
20 ADAMS|FORD|JONES
30 ALLEN|BLAKE|MARTIN
----

=== 省略分隔符、NULL 值和空集

[source,sql]
----
SELECT LISTAGG(ename) WITHIN GROUP (ORDER BY ename) AS employees
FROM listagg_test WHERE deptno = 10;
-- CLARKKINGMILLER

INSERT INTO listagg_test VALUES (10, NULL, 999);

SELECT LISTAGG(ename, ',') WITHIN GROUP (ORDER BY ename) AS employees
FROM listagg_test WHERE deptno = 10;
-- CLARK,KING,MILLER

SELECT LISTAGG(ename, ',') WITHIN GROUP (ORDER BY ename) AS employees
FROM listagg_test WHERE ename IS NULL;
-- NULL

SELECT LISTAGG(ename, ',') WITHIN GROUP (ORDER BY ename) AS employees
FROM listagg_test WHERE deptno = 99;
-- NULL
----

=== 过滤聚合输入

[source,sql]
----
SELECT LISTAGG(ename, ',') WITHIN GROUP (ORDER BY ename)
FILTER (WHERE sal > 1500) AS employees
FROM listagg_test;
-- ALLEN,BLAKE,CLARK,FORD,JONES,KING

DROP TABLE listagg_test;
----

== 长度限制与语法注意事项

4000 字节上限针对最终结果的有效数据,包含值之间的分隔符。多字节字符按数据库编码后的字节长度计算,不能用字符数代替字节数判断是否超限。

[source,sql]
----
SELECT octet_length(
LISTAGG(repeat('x', 1000)) WITHIN GROUP (ORDER BY n)
) AS result_bytes
FROM generate_series(1, 4) AS t(n);
-- 4000

SELECT LISTAGG(repeat('x', 1000), ',')
WITHIN GROUP (ORDER BY n) AS result_text
FROM generate_series(1, 4) AS t(n);
-- ERROR: result of aggregation exceeds 4000 bytes
----

第二个查询包含三个逗号,结果长度为 4003 字节,因此报错。

排序子句应放在 `WITHIN GROUP` 中,不要写入 `LISTAGG` 的参数列表。省略 `WITHIN GROUP` 会报 `LISTAGG requires WITHIN GROUP (ORDER BY ...)`;例如 `LISTAGG(ename, ',' ORDER BY ename)` 这种参数内排序写法会产生语法错误。
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
:sectnums:
:sectnumlevels: 5

= LISTAGG Aggregate Function Design

== Purpose

The Oracle parser translates ordered concatenation with `LISTAGG` into an existing `string_agg` expression, wrapped in a 4000-byte result check. The implementation reuses ordinary aggregate sorting, grouping, and NULL handling without registering separate LISTAGG aggregate state or transition functions.

See xref:master/oracle_compatibility/compat_listagg_function.adoc[LISTAGG Aggregate Function] for usage.

== Implementation

=== Code organization

Keyword and grammar files are in `src/include/oracle_parser/` and `src/backend/oracle_parser/`, respectively. Function registration and implementation files are in `contrib/ivorysql_ora/src/builtin_functions/`.

Regression files are under `src/oracle_test/regress/`: test SQL and expected output are in the `sql/` and `expected/` subdirectories, and the schedule is at the directory root. The table lists filenames only.

[cols="3,4",options="header"]
|===
|File |Responsibility
|`ora_kwlist.h` |Registers `LISTAGG` as an Oracle parser keyword
|`ora_gram.y` |Parses `LISTAGG` and constructs the aggregate and outer check calls
|`builtin_functions--1.0.sql` |Registers `sys.ora_listagg_check(text)`
|`character_datatype_functions.c` |Implements NULL handling and aggregate result byte-length validation
|`ora_listagg.sql` |Tests aggregation, delimiters, NULLs, filtering, and invalid syntax
|`ora_listagg.out` |Expected query results and errors
|`parallel_schedule` |Schedules `ora_listagg` in the Oracle regression suite
|===

=== Syntax transformation

The Oracle parser has a dedicated `func_expr` production for `LISTAGG`. It constructs two nested `FuncCall` nodes: an inner call to the system aggregate `string_agg`, and an outer call to the extension function `sys.ora_listagg_check`.

[source,text]
----
LISTAGG(ename, ',') WITHIN GROUP (ORDER BY sal DESC)
→ sys.ora_listagg_check(
pg_catalog.string_agg(ename, ',' ORDER BY sal DESC)
)
----

The transformation proceeds as follows:

. Check that `within_group_clause` exists. If missing, raise the syntax error `LISTAGG requires WITHIN GROUP (ORDER BY ...)`.
. For a single argument, append an empty-string delimiter so the inner aggregate receives two arguments.
. Create the inner call with `SystemFuncName("string_agg")`, assigning the `WITHIN GROUP` sort list to `agg_order` and the `FILTER` condition to `agg_filter`.
. Create the outer call with `OracleSystemFuncName("ora_listagg_check")`, passing the aggregate expression as its only argument.

Ordinary aggregate execution handles the sort list, rather than preserving `WITHIN GROUP` as an ordered-set aggregate marker. The query's `GROUP BY` still determines groups. Sort expressions can differ from the concatenated expression, and multicolumn sorting and sort directions follow the underlying aggregate mechanism.

The argument production is `func_arg_list`, so ordering belongs in `WITHIN GROUP`. An `ORDER BY` inside the argument list is outside this production.

=== Aggregate state and result type

The inner aggregate reuses `string_agg` state management and result construction. Non-NULL values enter the buffer in aggregate sort order, while NULL values are skipped. Delimiters appear only between participating values; a NULL delimiter adds no content. With no valid input values, the aggregate returns `NULL`.

Length validation runs after the aggregate result is constructed. The 4000-byte limit is not checked separately for each input value, and aggregation does not stop early when the buffer reaches the limit. Grouped queries validate each group's final result separately.

The outer function returns `text`, so the LISTAGG expression also returns `text`. Without an explicit alias, the result column can appear under the outer function name `ora_listagg_check`. An alias provides a stable column name for applications.

=== Byte-length validation

`sys.ora_listagg_check(text)` is registered as `IMMUTABLE`, `PARALLEL SAFE`, and `CALLED ON NULL INPUT`. The C function explicitly handles NULL, retrieves text with `PG_GETARG_TEXT_PP(0)`, and uses `VARSIZE_ANY_EXHDR()` to measure payload bytes without the variable-length header.

[source,c]
----
if (PG_ARGISNULL(0))
PG_RETURN_NULL();

text *value = PG_GETARG_TEXT_PP(0);
int len = VARSIZE_ANY_EXHDR(value);

if (len <= 4000)
PG_RETURN_TEXT_P(value);

elog(ERROR, "result of aggregation exceeds 4000 bytes");
----

This excerpt shows the core decision logic. The length includes delimiters between values, without adding a nonexistent trailing delimiter. Multibyte characters count according to their encoded byte length.

[cols="3,4",options="header"]
|===
|Result passed to the check function |Behavior
|`NULL` |Returns `NULL`
|Payload length of at most 4000 bytes |Returns the text unchanged
|Payload length above 4000 bytes |Raises `result of aggregation exceeds 4000 bytes`
|===

== Verification

The `ora_listagg` regression cases cover sorting by the value, descending sorting by another column, grouped aggregation, the default delimiter, NULL values, multicolumn sorting, empty inputs, `FILTER`, length checks, and invalid argument-list ordering syntax.

Length boundary cases call the check function directly to verify that 4000 bytes succeeds and 4001 bytes raises `result of aggregation exceeds 4000 bytes`.

These cases are in the Oracle core regression directory. From a configured, built, and installed source tree, run:

[source,shell]
----
make -C src/oracle_test/regress oracle-installcheck-tests \
TESTS=ora_listagg EXTRA_REGRESS_OPTS='--port=1521'
----

The target instance needs the matching version of `ivorysql_ora`. Adjust the port for the environment.
Loading
Loading