From 647b1517259ac8dc88e893974c92972b638d7480 Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Wed, 14 Oct 2026 18:37:12 +0000 Subject: [PATCH 01/10] fix(expressions): validate requests in the service's order and with its messages Issue #108 lists 106 requests whose response differs from Amazon DynamoDB. Every expectation in this change was measured against the service (us-east-1, 2026-10-14); the probes and captured responses are in the issue's notes, and every new test file was run against the service before it was run here. Semantic validation of expressions. After the grammar pass, the service walks the parsed expression once, in expression order, checking each node after its operands, then checks overlapping paths, then unused names and values. A new `validate_update_semantics` and `validate_condition_semantics` in the core crate follow that walk; the parsers now do grammar only, keep function names as written, and record section order for the walk. - UpdateExpression: `+` and `-` take numbers, `list_append` takes lists, `ADD` takes a number or a set, `DELETE` takes a set; a placeholder of the wrong type is refused with the service's type vocabulary (`S`, `N`, `L` for operators and functions; `STRING`, `NUMBER`, `Binary`, `BOOL`, `NULL`, `LIST`, `MAP` for ADD and DELETE). Function names are case-sensitive: `IF_NOT_EXISTS(x, :v)` and `foo(x)` are `Invalid function name`, a condition function is `not allowed in an update expression`, `list_append` is refused as an operand of a function or of `+`, both functions take two operands, the first operand of `if_not_exists` is a document path. A repeated section is reported after that section's own actions (`SET a = :v SET b = :q` reports `:q`). Previously `IF_NOT_EXISTS`, `DELETE s :string`, and `SET a = :string + :n` were accepted and written. - ConditionExpression and FilterExpression: the six condition functions are matched as written (`ATTRIBUTE_EXISTS(pk)` was accepted and evaluated); `if_not_exists` and `list_append` are `not allowed in a condition expression`; a boolean function in operand position and `size` in condition position are `not allowed to be used this way`; operand counts are fixed per function; `attribute_exists` and `attribute_not_exists` take a document path; `begins_with` takes a string or binary and `attribute_type` a string naming a type, both checked at request time with the operation's prefix. - Every unused name or value is listed in one set, sorted, so the message is the same on every backend; it named one key chosen by hash order. Request stage on the read APIs. Query and Scan check the FilterExpression, then the KeyConditionExpression (with its `Invalid KeyConditionExpression:` prefix, and ahead of the key schema match, so an undefined `#n` is reported as such), then the ProjectionExpression, then unused names and values, all before the table is looked up. An undefined `:q` in a filter returned 200 on an empty table. Limit and Segment checks precede the expressions on Scan. Single-item order. UpdateItem checks the whole ConditionExpression before the UpdateExpression. PutItem, UpdateItem, and DeleteItem wrap every expression-stage error in `1 validation error detected: ` and omit the `: ConditionExpression is null` suffix; the read, batch, and transaction APIs return the messages bare. The ` expression size: N` suffix follows the operation. Three existing dual-target tests asserted the unprefixed text and fail against the service; they are corrected. Transactions. TransactGetItems and TransactWriteItems validate in phases, each over every item before the next: per item, expressions (UpdateExpression before ConditionExpression), unused names and values, then the item's 400 KB size; then table lookup; then the 4 MB total; then key schema, as a cancellation with ValidationError on the first offending item only and None on every other (it flagged every offender); then duplicates. An undefined reference is the request error the service returns rather than a per-item cancellation reason, and a key schema mismatch leaves other items' conditions unevaluated. TransactWriteItems returns `"ItemCollectionMetrics": {}` when SIZE is requested and no table has a local secondary index, and its ConsumedCapacity entries are in first-appearance order rather than per-process hash order. Key-shaped messages. GetItem, DeleteItem, UpdateItem, BatchGetItem, and BatchWriteItem (Put and Delete) refuse a key with a missing, mistyped, or extra attribute with `The provided key element does not match the schema`; PutItem keeps `Missing the key pk in the item` and `Type mismatch for key`. BatchWriteItem returns `"ItemCollectionMetrics": {}` under SIZE with no LSI. Table ARNs in `TableName` (eight #108 rows) stay unsupported by decision: an ExtendDB table ARN names ExtendDB's account and region, so a table that streams or replicates to Amazon DynamoDB would answer to two ARNs. docs/differences-from-dynamodb.md records this. Remaining #108 gaps after this change, all stated in the PR: the parser's syntax wording (`Syntax error; token: X, near: Y`), the ResourceNotFound message text, the 4 MB message (the service's carries its own payload size), the service's rendering of over-limit list values, and the table ARN rows. Testing: tests/test_transact_validation_order.py (31 cases), tests/test_update_expression_rules.py (37), and tests/test_expression_semantics.py (155) pass against the service and on SQLite, PostgreSQL, and MongoDB; 164 of the 186 cases in the first and third fail on main. The expression, validation, transaction, batch, query, and scan suites pass on all three backends (767 on SQLite and PostgreSQL, 751 on MongoDB with the four #387 cases unchanged). Of the 106 #108 rows, 83 now return the service's status, error type, and message (30 on main). cargo test --workspace 1,500 passed; fmt; clippy -D warnings on 1.97.0; cargo +1.88.0 check. Closes part of #108. --- crates/core/src/expression/key_condition.rs | 11 +- crates/core/src/expression/mod.rs | 9 +- crates/core/src/expression/parser.rs | 54 +- crates/core/src/expression/projection.rs | 15 +- crates/core/src/expression/resolver.rs | 128 ++- crates/core/src/expression/semantics.rs | 819 ++++++++++++++++++++ crates/core/src/expression/update_parser.rs | 188 +++-- crates/core/src/validation/mod.rs | 62 +- crates/engine/src/batch_write_item.rs | 11 +- crates/engine/src/capacity_helpers.rs | 25 + crates/engine/src/delete_item.rs | 18 +- crates/engine/src/expression_helpers.rs | 208 +++-- crates/engine/src/put_item.rs | 18 +- crates/engine/src/query.rs | 472 +++++------ crates/engine/src/read_helpers.rs | 3 +- crates/engine/src/scan.rs | 161 ++-- crates/engine/src/transact_get_items.rs | 115 +-- crates/engine/src/transact_write_helpers.rs | 42 +- crates/engine/src/transact_write_items.rs | 342 ++++---- crates/engine/src/update_item.rs | 51 +- docs/differences-from-dynamodb.md | 1 + tests/test_expression_error_prefix.py | 6 +- tests/test_expression_param_usage.py | 20 +- tests/test_expression_semantics.py | 681 ++++++++++++++++ tests/test_expression_size_limit.py | 43 +- tests/test_transact_validation_order.py | 664 ++++++++++++++++ tests/test_update_expression_rules.py | 303 ++++++++ 27 files changed, 3650 insertions(+), 820 deletions(-) create mode 100644 crates/core/src/expression/semantics.rs create mode 100644 tests/test_expression_semantics.py create mode 100644 tests/test_transact_validation_order.py create mode 100644 tests/test_update_expression_rules.py diff --git a/crates/core/src/expression/key_condition.rs b/crates/core/src/expression/key_condition.rs index 2bdbd8b7d..cc5bd05a4 100755 --- a/crates/core/src/expression/key_condition.rs +++ b/crates/core/src/expression/key_condition.rs @@ -479,12 +479,19 @@ fn parse_key_clause_inner(tokens: &[Token], pos: &mut usize) -> Result Result { - // size(path) is a function that returns a value, usable as an operand - if name.eq_ignore_ascii_case("size") - && *pos + 1 < tokens.len() - && tokens[*pos + 1] == Token::LParen - { + Token::Ident(_) => { + if *pos + 1 < tokens.len() && tokens[*pos + 1] == Token::LParen { return parse_function_call(tokens, pos); } let elements = parser_common::parse_path(tokens, pos)?; @@ -230,7 +224,7 @@ fn parse_operand(tokens: &[Token], pos: &mut usize) -> Result Result { let name = match &tokens[*pos] { - Token::Ident(n) => n.to_ascii_lowercase(), + Token::Ident(n) => n.clone(), _ => { return Err(validation_err( "Invalid ConditionExpression: expected function name", @@ -241,13 +235,16 @@ fn parse_function_call(tokens: &[Token], pos: &mut usize) -> Result Option { } } -fn is_function_name(name: &str) -> bool { - matches!( - name, - "attribute_exists" | "attribute_not_exists" | "attribute_type" | "begins_with" | "contains" - ) -} - fn validation_err(msg: &str) -> DynamoDbError { DynamoDbError::ValidationException(msg.to_owned()) } diff --git a/crates/core/src/expression/projection.rs b/crates/core/src/expression/projection.rs index fd84f4c5f..ee90d5201 100755 --- a/crates/core/src/expression/projection.rs +++ b/crates/core/src/expression/projection.rs @@ -125,13 +125,13 @@ fn prefix_projection_error(err: DynamoDbError) -> DynamoDbError { /// A path element after `#name` resolution. #[derive(PartialEq, Eq)] -enum ResolvedElement { +pub(crate) enum ResolvedElement { Attr(String), Index(usize), } /// Resolve `#name` references in parsed paths, rejecting index-start paths. -fn resolve_paths( +pub(crate) fn resolve_paths( paths: &[Vec], maps: &ExpressionMaps, ) -> Result>, DynamoDbError> { @@ -166,11 +166,20 @@ fn resolve_paths( /// Reject paths where one is a prefix of another (DynamoDB's overlap rule), /// reporting the first offending pair in expression order. Siblings are fine. fn check_overlap(resolved: &[Vec]) -> Result<(), DynamoDbError> { + check_overlap_for(resolved, super::ExpressionKind::Projection) +} + +/// The overlap rule for any expression kind that names document paths; the +/// message carries the kind's `Invalid :` prefix. +pub(crate) fn check_overlap_for( + resolved: &[Vec], + kind: super::ExpressionKind, +) -> Result<(), DynamoDbError> { for i in 0..resolved.len() { for j in (i + 1)..resolved.len() { if is_prefix(&resolved[i], &resolved[j]) || is_prefix(&resolved[j], &resolved[i]) { return Err(DynamoDbError::ValidationException(format!( - "Invalid ProjectionExpression: Two document paths overlap with each other; \ + "Invalid {kind}: Two document paths overlap with each other; \ must remove or rewrite one of these paths; path one: {}, path two: {}", render_path(&resolved[i]), render_path(&resolved[j]), diff --git a/crates/core/src/expression/resolver.rs b/crates/core/src/expression/resolver.rs index fa14081ec..a29fb09cf 100755 --- a/crates/core/src/expression/resolver.rs +++ b/crates/core/src/expression/resolver.rs @@ -273,33 +273,129 @@ pub fn validate_unused_attributes( collect_action_refs(action, &mut used_names, &mut used_values); } - for key in names.keys() { - if !used_names.contains(key.strip_prefix('#').unwrap_or(key)) { - let display_key = if key.starts_with('#') { + // The service lists every unused key in one set (`keys: {:x, :y, :z}`). + // Its order is a hash artifact no client can depend on; the keys are + // sorted here so the message is the same on every backend and process. + let mut unused_names: Vec = names + .keys() + .filter(|key| !used_names.contains(key.strip_prefix('#').unwrap_or(key))) + .map(|key| { + if key.starts_with('#') { key.clone() } else { format!("#{key}") - }; - return Err(DynamoDbError::ValidationException(format!( - "Value provided in ExpressionAttributeNames unused in expressions: keys: {{{display_key}}}" - ))); - } + } + }) + .collect(); + if !unused_names.is_empty() { + unused_names.sort(); + return Err(DynamoDbError::ValidationException(format!( + "Value provided in ExpressionAttributeNames unused in expressions: keys: {{{}}}", + unused_names.join(", ") + ))); } - for key in values.keys() { - if !used_values.contains(key.strip_prefix(':').unwrap_or(key)) { - let display_key = if key.starts_with(':') { + let mut unused_values: Vec = values + .keys() + .filter(|key| !used_values.contains(key.strip_prefix(':').unwrap_or(key))) + .map(|key| { + if key.starts_with(':') { key.clone() } else { format!(":{key}") - }; - return Err(DynamoDbError::ValidationException(format!( - "Value provided in ExpressionAttributeValues unused in expressions: keys: {{{display_key}}}" - ))); - } + } + }) + .collect(); + if !unused_values.is_empty() { + unused_values.sort(); + return Err(DynamoDbError::ValidationException(format!( + "Value provided in ExpressionAttributeValues unused in expressions: keys: {{{}}}", + unused_values.join(", ") + ))); + } + Ok(()) +} + +/// Validate that every `#name` and `:value` reference in the expressions is +/// defined in the supplied maps. +/// +/// The single-item APIs discover an undefined reference while evaluating the +/// expression against the item. Transactions validate every item's expressions +/// before any table is looked up, so this runs the same check on the parsed +/// expression ahead of time. Messages match the resolver's, without the +/// `Invalid :` prefix, which the caller adds for the expression at hand. +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` naming the first undefined +/// reference, names before values, in the order the expressions are given. +pub fn validate_refs_defined( + names: &HashMap, + values: &HashMap, + exprs: &[&super::ast::Expr], + update_actions: &[&super::ast::UpdateAction], +) -> Result<(), DynamoDbError> { + let mut used_names = std::collections::HashSet::new(); + let mut used_values = std::collections::HashSet::new(); + for expr in exprs { + collect_expr_refs(expr, &mut used_names, &mut used_values); + } + for action in update_actions { + collect_action_refs(action, &mut used_names, &mut used_values); + } + let mut missing_names: Vec<&String> = used_names + .iter() + .filter(|n| !names.contains_key(*n)) + .collect(); + missing_names.sort(); + if let Some(name) = missing_names.first() { + return Err(DynamoDbError::ValidationException(format!( + "An expression attribute name used in the document path is not defined; attribute name: #{name}" + ))); + } + let mut missing_values: Vec<&String> = used_values + .iter() + .filter(|v| !values.contains_key(*v)) + .collect(); + missing_values.sort(); + if let Some(value) = missing_values.first() { + return Err(DynamoDbError::ValidationException(format!( + "An expression attribute value used in expression is not defined; attribute value: :{value}" + ))); } Ok(()) } +/// Reject an update expression whose actions name overlapping document paths. +/// +/// Amazon DynamoDB refuses two actions whose paths are equal or where one is a +/// prefix of the other, across all four sections, after `#name` resolution: +/// `SET a = :v, a = :w`, `SET m = :v, m.x = :w`, `SET a.b = :v REMOVE a`, and +/// `SET #l[1] = :v, #l[1].x = :w` with `#l` bound to `lst` all fail. Siblings +/// (`m.x` and `m.y`) are fine. The first offending pair in expression order is +/// named, rendered as the service renders it: `[lst, [1]]`. +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` with the `Invalid +/// UpdateExpression:` prefix. +pub fn validate_update_paths_disjoint( + actions: &[super::ast::UpdateAction], + maps: &ExpressionMaps, +) -> Result<(), DynamoDbError> { + use super::ast::UpdateAction; + let paths: Vec> = actions + .iter() + .map(|action| match action { + UpdateAction::Set { path, .. } + | UpdateAction::Remove { path } + | UpdateAction::Add { path, .. } + | UpdateAction::Delete { path, .. } => path.clone(), + }) + .collect(); + let resolved = super::projection::resolve_paths(&paths, maps)?; + super::projection::check_overlap_for(&resolved, super::ExpressionKind::Update) +} + fn collect_expr_refs( expr: &super::ast::Expr, names: &mut std::collections::HashSet, diff --git a/crates/core/src/expression/semantics.rs b/crates/core/src/expression/semantics.rs new file mode 100644 index 000000000..c3c8a1e77 --- /dev/null +++ b/crates/core/src/expression/semantics.rs @@ -0,0 +1,819 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Semantic validation of parsed `UpdateExpression` and `ConditionExpression` +//! trees, run after the grammar pass and before any table is read. +//! +//! Amazon DynamoDB reports every syntax error first, then walks the parsed +//! tree once, in expression order, checking each node after its operands +//! (a post-order walk), and only then checks overlapping paths and unused +//! names and values. The order of the messages below was measured against +//! the service on 2026-10-14; each rule names the request that pinned it. +//! +//! Update expression, per node: +//! - a document path: every `#name` element is defined; +//! - a `:value` placeholder: defined; +//! - `+` and `-`: after both operands, a placeholder operand must be a +//! number (`SET a = :s + :n` with a string `:s`); +//! - a function call: after its operands, `list_append` may not be an +//! operand of a function or of `+`/`-` (`if_not_exists(list_append(:a, +//! :b), :v)`); the name must be `if_not_exists` or `list_append` in that +//! case (`IF_NOT_EXISTS(x, :v)` is refused); a condition function is +//! refused by name (`SET x = size(x)`); both take two operands; the first +//! operand of `if_not_exists` is a document path; a placeholder operand of +//! `list_append` is a list; +//! - an `ADD` or `DELETE` action: after the path and the placeholder, the +//! value is a number or a set for `ADD` and a set for `DELETE`; +//! - a section: after its actions, the keyword may not have appeared before +//! (`SET a = :v SET b = :q` reports `:q`, `SET a = :v SET b = :v ADD n :s` +//! reports the repeated `SET`). +//! +//! Condition expression, per node: +//! - operands of a comparison, `BETWEEN`, or `IN` are walked first; a +//! function in operand position must be `size`, and `size` in condition +//! position is refused (`attribute_exists(pk) = :v`, `size(pk)`); +//! - a function call: after its operands, the name must be one of the six +//! condition functions as written (`ATTRIBUTE_EXISTS(pk)` is refused), +//! `if_not_exists` and `list_append` are refused by name, the operand count +//! is fixed per function, `attribute_exists` and `attribute_not_exists` +//! take a document path (`attribute_type(:v, :t)` is accepted), the second operand of +//! `begins_with` is a string or binary, and the second operand of +//! `attribute_type` is a string naming a type; +//! - a comparison or `BETWEEN` with an ordering operator: a placeholder +//! operand is a string, number, or binary. + +use super::ast::{ArithOp, Expr, PathElement, UpdateAction}; +use super::kind::ExpressionKind; +use super::resolver::{ExpressionMaps, attribute_type_code, validate_ordering_operand_types}; +use super::update_parser::{ParsedUpdate, repeated_section_err}; +use crate::error::DynamoDbError; +use crate::types::AttributeValue; + +const CONDITION_FUNCTIONS: [&str; 6] = [ + "attribute_exists", + "attribute_not_exists", + "attribute_type", + "begins_with", + "contains", + "size", +]; + +const UPDATE_FUNCTIONS: [&str; 2] = ["if_not_exists", "list_append"]; + +/// Validate a parsed update expression against the service's semantic rules, +/// in the service's order, then reject overlapping paths. +/// +/// Messages carry the `Invalid UpdateExpression:` prefix. +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` for the first rule the +/// expression breaks in the order described in the module documentation. +pub fn validate_update_semantics( + parsed: &ParsedUpdate, + maps: &ExpressionMaps, +) -> Result<(), DynamoDbError> { + let kind = ExpressionKind::Update; + let mut seen: Vec<&'static str> = Vec::new(); + for section in &parsed.sections { + for action in &parsed.actions[section.actions.clone()] { + match action { + UpdateAction::Set { path, value } => { + check_path_refs(path, maps, kind)?; + walk_set_value(value, maps, false)?; + } + UpdateAction::Remove { path } => check_path_refs(path, maps, kind)?, + UpdateAction::Add { path, value } => { + check_path_refs(path, maps, kind)?; + check_add_delete_value("ADD", value, maps)?; + } + UpdateAction::Delete { path, value } => { + check_path_refs(path, maps, kind)?; + check_add_delete_value("DELETE", value, maps)?; + } + } + } + if seen.contains(§ion.keyword) { + return Err(repeated_section_err(section.keyword)); + } + seen.push(section.keyword); + } + super::resolver::validate_update_paths_disjoint(&parsed.actions, maps) +} + +/// Walk a SET value. `nested` is true inside a function's operands or an +/// arithmetic operand, where `list_append` is refused. +fn walk_set_value(expr: &Expr, maps: &ExpressionMaps, nested: bool) -> Result<(), DynamoDbError> { + let kind = ExpressionKind::Update; + match expr { + Expr::Path(path) => check_path_refs(path, maps, kind), + Expr::Placeholder(name) => check_value_defined(name, maps, kind), + Expr::Arithmetic { left, op, right } => { + walk_set_value(left, maps, true)?; + walk_set_value(right, maps, true)?; + let symbol = match op { + ArithOp::Add => "+", + ArithOp::Sub => "-", + }; + for operand in [left.as_ref(), right.as_ref()] { + if let Expr::Placeholder(name) = operand + && let Some(val) = maps.values.get(name) + && !matches!(val, AttributeValue::N(_)) + { + return Err(operand_type_err(kind, symbol, attribute_type_code(val))); + } + } + Ok(()) + } + Expr::Function { name, args } => { + for arg in args { + walk_set_value(arg, maps, true)?; + } + if nested && name == "list_append" { + return Err(validation( + kind, + "The function is not allowed to be used this way in an expression; function: list_append", + )); + } + if !UPDATE_FUNCTIONS.contains(&name.as_str()) { + if CONDITION_FUNCTIONS.contains(&name.as_str()) { + return Err(validation( + kind, + &format!( + "The function is not allowed in an update expression; function: {name}" + ), + )); + } + return Err(invalid_function_name(kind, name)); + } + if args.len() != 2 { + return Err(operand_count_err(kind, name, args.len())); + } + if name == "if_not_exists" && !matches!(args[0], Expr::Path(_)) { + return Err(requires_document_path(kind, name)); + } + if name == "list_append" { + for arg in args { + if let Expr::Placeholder(p) = arg + && let Some(val) = maps.values.get(p) + && !matches!(val, AttributeValue::L(_)) + { + return Err(operand_type_err(kind, name, attribute_type_code(val))); + } + } + } + Ok(()) + } + // The update grammar produces no other node kinds in a SET value. + _ => Ok(()), + } +} + +/// `ADD` takes a number or a set; `DELETE` takes a set. The service names +/// the offending type in its own vocabulary and the same allowed set for +/// both operators (measured: `DELETE s :n` reports `NUMBER`, `typeSet: +/// ALLOWED_FOR_ADD_OPERAND`). +fn check_add_delete_value( + operator: &str, + value: &Expr, + maps: &ExpressionMaps, +) -> Result<(), DynamoDbError> { + let kind = ExpressionKind::Update; + let Expr::Placeholder(name) = value else { + return Ok(()); + }; + check_value_defined(name, maps, kind)?; + let Some(val) = maps.values.get(name) else { + return Ok(()); + }; + let type_name = match val { + AttributeValue::SS(_) | AttributeValue::NS(_) | AttributeValue::BS(_) => return Ok(()), + AttributeValue::N(_) if operator == "ADD" => return Ok(()), + AttributeValue::N(_) => "NUMBER", + AttributeValue::S(_) => "STRING", + AttributeValue::B(_) => "Binary", + AttributeValue::Bool(_) => "BOOL", + AttributeValue::Null => "NULL", + AttributeValue::L(_) => "LIST", + AttributeValue::M(_) => "MAP", + }; + Err(validation( + kind, + &format!( + "Incorrect operand type for operator or function; operator: {operator}, operand type: {type_name}, typeSet: ALLOWED_FOR_ADD_OPERAND" + ), + )) +} + +/// Validate a parsed condition, filter, or key-condition tree against the +/// service's semantic rules, in the service's order. +/// +/// `kind` names the parameter in the messages (`Invalid ConditionExpression:`, +/// `Invalid FilterExpression:`). +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` for the first rule the +/// expression breaks in the order described in the module documentation. +pub fn validate_condition_semantics( + expr: &Expr, + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + walk_condition(expr, maps, kind) +} + +/// A node in boolean position: the root, or an operand of AND, OR, NOT. +fn walk_condition( + expr: &Expr, + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + match expr { + Expr::And(l, r) | Expr::Or(l, r) => { + walk_condition(l, maps, kind)?; + walk_condition(r, maps, kind) + } + Expr::Not(inner) => walk_condition(inner, maps, kind), + Expr::Compare { left, right, .. } => { + walk_condition_operand(left, maps, kind)?; + walk_condition_operand(right, maps, kind)?; + validate_ordering_operand_types(expr, maps).map_err(|e| with_kind(e, kind)) + } + Expr::Between { operand, low, high } => { + walk_condition_operand(operand, maps, kind)?; + walk_condition_operand(low, maps, kind)?; + walk_condition_operand(high, maps, kind)?; + validate_ordering_operand_types(expr, maps).map_err(|e| with_kind(e, kind)) + } + Expr::In { operand, list } => { + walk_condition_operand(operand, maps, kind)?; + for item in list { + walk_condition_operand(item, maps, kind)?; + } + Ok(()) + } + Expr::Function { name, args } => walk_condition_function(name, args, maps, kind, false), + // A bare path or placeholder in condition position is a syntax error + // the grammar pass already reported; `Expected` desugaring never + // produces one either. + Expr::Path(_) | Expr::Placeholder(_) | Expr::Arithmetic { .. } => Ok(()), + } +} + +/// A node in operand position: a side of a comparison, a BETWEEN bound, an +/// IN member, or a function operand. +fn walk_condition_operand( + expr: &Expr, + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + match expr { + Expr::Path(path) => check_path_refs(path, maps, kind), + Expr::Placeholder(name) => check_value_defined(name, maps, kind), + Expr::Function { name, args } => walk_condition_function(name, args, maps, kind, true), + _ => Ok(()), + } +} + +/// The checks on a condition function call, after its operands: the name, +/// then its position (`size` returns a value and is refused in condition +/// position; the five boolean functions are refused in operand position: +/// `attribute_type(pk, :v) = :v` reports the position, not the operand type), +/// then the operand count, the document-path rule, and the operand types. +fn walk_condition_function( + name: &str, + args: &[Expr], + maps: &ExpressionMaps, + kind: ExpressionKind, + operand_position: bool, +) -> Result<(), DynamoDbError> { + for arg in args { + walk_condition_operand(arg, maps, kind)?; + } + if !CONDITION_FUNCTIONS.contains(&name) { + if UPDATE_FUNCTIONS.contains(&name) { + return Err(validation( + kind, + &format!("The function is not allowed in a condition expression; function: {name}"), + )); + } + return Err(invalid_function_name(kind, name)); + } + if operand_position != (name == "size") { + return Err(not_allowed_this_way(kind, name)); + } + let expected = match name { + "attribute_exists" | "attribute_not_exists" | "size" => 1, + _ => 2, + }; + if args.len() != expected { + return Err(operand_count_err(kind, name, args.len())); + } + if matches!(name, "attribute_exists" | "attribute_not_exists") + && !matches!(args[0], Expr::Path(_)) + { + return Err(requires_document_path(kind, name)); + } + if name == "begins_with" + && let Expr::Placeholder(p) = &args[1] + && let Some(val) = maps.values.get(p) + && !matches!(val, AttributeValue::S(_) | AttributeValue::B(_)) + { + return Err(operand_type_err(kind, name, attribute_type_code(val))); + } + if name == "attribute_type" + && let Expr::Placeholder(p) = &args[1] + && let Some(val) = maps.values.get(p) + { + match val { + AttributeValue::S(type_name) => { + const VALID: [&str; 10] = + ["S", "SS", "N", "NS", "B", "BS", "BOOL", "NULL", "L", "M"]; + if !VALID.contains(&type_name.as_str()) { + return Err(validation( + kind, + &format!( + "Invalid attribute type name found; type: {type_name}, valid types: {{S,SS,N,NS,B,BS,BOOL,NULL,L,M}}" + ), + )); + } + } + other => return Err(operand_type_err(kind, name, attribute_type_code(other))), + } + } + Ok(()) +} + +fn check_path_refs( + path: &[PathElement], + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + for element in path { + if let PathElement::Attribute(name) = element + && let Some(ref_name) = name.strip_prefix('#') + && !maps.names.contains_key(ref_name) + { + return Err(validation( + kind, + &format!( + "An expression attribute name used in the document path is not defined; attribute name: #{ref_name}" + ), + )); + } + } + Ok(()) +} + +fn check_value_defined( + name: &str, + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + if maps.values.contains_key(name) { + Ok(()) + } else { + Err(validation( + kind, + &format!( + "An expression attribute value used in expression is not defined; attribute value: :{name}" + ), + )) + } +} + +fn validation(kind: ExpressionKind, msg: &str) -> DynamoDbError { + DynamoDbError::ValidationException(format!("Invalid {kind}: {msg}")) +} + +fn with_kind(err: DynamoDbError, kind: ExpressionKind) -> DynamoDbError { + match err { + DynamoDbError::ValidationException(msg) if !msg.starts_with("Invalid ") => { + validation(kind, &msg) + } + other => other, + } +} + +fn invalid_function_name(kind: ExpressionKind, name: &str) -> DynamoDbError { + validation(kind, &format!("Invalid function name; function: {name}")) +} + +fn not_allowed_this_way(kind: ExpressionKind, name: &str) -> DynamoDbError { + validation( + kind, + &format!( + "The function is not allowed to be used this way in an expression; function: {name}" + ), + ) +} + +fn operand_count_err(kind: ExpressionKind, name: &str, count: usize) -> DynamoDbError { + validation( + kind, + &format!( + "Incorrect number of operands for operator or function; operator or function: {name}, number of operands: {count}" + ), + ) +} + +fn requires_document_path(kind: ExpressionKind, name: &str) -> DynamoDbError { + validation( + kind, + &format!("Operator or function requires a document path; operator or function: {name}"), + ) +} + +fn operand_type_err(kind: ExpressionKind, name: &str, type_code: &str) -> DynamoDbError { + validation( + kind, + &format!( + "Incorrect operand type for operator or function; operator or function: {name}, operand type: {type_code}" + ), + ) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::expression::parser::parse_condition; + use crate::expression::tokenizer::tokenize; + use crate::expression::update_parser::parse_update_sections; + use std::collections::HashMap; + + const U: &str = "Invalid UpdateExpression: "; + const C: &str = "Invalid ConditionExpression: "; + + fn maps(names: &[(&str, &str)], values: &[(&str, AttributeValue)]) -> ExpressionMaps { + ExpressionMaps::new( + names + .iter() + .map(|(k, v)| (k.trim_start_matches('#').to_owned(), (*v).to_owned())) + .collect::>(), + values + .iter() + .map(|(k, v)| (k.trim_start_matches(':').to_owned(), v.clone())) + .collect::>(), + ) + } + + fn n() -> AttributeValue { + AttributeValue::N("1".to_owned()) + } + fn s() -> AttributeValue { + AttributeValue::S("x".to_owned()) + } + fn l() -> AttributeValue { + AttributeValue::L(vec![]) + } + + fn update_err(expr: &str, m: &ExpressionMaps) -> String { + let tokens = tokenize(expr).unwrap(); + let parsed = parse_update_sections(&tokens, Some(expr)).unwrap(); + match validate_update_semantics(&parsed, m) { + Ok(()) => "OK".to_owned(), + Err(DynamoDbError::ValidationException(msg)) => msg, + Err(other) => panic!("unexpected {other:?}"), + } + } + + fn condition_err(expr: &str, m: &ExpressionMaps) -> String { + let tokens = tokenize(expr).unwrap(); + let parsed = parse_condition(&tokens).unwrap(); + match validate_condition_semantics(&parsed, m, ExpressionKind::Condition) { + Ok(()) => "OK".to_owned(), + Err(DynamoDbError::ValidationException(msg)) => msg, + Err(other) => panic!("unexpected {other:?}"), + } + } + + // Every expectation below is a response captured from Amazon DynamoDB + // on 2026-10-14 (issue #108 notes, probe-fix-round-*.py). + + #[test] + fn update_refs_in_expression_order() { + let m = maps(&[], &[(":v", n())]); + assert_eq!( + update_err("SET a = :q, #n = :v", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + update_err("SET #n = :q", &m), + format!( + "{U}An expression attribute name used in the document path is not defined; attribute name: #n" + ) + ); + } + + #[test] + fn repeated_section_after_its_own_actions() { + let m = maps(&[], &[(":v", n()), (":s", s()), (":n", n())]); + assert_eq!( + update_err("SET a = :v SET b = :q", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + update_err("SET a = :v SET b = :v ADD n :s", &m), + format!("{U}The \"SET\" section can only be used once in an update expression;") + ); + assert_eq!( + update_err("ADD n :s ADD m :n", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("SET a = :n SET b = :s + :n", &m), + format!( + "{U}Incorrect operand type for operator or function; operator or function: +, operand type: S" + ) + ); + } + + #[test] + fn function_names_are_case_sensitive() { + let m = maps(&[], &[(":v", n())]); + assert_eq!( + update_err("SET x = IF_NOT_EXISTS(x, :v)", &m), + format!("{U}Invalid function name; function: IF_NOT_EXISTS") + ); + assert_eq!( + update_err("SET x = foo(x, :v)", &m), + format!("{U}Invalid function name; function: foo") + ); + assert_eq!( + update_err("SET x = attribute_exists(x)", &m), + format!( + "{U}The function is not allowed in an update expression; function: attribute_exists" + ) + ); + assert_eq!( + update_err("SET x = IF_NOT_EXISTS(x, :q)", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!(update_err("SET x = if_not_exists(x, :v)", &m), "OK"); + } + + #[test] + fn nested_list_append_before_its_siblings() { + let m = maps(&[], &[(":a", l()), (":b", l()), (":n", n())]); + assert_eq!( + update_err("SET l = if_not_exists(list_append(:a, :b), :q)", &m), + format!( + "{U}The function is not allowed to be used this way in an expression; function: list_append" + ) + ); + assert_eq!( + update_err("SET l = if_not_exists(list_append(:q, :b), :a)", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + update_err("SET l = list_append(:a, :a) + :n", &m), + format!( + "{U}The function is not allowed to be used this way in an expression; function: list_append" + ) + ); + assert_eq!( + update_err("SET l = list_append(if_not_exists(l, :a), :b)", &m), + "OK" + ); + } + + #[test] + fn operand_counts_and_document_path() { + let m = maps(&[], &[(":a", l()), (":s", s())]); + assert_eq!( + update_err("SET a = if_not_exists(:a)", &m), + format!( + "{U}Incorrect number of operands for operator or function; operator or function: if_not_exists, number of operands: 1" + ) + ); + assert_eq!( + update_err("SET a = if_not_exists(a, :q, :q2)", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + update_err("SET l = if_not_exists(:a, :a)", &m), + format!( + "{U}Operator or function requires a document path; operator or function: if_not_exists" + ) + ); + assert_eq!( + update_err("SET l = list_append(:s, :a)", &m), + format!( + "{U}Incorrect operand type for operator or function; operator or function: list_append, operand type: S" + ) + ); + } + + #[test] + fn add_and_delete_operand_types() { + let m = maps( + &[], + &[ + (":s", s()), + (":n", n()), + (":b", AttributeValue::B(vec![1])), + ( + ":ss", + AttributeValue::SS(std::iter::once("a".to_owned()).collect()), + ), + (":l", l()), + ], + ); + assert_eq!( + update_err("ADD n :s", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("ADD n :b", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: ADD, operand type: Binary, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("DELETE s :n", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: DELETE, operand type: NUMBER, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("DELETE s :l", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: DELETE, operand type: LIST, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!(update_err("ADD n :n, s :ss DELETE t :ss", &m), "OK"); + assert_eq!( + update_err("ADD n :q", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + } + + #[test] + fn overlap_after_the_walk() { + let m = maps(&[], &[(":v", n()), (":s", s())]); + assert_eq!( + update_err("SET a = :v, a.b = :v ADD n :s", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("SET a = :v, a.b = :v", &m), + format!( + "{U}Two document paths overlap with each other; must remove or rewrite one of these paths; path one: [a], path two: [a, b]" + ) + ); + } + + #[test] + fn condition_function_names_and_positions() { + let m = maps(&[], &[(":v", s()), (":n", n())]); + assert_eq!( + condition_err("ATTRIBUTE_EXISTS(pk)", &m), + format!("{C}Invalid function name; function: ATTRIBUTE_EXISTS") + ); + assert_eq!( + condition_err("foo(pk) AND bar(pk)", &m), + format!("{C}Invalid function name; function: foo") + ); + assert_eq!( + condition_err("if_not_exists(a, :v) = :v", &m), + format!( + "{C}The function is not allowed in a condition expression; function: if_not_exists" + ) + ); + assert_eq!( + condition_err("attribute_exists(pk) = :v", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: attribute_exists" + ) + ); + assert_eq!( + condition_err("size(pk)", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: size" + ) + ); + assert_eq!( + condition_err("NOT size(pk)", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: size" + ) + ); + assert_eq!( + condition_err("size(attribute_exists(pk)) > :n", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: attribute_exists" + ) + ); + assert_eq!( + condition_err("attribute_exists(size(pk))", &m), + format!( + "{C}Operator or function requires a document path; operator or function: attribute_exists" + ) + ); + assert_eq!( + condition_err("attribute_exists(:v)", &m), + format!( + "{C}Operator or function requires a document path; operator or function: attribute_exists" + ) + ); + assert_eq!( + condition_err("foo(#n)", &m), + format!( + "{C}An expression attribute name used in the document path is not defined; attribute name: #n" + ) + ); + assert_eq!( + condition_err(":q = #n", &m), + format!( + "{C}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + condition_err("attribute_type(pk, :v) = :v", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: attribute_type" + ) + ); + assert_eq!(condition_err("size(pk) = size(pk)", &m), "OK"); + assert_eq!(condition_err("begins_with(:v, pk)", &m), "OK"); + assert_eq!( + condition_err("size(pk) > :n AND attribute_exists(pk)", &m), + "OK" + ); + } + + #[test] + fn condition_operand_counts_and_types() { + let m = maps( + &[], + &[ + (":n", n()), + (":t", AttributeValue::S("S".to_owned())), + (":xx", AttributeValue::S("XX".to_owned())), + (":l", l()), + ], + ); + assert_eq!( + condition_err("attribute_exists(pk, pk)", &m), + format!( + "{C}Incorrect number of operands for operator or function; operator or function: attribute_exists, number of operands: 2" + ) + ); + assert_eq!( + condition_err("begins_with(pk)", &m), + format!( + "{C}Incorrect number of operands for operator or function; operator or function: begins_with, number of operands: 1" + ) + ); + assert_eq!( + condition_err("size(pk, pk) > :n", &m), + format!( + "{C}Incorrect number of operands for operator or function; operator or function: size, number of operands: 2" + ) + ); + assert_eq!( + condition_err("begins_with(pk, :n)", &m), + format!( + "{C}Incorrect operand type for operator or function; operator or function: begins_with, operand type: N" + ) + ); + assert_eq!( + condition_err("attribute_type(pk, :n)", &m), + format!( + "{C}Incorrect operand type for operator or function; operator or function: attribute_type, operand type: N" + ) + ); + assert_eq!( + condition_err("attribute_type(pk, :xx)", &m), + format!( + "{C}Invalid attribute type name found; type: XX, valid types: {{S,SS,N,NS,B,BS,BOOL,NULL,L,M}}" + ) + ); + assert_eq!( + condition_err("pk > :l", &m), + format!( + "{C}Incorrect operand type for operator or function; operator or function: >, operand type: L" + ) + ); + assert_eq!(condition_err("attribute_type(pk, :t)", &m), "OK"); + } +} diff --git a/crates/core/src/expression/update_parser.rs b/crates/core/src/expression/update_parser.rs index a67c22e6c..6d5611b43 100755 --- a/crates/core/src/expression/update_parser.rs +++ b/crates/core/src/expression/update_parser.rs @@ -21,92 +21,127 @@ use super::parser_common; use super::tokenizer::Token; use crate::error::DynamoDbError; -/// Parse an update expression token stream into a list of actions. +/// One `SET`, `REMOVE`, `ADD`, or `DELETE` section of a parsed update +/// expression: its keyword and the range of actions it holds, in expression +/// order. A keyword may appear more than once in the parse; the semantic pass +/// reports the repetition after it has checked the section's own actions, as +/// Amazon DynamoDB does. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct UpdateSection { + pub keyword: &'static str, + pub actions: std::ops::Range, +} + +/// The grammar-level result of parsing an `UpdateExpression`: the actions in +/// expression order and the sections that group them. +#[derive(Debug, Clone, PartialEq)] +pub struct ParsedUpdate { + pub actions: Vec, + pub sections: Vec, +} + +/// Parse an update expression token stream into a list of actions, with the +/// grammar-only checks plus the repeated-section rule. +/// +/// Callers that go on to run [`validate_update_semantics`](super::semantics::validate_update_semantics) should use +/// [`parse_update_sections`] instead; this entry point serves the storage +/// crates' tests and other callers that need only a parsed action list. /// /// # Errors /// /// Returns `ValidationException` for syntax errors. pub fn parse_update(tokens: &[Token]) -> Result, DynamoDbError> { - parser_common::check_redundant_parens(tokens).map_err(|msg| { - DynamoDbError::ValidationException(format!("Invalid UpdateExpression: {msg}")) - })?; - - let mut pos = 0; - let mut actions = Vec::new(); - - while pos < tokens.len() { - match &tokens[pos] { - Token::Set => { - pos += 1; - parse_set_actions(tokens, &mut pos, &mut actions)?; - } - Token::Remove => { - pos += 1; - parse_remove_actions(tokens, &mut pos, &mut actions)?; - } - Token::Add => { - pos += 1; - parse_add_actions(tokens, &mut pos, &mut actions)?; - } - Token::Delete => { - pos += 1; - parse_delete_actions(tokens, &mut pos, &mut actions)?; - } - _ => { - return Err(validation_err( - "Invalid UpdateExpression: expected SET, REMOVE, ADD, or DELETE", - )); - } - } - } - - if actions.is_empty() { - return Err(validation_err( - "Invalid UpdateExpression: expression must contain at least one action", - )); - } - - Ok(actions) + let parsed = parse_update_sections(tokens, None)?; + check_sections_unique(&parsed.sections)?; + Ok(parsed.actions) } -/// Parse an update expression with DynamoDB-compatible syntax error messages. +/// Parse an update expression with DynamoDB-compatible syntax error messages, +/// including the repeated-section rule. +/// +/// # Errors +/// +/// Returns `ValidationException` for syntax errors. pub fn parse_update_from( tokens: &[Token], source: &str, ) -> Result, DynamoDbError> { + let parsed = parse_update_sections(tokens, Some(source))?; + check_sections_unique(&parsed.sections)?; + Ok(parsed.actions) +} + +/// Grammar pass over an `UpdateExpression`. +/// +/// Only syntax is checked here: the section keywords, the shape of each +/// action, and balanced parentheses. Any identifier followed by `(` parses +/// as a function call with its name kept as written, operand counts are not +/// checked, and a section keyword may repeat. Amazon DynamoDB reports every +/// syntax error ahead of every semantic one, so the semantic rules (function +/// names, operand counts and types, undefined references, repeated sections, +/// overlapping paths) run afterwards in [`validate_update_semantics`](super::semantics::validate_update_semantics) over +/// the returned tree. +/// +/// `source`, when given, enables the service's `Syntax error; token: ..., +/// near: ...` wording for an unexpected token at section level. +/// +/// # Errors +/// +/// Returns `ValidationException` for syntax errors. +pub fn parse_update_sections( + tokens: &[Token], + source: Option<&str>, +) -> Result { parser_common::check_redundant_parens(tokens).map_err(|msg| { DynamoDbError::ValidationException(format!("Invalid UpdateExpression: {msg}")) })?; let mut pos = 0; let mut actions = Vec::new(); + let mut sections: Vec = Vec::new(); while pos < tokens.len() { - match &tokens[pos] { + let start = actions.len(); + let keyword = match &tokens[pos] { Token::Set => { pos += 1; parse_set_actions(tokens, &mut pos, &mut actions)?; + "SET" } Token::Remove => { pos += 1; parse_remove_actions(tokens, &mut pos, &mut actions)?; + "REMOVE" } Token::Add => { pos += 1; parse_add_actions(tokens, &mut pos, &mut actions)?; + "ADD" } Token::Delete => { pos += 1; parse_delete_actions(tokens, &mut pos, &mut actions)?; + "DELETE" } token => { - let token_text = token_display_text(token); - let near = build_near_context(source, &token_text); - return Err(validation_err(&format!( - "Invalid UpdateExpression: Syntax error; token: \"{token_text}\", near: \"{near}\"" - ))); + return Err(match source { + Some(source) => { + let token_text = token_display_text(token); + let near = build_near_context(source, &token_text); + validation_err(&format!( + "Invalid UpdateExpression: Syntax error; token: \"{token_text}\", near: \"{near}\"" + )) + } + None => validation_err( + "Invalid UpdateExpression: expected SET, REMOVE, ADD, or DELETE", + ), + }); } - } + }; + sections.push(UpdateSection { + keyword, + actions: start..actions.len(), + }); } if actions.is_empty() { @@ -115,7 +150,24 @@ pub fn parse_update_from( )); } - Ok(actions) + Ok(ParsedUpdate { actions, sections }) +} + +/// Each of SET, REMOVE, ADD, and DELETE may open one section. The first +/// repeated section in order of appearance is named. +pub(crate) fn check_sections_unique(sections: &[UpdateSection]) -> Result<(), DynamoDbError> { + for (i, section) in sections.iter().enumerate() { + if sections[..i].iter().any(|s| s.keyword == section.keyword) { + return Err(repeated_section_err(section.keyword)); + } + } + Ok(()) +} + +pub(crate) fn repeated_section_err(keyword: &str) -> DynamoDbError { + validation_err(&format!( + "Invalid UpdateExpression: The \"{keyword}\" section can only be used once in an update expression;" + )) } fn token_display_text(token: &Token) -> String { @@ -223,7 +275,12 @@ fn parse_value_expr(tokens: &[Token], pos: &mut usize) -> Result Result { if *pos >= tokens.len() { return Err(validation_err("Invalid UpdateExpression: expected operand")); @@ -235,12 +292,8 @@ fn parse_set_operand(tokens: &[Token], pos: &mut usize) -> Result { - let fn_lower = name.to_ascii_lowercase(); - if is_set_function(&fn_lower) - && *pos + 1 < tokens.len() - && tokens[*pos + 1] == Token::LParen - { + Token::Ident(_) => { + if *pos + 1 < tokens.len() && tokens[*pos + 1] == Token::LParen { parse_set_function_call(tokens, pos) } else { parser_common::parse_path(tokens, pos).map(Expr::Path) @@ -266,7 +319,7 @@ fn parse_set_operand(tokens: &[Token], pos: &mut usize) -> Result Result { let name = match &tokens[*pos] { - Token::Ident(n) => n.to_ascii_lowercase(), + Token::Ident(n) => n.clone(), _ => { return Err(validation_err( "Invalid UpdateExpression: expected function name", @@ -276,23 +329,22 @@ fn parse_set_function_call(tokens: &[Token], pos: &mut usize) -> Result bool { - matches!(name, "if_not_exists" | "list_append") -} - /// Parse ADD actions: `ADD path value (, path value)*` fn parse_add_actions( tokens: &[Token], diff --git a/crates/core/src/validation/mod.rs b/crates/core/src/validation/mod.rs index 5b780fd6a..93a0d3d5b 100755 --- a/crates/core/src/validation/mod.rs +++ b/crates/core/src/validation/mod.rs @@ -1201,6 +1201,14 @@ pub fn validate_item_keys( /// Validate that a key map contains exactly the key attributes and nothing else. /// +/// A key with an attribute missing, an attribute of the wrong type, or an +/// extra attribute is refused with `The provided key element does not match +/// the schema` on every key-shaped API (GetItem, DeleteItem, UpdateItem, the +/// batch and transaction key paths; measured against Amazon DynamoDB +/// 2026-10-14). Only `PutItem`, whose key arrives inside the item, names the +/// attribute (see [`validate_item_keys`]). An empty key value keeps its own +/// message. +/// /// # Errors /// /// Returns `DynamoDbError::ValidationException` if the key has extra/missing attributes or wrong types. @@ -1212,28 +1220,22 @@ pub fn validate_key_only( // Must contain exactly the key attributes let expected_count = key_schema.len(); if key.len() != expected_count { - return Err(DynamoDbError::ValidationException( - "The provided key element does not match the schema".to_owned(), - )); + return Err(key_schema_mismatch()); } for ks in key_schema { - let value = key.get(&ks.attribute_name).ok_or_else(|| { - DynamoDbError::ValidationException(format!( - "One or more parameter values were invalid: Missing the key {} in the item", - ks.attribute_name - )) - })?; - validate_key_attribute_type(&ks.attribute_name, value, attr_defs)?; + let value = key + .get(&ks.attribute_name) + .ok_or_else(key_schema_mismatch)?; + validate_key_attribute_type(&ks.attribute_name, value, attr_defs) + .map_err(remap_key_shape_errors)?; validate_no_empty_key_value(&ks.attribute_name, value)?; } Ok(()) } -/// Batch-specific key validation: uses `DynamoDB`'s batch error message for type mismatches. -/// -/// Real `DynamoDB` returns "The provided key element does not match the schema" for -/// batch operations, not the single-item "Type mismatch" message. +/// Key validation for the batch and transaction key paths. Same rules and +/// message as [`validate_key_only`]; kept as a separate name for the callers. /// /// # Errors /// @@ -1243,10 +1245,12 @@ pub fn validate_batch_key_only( key_schema: &[KeySchemaElement], attr_defs: &[AttributeDefinition], ) -> Result<(), DynamoDbError> { - validate_key_only(key, key_schema, attr_defs).map_err(remap_key_type_mismatch) + validate_key_only(key, key_schema, attr_defs) } -/// Batch-specific item key validation: uses `DynamoDB`'s batch error message for type mismatches. +/// `BatchWriteItem` Put key validation: a missing or mistyped key attribute +/// inside the item is `The provided key element does not match the schema`, +/// where `PutItem` names the attribute (measured 2026-10-14). /// /// # Errors /// @@ -1256,20 +1260,24 @@ pub fn validate_batch_item_keys( key_schema: &[KeySchemaElement], attr_defs: &[AttributeDefinition], ) -> Result<(), DynamoDbError> { - validate_item_keys(item, key_schema, attr_defs).map_err(remap_key_type_mismatch) + validate_item_keys(item, key_schema, attr_defs).map_err(remap_key_shape_errors) } -/// Remap key type-mismatch errors to the batch/transaction-specific message. -/// -/// Real `DynamoDB` uses "The provided key element does not match the schema" -/// for batch and transaction operations, not the single-item "Type mismatch" -/// message. -fn remap_key_type_mismatch(err: DynamoDbError) -> DynamoDbError { +fn key_schema_mismatch() -> DynamoDbError { + DynamoDbError::ValidationException( + "The provided key element does not match the schema".to_owned(), + ) +} + +/// Remap the item-shaped key messages (`Missing the key`, `Type mismatch for +/// key`) to the key-shaped `The provided key element does not match the +/// schema`. +fn remap_key_shape_errors(err: DynamoDbError) -> DynamoDbError { match &err { - DynamoDbError::ValidationException(msg) if msg.contains("Type mismatch for key ") => { - DynamoDbError::ValidationException( - "The provided key element does not match the schema".to_owned(), - ) + DynamoDbError::ValidationException(msg) + if msg.contains("Type mismatch for key ") || msg.contains("Missing the key ") => + { + key_schema_mismatch() } _ => err, } diff --git a/crates/engine/src/batch_write_item.rs b/crates/engine/src/batch_write_item.rs index e285a921a..c6b100275 100755 --- a/crates/engine/src/batch_write_item.rs +++ b/crates/engine/src/batch_write_item.rs @@ -273,13 +273,16 @@ pub async fn handle_batch_write_item( // Per-item WCU already accumulated above (M-1: DynamoDB rounds per item, then sums). let wcu = total_wcu; + // With ReturnItemCollectionMetrics SIZE the service always returns the + // member: the populated map when a written table has a local secondary + // index, and `{}` when none does (measured 2026-10-14, as for + // TransactWriteItems). Without SIZE the member is absent. let output = BatchWriteItemOutput { unprocessed_items: HashMap::new(), consumed_capacity, - item_collection_metrics: if all_icm.is_empty() { - None - } else { - Some(all_icm) + item_collection_metrics: match input.return_item_collection_metrics { + ReturnItemCollectionMetrics::Size => Some(all_icm), + ReturnItemCollectionMetrics::None => None, }, }; let body = serialize_output(&output)?; diff --git a/crates/engine/src/capacity_helpers.rs b/crates/engine/src/capacity_helpers.rs index 73b8c04fb..534b70cc3 100755 --- a/crates/engine/src/capacity_helpers.rs +++ b/crates/engine/src/capacity_helpers.rs @@ -380,6 +380,31 @@ pub fn batch_write_capacity<'a>( } } +/// Per-table capacity totals in the order the tables first appear in the +/// request. +/// +/// Amazon DynamoDB's own order of the `ConsumedCapacity` entries of a +/// transaction follows no documented rule (measured 2026-10-14: hash order of +/// the table names, which differs between name sets), so no client can depend +/// on it. A `HashMap` here made ExtendDB's order change from one process to +/// the next; first appearance is stable and is what a reader of the request +/// expects. +#[derive(Default)] +pub struct PerTableCapacity(Vec<(String, f64)>); + +impl PerTableCapacity { + pub fn add(&mut self, table: &str, units: f64) { + match self.0.iter_mut().find(|(t, _)| t == table) { + Some((_, total)) => *total += units, + None => self.0.push((table.to_owned(), units)), + } + } + + pub fn iter(&self) -> impl Iterator { + self.0.iter().map(|(t, cu)| (t.as_str(), *cu)) + } +} + /// Build a `Vec` for `TransactGetItems`, or `None` if not requested. /// /// Transactions differ from single-item/batch reads: real DynamoDB emits the diff --git a/crates/engine/src/delete_item.rs b/crates/engine/src/delete_item.rs index 049c368f6..f22f44f11 100755 --- a/crates/engine/src/delete_item.rs +++ b/crates/engine/src/delete_item.rs @@ -11,7 +11,7 @@ use extenddb_core::types::{DeleteItemInput, DeleteItemOutput, ReturnValues, item use crate::OperationContext; use crate::capacity_helpers; use crate::create_table::{storage_err_to_dynamo, storage_err_to_dynamo_with_ccf}; -use crate::expression_helpers::resolve_condition; +use crate::expression_helpers::{resolve_condition, single_item_write_prefix}; use crate::serialize_output; use crate::stream_capture; use crate::{DispatchMetrics, DispatchResult}; @@ -68,13 +68,16 @@ pub async fn handle_delete_item( ], &[("ConditionExpression", has_condition)], )?; + // The single-item write APIs name no expression in the values message and + // wrap every expression-stage error in `1 validation error detected: `. extenddb_core::expression::validate_expression_param_usage( input.expression_attribute_names.as_ref(), has_condition, input.expression_attribute_values.as_ref(), has_condition, - &[extenddb_core::expression::ExpressionKind::Condition], - )?; + &[], + ) + .map_err(single_item_write_prefix)?; // ConditionalOperator requires an Expected with two or more conditions. extenddb_core::validation::validate_conditional_operator_usage( @@ -103,7 +106,8 @@ pub async fn handle_delete_item( input.expected.as_ref(), input.conditional_operator, &ctx.limits, - )?; + ) + .map_err(single_item_write_prefix)?; if input.expected.is_none() || input @@ -112,6 +116,9 @@ pub async fn handle_delete_item( .is_some_and(std::collections::HashMap::is_empty) { let exprs: Vec<&extenddb_core::expression::Expr> = condition.iter().collect(); + // The condition's references, functions, and operand types were + // checked by `resolve_condition`; the unused check follows them and + // runs ahead of the table lookup, as on the service. extenddb_core::expression::validate_unused_attributes( &maps.names, &maps.values, @@ -119,7 +126,8 @@ pub async fn handle_delete_item( &[], &std::collections::HashSet::new(), &std::collections::HashSet::new(), - )?; + ) + .map_err(single_item_write_prefix)?; } let key_info = ctx diff --git a/crates/engine/src/expression_helpers.rs b/crates/engine/src/expression_helpers.rs index 913333015..5d02dd41d 100755 --- a/crates/engine/src/expression_helpers.rs +++ b/crates/engine/src/expression_helpers.rs @@ -8,8 +8,9 @@ use std::collections::HashMap; use extenddb_core::error::DynamoDbError; use extenddb_core::expression::{ Expr, ExpressionKind, ExpressionMaps, KeyCondition, PathElement, Token, UpdateAction, - parse_condition_with_depth_limit, parse_key_condition, parse_projection, parse_update_from, - tokenize_for, tokenize_with_limit, validate_no_reserved_words, validate_ordering_operand_types, + parse_condition_with_depth_limit, parse_key_condition, parse_projection, parse_update_sections, + tokenize_for, tokenize_with_limit, validate_condition_semantics, validate_no_reserved_words, + validate_update_semantics, }; use extenddb_core::limits::LimitsConfig; use extenddb_core::types::{AttributeValue, ConditionalOperator, ExpectedAttributeValue}; @@ -66,11 +67,32 @@ pub fn check_expression_size( expr: &str, kind: ExpressionKind, limits: &LimitsConfig, +) -> Result<(), DynamoDbError> { + check_expression_size_with(expr, kind, limits, kind.size_error_includes_length()) +} + +/// `check_expression_size` with the length suffix decided by the caller. +/// +/// Amazon DynamoDB appends ` expression size: N` for a `ConditionExpression` +/// inside a transaction and for a Scan `FilterExpression`, and omits it for a +/// `ConditionExpression` on PutItem, UpdateItem, or DeleteItem and for a Query +/// `FilterExpression` (measured 2026-10-14). The suffix is therefore a property +/// of the operation, not of the expression kind alone. +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` when the expression is over +/// the configured byte limit. +pub fn check_expression_size_with( + expr: &str, + kind: ExpressionKind, + limits: &LimitsConfig, + include_length: bool, ) -> Result<(), DynamoDbError> { if expr.len() > limits.max_expression_length_bytes { let mut msg = format!("Invalid {kind}: Expression size has exceeded the maximum allowed size;"); - if kind.size_error_includes_length() { + if include_length { use std::fmt::Write as _; let _ = write!(msg, " expression size: {}", expr.len()); } @@ -79,33 +101,68 @@ pub fn check_expression_size( Ok(()) } -/// Tokenize, reserved-word check, and parse a required `ConditionExpression`. +/// Amazon DynamoDB wraps the expression-stage errors of the single-item write +/// APIs (PutItem, UpdateItem, DeleteItem) in `1 validation error detected: `: +/// every `Invalid ConditionExpression:` and `Invalid UpdateExpression:` message, +/// the unused-name and unused-value messages, and the names-or-values-without- +/// expression messages. The read APIs, the batch APIs, and the transactions +/// return the same messages bare. Apply this at the expression stage of those +/// three handlers only; other validation messages there ("Cannot update +/// attribute pk", key schema, item size) carry no prefix on the service. +pub fn single_item_write_prefix(err: DynamoDbError) -> DynamoDbError { + match err { + DynamoDbError::ValidationException(msg) + if !msg.starts_with("1 validation error detected") + && (msg.starts_with("Invalid ConditionExpression:") + || msg.starts_with("Invalid UpdateExpression:") + || msg.starts_with("Value provided in ExpressionAttribute") + || msg.starts_with("ExpressionAttributeNames can only be specified") + || msg.starts_with("ExpressionAttributeValues can only be specified")) => + { + DynamoDbError::ValidationException(format!("1 validation error detected: {msg}")) + } + other => other, + } +} + +/// Tokenize, reserved-word check, and parse a required `ConditionExpression`, +/// with the size-error length suffix decided by the caller (see +/// `check_expression_size_with`). /// /// Errors carry the `ConditionExpression` prefix, matching Amazon DynamoDB. /// /// # Errors /// -/// Returns `DynamoDbError::ValidationException` for syntax or reserved-word errors. -pub fn parse_condition_expr(expr: &str, limits: &LimitsConfig) -> Result { - check_expression_size(expr, ExpressionKind::Condition, limits)?; +/// Returns `DynamoDbError::ValidationException` for over-size, syntax, or +/// reserved-word errors. +pub fn parse_condition_expr_with( + expr: &str, + limits: &LimitsConfig, + include_length: bool, +) -> Result { + check_expression_size_with(expr, ExpressionKind::Condition, limits, include_length)?; tokenize_expression(expr, limits) .and_then(|tokens| parse_condition_with_depth_limit(&tokens, limits.max_expression_depth)) .map_err(|e| prefix_expression_error(e, ExpressionKind::Condition)) } -/// Tokenize, reserved-word check, and parse an `UpdateExpression`. -/// -/// Errors carry the `UpdateExpression` prefix, matching Amazon DynamoDB. +/// Size-check, tokenize, parse, and semantically validate an +/// `UpdateExpression` against the request's expression maps: every syntax +/// error first, then the service's single ordered walk (function names, +/// operand counts and types, undefined references, repeated sections), then +/// overlapping paths. See `extenddb_core::expression::validate_update_semantics`. /// /// # Errors /// -/// Returns `DynamoDbError::ValidationException` for syntax or reserved-word errors. -pub fn parse_update_expr( +/// Returns `DynamoDbError::ValidationException` with the `Invalid +/// UpdateExpression:` prefix. +pub fn parse_update_expr_checked( update_expr: &str, + maps: &ExpressionMaps, limits: &LimitsConfig, ) -> Result, DynamoDbError> { check_expression_size(update_expr, ExpressionKind::Update, limits)?; - tokenize_for( + let parsed = tokenize_for( update_expr, limits.max_expression_tokens, ExpressionKind::Update, @@ -114,42 +171,54 @@ pub fn parse_update_expr( if limits.enforce_reserved_keywords { validate_no_reserved_words(&update_tokens)?; } - parse_update_from(&update_tokens, update_expr) + parse_update_sections(&update_tokens, Some(update_expr)) }) - .map_err(|e| prefix_expression_error(e, ExpressionKind::Update)) + .map_err(|e| prefix_expression_error(e, ExpressionKind::Update))?; + validate_update_semantics(&parsed, maps)?; + Ok(parsed.actions) } -/// Parse an optional condition expression string into an AST. +/// Size-check, tokenize, parse, and semantically validate a condition-grammar +/// expression (`ConditionExpression` or `FilterExpression`) against the +/// request's expression maps. Syntax errors come first, then the service's +/// ordered walk: undefined references, function names, operand counts and +/// types, and functions in the wrong position. See +/// `extenddb_core::expression::validate_condition_semantics`. /// -/// Returns `None` if the input is `None` or empty. +/// `kind` labels the messages; `include_length` decides the size-error suffix +/// (see [`check_expression_size_with`]). /// /// # Errors /// -/// Returns `DynamoDbError::ValidationException` for syntax errors. -pub fn parse_optional_condition( - expr: Option<&str>, +/// Returns `DynamoDbError::ValidationException` with the `Invalid {kind}:` prefix. +pub fn parse_condition_checked( + expr: &str, + maps: &ExpressionMaps, limits: &LimitsConfig, -) -> Result, DynamoDbError> { - match expr { - Some(s) if !s.is_empty() => parse_condition_expr(s, limits).map(Some), - _ => Ok(None), - } + include_length: bool, + kind: ExpressionKind, +) -> Result { + let parsed = parse_condition_expr_with(expr, limits, include_length) + .map_err(|e| prefix_expression_error(e, kind))?; + validate_condition_semantics(&parsed, maps, kind)?; + Ok(parsed) } -/// Parse an optional filter expression string into an AST. -/// -/// `FilterExpression` uses the same grammar as `ConditionExpression`. -/// Returns `None` if the input is `None` or empty. -/// -/// # Errors -/// -/// Returns `DynamoDbError::ValidationException` for syntax errors. +/// Parse an optional filter expression (grammar only, `FilterExpression` +/// prefix). The handlers run the semantic pass through +/// [`parse_condition_checked`]; this remains for the parser tests. +#[cfg(test)] pub fn parse_optional_filter( expr: Option<&str>, limits: &LimitsConfig, + include_length: bool, ) -> Result, DynamoDbError> { - parse_optional_condition(expr, limits) - .map_err(|e| prefix_expression_error(e, ExpressionKind::Filter)) + match expr { + Some(s) if !s.is_empty() => parse_condition_expr_with(s, limits, include_length) + .map(Some) + .map_err(|e| prefix_expression_error(e, ExpressionKind::Filter)), + _ => Ok(None), + } } /// Resolve a condition from either `ConditionExpression` or legacy `Expected`. @@ -201,11 +270,20 @@ pub fn resolve_condition( } let maps = build_expression_maps(names, values); - let condition = parse_optional_condition(condition_expression, limits)?; - if let Some(ref expr) = condition { - validate_ordering_operand_types(expr, &maps) - .map_err(|e| prefix_expression_error(e, ExpressionKind::Condition))?; - } + // The single-item write APIs report an over-size condition without the + // length suffix; transactions keep it (see `check_expression_size_with`). + // The whole condition, syntax and semantics, is checked here before the + // caller looks at any other expression, as the service does. + let condition = match condition_expression { + Some(s) if !s.is_empty() => Some(parse_condition_checked( + s, + &maps, + limits, + false, + ExpressionKind::Condition, + )?), + _ => None, + }; Ok((condition, maps)) } @@ -262,6 +340,44 @@ pub fn parse_key_condition_expr( .map_err(|e| prefix_expression_error(e, ExpressionKind::KeyCondition)) } +/// Reject a parsed `KeyConditionExpression` that references a `#name` or +/// `:value` absent from the request maps, with the `Invalid +/// KeyConditionExpression:` prefix the service uses. Runs before the key +/// schema is consulted, so an undefined `#n` is reported as such rather than +/// as a key schema mismatch (measured 2026-10-14). +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` naming the first undefined +/// reference, names before values. +pub fn check_key_condition_refs( + kc: &KeyCondition, + maps: &ExpressionMaps, +) -> Result<(), DynamoDbError> { + let (names, values) = extenddb_core::expression::collect_key_condition_refs(kc); + let mut missing: Vec<&String> = names + .iter() + .filter(|n| !maps.names.contains_key(*n)) + .collect(); + missing.sort(); + if let Some(name) = missing.first() { + return Err(DynamoDbError::ValidationException(format!( + "Invalid KeyConditionExpression: An expression attribute name used in the document path is not defined; attribute name: #{name}" + ))); + } + let mut missing: Vec<&String> = values + .iter() + .filter(|v| !maps.values.contains_key(*v)) + .collect(); + missing.sort(); + if let Some(value) = missing.first() { + return Err(DynamoDbError::ValidationException(format!( + "Invalid KeyConditionExpression: An expression attribute value used in expression is not defined; attribute value: :{value}" + ))); + } + Ok(()) +} + /// Prefix an expression error with the expression type, matching `DynamoDB`'s format. /// /// `FilterExpression` shares the condition parser, so its errors arrive labelled @@ -302,7 +418,7 @@ mod tests { "((a = :v AND b = :v2))", "((NOT (a = :v)))", ] { - let err = parse_optional_condition(Some(expr), &limits).unwrap_err(); + let err = parse_condition_expr_with(expr, &limits, true).unwrap_err(); assert!( matches!(&err, DynamoDbError::ValidationException(msg) if msg == CONDITION_REDUNDANT), "expr {expr}: got {err:?}" @@ -320,7 +436,7 @@ mod tests { "(NOT (a = :v))", ] { assert!( - parse_optional_condition(Some(expr), &limits).is_ok(), + parse_condition_expr_with(expr, &limits, true).is_ok(), "expr {expr} should parse" ); } @@ -329,7 +445,7 @@ mod tests { #[test] fn filter_redundant_parens_rejected_with_filter_label() { let limits = LimitsConfig::default(); - let err = parse_optional_filter(Some("((a = :v))"), &limits).unwrap_err(); + let err = parse_optional_filter(Some("((a = :v))"), &limits, true).unwrap_err(); assert!( matches!(&err, DynamoDbError::ValidationException(msg) if msg == "Invalid FilterExpression: The expression has redundant parentheses;"), @@ -340,7 +456,7 @@ mod tests { #[test] fn filter_parser_errors_carry_filter_label() { let limits = LimitsConfig::default(); - let err = parse_optional_filter(Some("a"), &limits).unwrap_err(); + let err = parse_optional_filter(Some("a"), &limits, true).unwrap_err(); assert!( matches!(&err, DynamoDbError::ValidationException(msg) if msg.starts_with("Invalid FilterExpression:")), @@ -354,7 +470,7 @@ mod tests { let limits = LimitsConfig::default(); let path = "a".to_owned() + &"z".repeat(limits.max_expression_length_bytes); let expr = format!("{path} = :v"); - let err = parse_optional_filter(Some(&expr), &limits).unwrap_err(); + let err = parse_optional_filter(Some(&expr), &limits, true).unwrap_err(); assert!( matches!(&err, DynamoDbError::ValidationException(msg) if *msg == format!( @@ -369,7 +485,7 @@ mod tests { let limits = LimitsConfig::default(); let path = "a".to_owned() + &"z".repeat(limits.max_expression_length_bytes); let expr = format!("attribute_not_exists({path})"); - let err = parse_optional_condition(Some(&expr), &limits).unwrap_err(); + let err = parse_condition_expr_with(&expr, &limits, true).unwrap_err(); assert!( matches!(&err, DynamoDbError::ValidationException(msg) if *msg == format!( diff --git a/crates/engine/src/put_item.rs b/crates/engine/src/put_item.rs index ffd6f70b4..b51d54729 100755 --- a/crates/engine/src/put_item.rs +++ b/crates/engine/src/put_item.rs @@ -13,7 +13,7 @@ use extenddb_core::types::{PutItemInput, PutItemOutput, ReturnValues, item_size_ use crate::OperationContext; use crate::capacity_helpers; use crate::create_table::{storage_err_to_dynamo, storage_err_to_dynamo_with_ccf}; -use crate::expression_helpers::resolve_condition; +use crate::expression_helpers::{resolve_condition, single_item_write_prefix}; use crate::serialize_output; use crate::stream_capture; use crate::{DispatchMetrics, DispatchResult}; @@ -109,13 +109,16 @@ pub async fn handle_put_item( ], &[("ConditionExpression", has_expression)], )?; + // The single-item write APIs name no expression in the values message and + // wrap every expression-stage error in `1 validation error detected: `. extenddb_core::expression::validate_expression_param_usage( input.expression_attribute_names.as_ref(), has_expression, input.expression_attribute_values.as_ref(), has_expression, - &[extenddb_core::expression::ExpressionKind::Condition], - )?; + &[], + ) + .map_err(single_item_write_prefix)?; // ConditionalOperator requires an Expected with two or more conditions. extenddb_core::validation::validate_conditional_operator_usage( @@ -138,7 +141,8 @@ pub async fn handle_put_item( input.expected.as_ref(), input.conditional_operator, &ctx.limits, - )?; + ) + .map_err(single_item_write_prefix)?; if input.expected.is_none() || input @@ -147,6 +151,9 @@ pub async fn handle_put_item( .is_some_and(std::collections::HashMap::is_empty) { let exprs: Vec<&extenddb_core::expression::Expr> = condition.iter().collect(); + // The condition's references, functions, and operand types were + // checked by `resolve_condition`; the unused check follows them and + // runs ahead of the table lookup, as on the service. extenddb_core::expression::validate_unused_attributes( &maps.names, &maps.values, @@ -154,7 +161,8 @@ pub async fn handle_put_item( &[], &std::collections::HashSet::new(), &std::collections::HashSet::new(), - )?; + ) + .map_err(single_item_write_prefix)?; } let key_info = ctx diff --git a/crates/engine/src/query.rs b/crates/engine/src/query.rs index f45526af8..1473de845 100755 --- a/crates/engine/src/query.rs +++ b/crates/engine/src/query.rs @@ -3,7 +3,7 @@ //! `Query` operation handler. -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use serde_json::Value; @@ -19,7 +19,7 @@ use extenddb_storage::error::StorageError; use crate::OperationContext; use crate::capacity_helpers; use crate::create_table::storage_err_to_dynamo; -use crate::expression_helpers::{build_expression_maps, parse_optional_filter}; +use crate::expression_helpers::build_expression_maps; use crate::index_helpers::{ VectorIndexReadRefusal, classify_unresolved_index_read, combined_lek_key_schema, validate_query_exclusive_start_key, @@ -71,72 +71,6 @@ pub async fn handle_query( )?; let input: QueryInput = serde_json::from_value(body).map_err(crate::deserialize_error)?; - // P118: Fetch key_info first so we can use table_id for index lookup. - let key_info = ctx - .table_key_info(&input.table_name) - .await - .map_err(storage_err_to_dynamo)?; - - // GSI/LSI: resolve index metadata if querying a secondary index. - // Uses table_id from pre-fetched key_info to skip redundant table lookup (P118 #4). - // A vector index is not a row in the `indexes` catalog, so a not-found - // result is re-resolved against the vector index metadata before the - // name is treated as absent. - let (index_info, vector_index_named) = match input.index_name { - Some(ref idx_name) => { - match ctx - .storage - .index_info_by_table_id(&key_info.table_id, idx_name) - .await - { - // Defense in depth: no in-tree backend stores a vector index in - // `indexes`, but if one ever surfaces here it must be refused, - // not sent down the GSI/LSI data path. - Ok(info) if info.index_type == IndexType::Vector => (None, true), - Ok(info) => (Some(info), false), - Err(err @ StorageError::IndexNotFound(_)) => { - match classify_unresolved_index_read(ctx, &key_info, idx_name).await? { - VectorIndexReadRefusal::NotFound => { - return Err(storage_err_to_dynamo(err)); - } - // Query refuses a backfilling vector index with the same - // message as an active one (measured 2026-08-20; Scan - // differs). The refusal itself fires further down, after - // the KeyConditionExpression checks. - VectorIndexReadRefusal::Backfilling - | VectorIndexReadRefusal::NotSupported => (None, true), - } - } - Err(err) => return Err(storage_err_to_dynamo(err)), - } - } - None => (None, false), - }; - - // ConsistentRead is not supported on GSI queries (tenet 1: fidelity). - // Measured 2026-08-20: it fires for a vector index too, with the same - // wording, and before the vector-index refusal. - if input.consistent_read == Some(true) - && (vector_index_named - || index_info - .as_ref() - .is_some_and(|idx| idx.index_type == IndexType::Gsi)) - { - return Err(DynamoDbError::ValidationException( - "Consistent reads are not supported on global secondary indexes".to_owned(), - )); - } - - // Select=ALL_ATTRIBUTES requires an ALL-projection GSI (shared with Scan). - if let Some(ref idx) = index_info { - extenddb_core::validation::validate_all_attributes_index_support( - input.select, - idx.index_type == IndexType::Gsi, - idx.projection.projection_type == ProjectionType::All, - &idx.index_name, - )?; - } - // Validate Limit >= 1 (REQ-QUERY-001) if let Some(limit) = input.limit && limit < 1 @@ -146,26 +80,6 @@ pub async fn handle_query( )); } - // For index queries, build a key_info that reflects the index's key schema - // so the storage layer uses the correct SK column for the index table. - let query_key_info = if let Some(ref idx) = index_info { - TableKeyInfo { - table_name: key_info.table_name.clone(), - account_id: key_info.account_id.clone(), - table_id: key_info.table_id.clone(), - key_schema: idx.key_schema.clone(), - base_key_schema: key_info.key_schema.clone(), - attribute_definitions: key_info.attribute_definitions.clone(), - has_lsi: key_info.has_lsi, - global_secondary_indexes: key_info.global_secondary_indexes.clone(), - local_secondary_indexes: key_info.local_secondary_indexes.clone(), - stream_specification: None, // Queries don't capture stream records - vector_indexes: key_info.vector_indexes.clone(), - } - } else { - key_info.clone() - }; - // --- Legacy vs expression mutual exclusivity checks --- let kce_present = input.key_condition_expression.is_some(); let has_kc = input.key_conditions.as_ref().is_some_and(|m| !m.is_empty()); @@ -233,116 +147,42 @@ pub async fn handle_query( input.query_filter.as_ref().map_or(0, HashMap::len), )?; - // Parse KeyConditionExpression or desugar legacy KeyConditions - let (mut key_condition, legacy_kc_maps) = if let Some(kce_str) = - input.key_condition_expression.as_deref() - { - let parsed = crate::expression_helpers::parse_key_condition_expr(kce_str, &ctx.limits)?; - (parsed, None) - } else if let Some(ref kc) = input.key_conditions { - let key_schema_pairs: Vec<(String, bool)> = query_key_info - .key_schema - .iter() - .map(|ks| (ks.attribute_name.clone(), ks.key_type == KeyType::Hash)) - .collect(); - let (kc_parsed, kc_maps) = desugar_key_conditions(kc, &key_schema_pairs)?; - (kc_parsed, Some(kc_maps)) - } else { - return Err(DynamoDbError::ValidationException( - "Either the KeyConditions or KeyConditionExpression parameter must be specified in the request." - .to_owned(), - )); - }; - - // A vector index is searched only via the vector search API, never - // queried. Measured 2026-08-20: the service refuses after the - // KeyConditionExpression presence and syntax checks, hence below the parse. - if vector_index_named { - return Err(DynamoDbError::ValidationException( - VECTOR_INDEX_QUERY_NOT_SUPPORTED.to_owned(), - )); - } - - // Use legacy maps for key condition resolution if KeyConditions was used - let effective_maps = if let Some(ref kc_maps) = legacy_kc_maps { - kc_maps - } else { - &maps - }; - - // Correct PK/SK assignment when both clauses are equality comparisons. - // The parser can't distinguish PK from SK without the key schema. - let pk_attr = &query_key_info.key_schema[0].attribute_name; - key_condition.resolve_pk_sk(pk_attr, &effective_maps.names)?; - - // Validate that the partition key is actually referenced in the condition. - let pk_resolved = resolve_path_attr_name(&key_condition.pk_path, &effective_maps.names); - if pk_resolved.as_deref() != Some(pk_attr.as_str()) { - return Err(DynamoDbError::ValidationException(format!( - "Query condition missed key schema element: {pk_attr}" - ))); - } - - // For multi-part key schemas (GSIs with >1 HASH attribute), reclassify - // the parsed conditions so all HASH attributes go to pk_path/extra_pk_conditions - // and the RANGE condition stays as sk_condition. - if extenddb_core::types::is_multipart_key_schema(&query_key_info.key_schema) { - let hash_elements = extenddb_core::types::hash_key_elements(&query_key_info.key_schema); - let hash_attrs: Vec<&str> = hash_elements - .iter() - .map(|ks| ks.attribute_name.as_str()) - .collect(); - key_condition.resolve_multipart(&hash_attrs, &effective_maps.names)?; - - // Validate all HASH attributes are present in the KeyConditionExpression. - let provided_count = 1 + key_condition.extra_pk_conditions.len(); - if provided_count != hash_attrs.len() { - // Find the first missing HASH attribute for the error message. - let missing = hash_attrs - .iter() - .find(|attr| { - let pk_name = - resolve_path_attr_name(&key_condition.pk_path, &effective_maps.names); - if pk_name.as_deref() == Some(*attr) { - return false; - } - !key_condition.extra_pk_conditions.iter().any(|(path, _)| { - resolve_path_attr_name(path, &effective_maps.names).as_deref() - == Some(*attr) - }) - }) - .unwrap_or(&hash_attrs[0]); - return Err(DynamoDbError::ValidationException(format!( - "Query condition missed key schema element: {missing}" - ))); - } - } - - // Parse FilterExpression or desugar legacy QueryFilter - let (filter, filter_maps) = if let Some(ref qf) = input.query_filter { - if qf.is_empty() { - ( - parse_optional_filter(input.filter_expression.as_deref(), &ctx.limits)?, - None, - ) - } else { + // Request-stage expression checks, before the table is looked up, in the + // order the service applies them (measured 2026-10-14): the + // FilterExpression (syntax, then references, functions, and operand + // types), the KeyConditionExpression (syntax, then references), the + // ProjectionExpression (syntax, then references and overlap), then unused + // names and values. A missing table is reported after all of these. + let (filter, filter_maps) = + if let Some(qf) = input.query_filter.as_ref().filter(|qf| !qf.is_empty()) { let cond_op = input.conditional_operator.unwrap_or_default(); let (expr, fmaps) = desugar_filter(qf, cond_op)?; (Some(expr), Some(fmaps)) + } else { + match input.filter_expression.as_deref() { + Some(expr) if !expr.is_empty() => ( + Some(crate::expression_helpers::parse_condition_checked( + expr, + &maps, + &ctx.limits, + false, + ExpressionKind::Filter, + )?), + None, + ), + _ => (None, None), + } + }; + + let parsed_key_condition = match input.key_condition_expression.as_deref() { + Some(kce_str) => { + let parsed = crate::expression_helpers::parse_key_condition_expr(kce_str, &ctx.limits)?; + crate::expression_helpers::check_key_condition_refs(&parsed, &maps)?; + Some(parsed) } - } else { - ( - parse_optional_filter(input.filter_expression.as_deref(), &ctx.limits)?, - None, - ) + None => None, }; - // Validate #name references in filter are defined in ExpressionAttributeNames - if let Some(ref filter_expr) = filter { - let names = input.expression_attribute_names.as_ref(); - validate_name_refs_in_expr(filter_expr, names, ExpressionKind::Filter)?; - } - // Parse ProjectionExpression or desugar legacy AttributesToGet let (effective_projection_str, extra_proj_names) = if input.projection_expression.is_some() { (input.projection_expression.clone(), HashMap::new()) @@ -393,8 +233,10 @@ pub async fn handle_query( // Validate unused expression attributes { let exprs: Vec<&extenddb_core::expression::Expr> = filter.iter().collect(); - let (mut kc_names, kc_values) = - extenddb_core::expression::collect_key_condition_refs(&key_condition); + let (mut kc_names, kc_values) = match parsed_key_condition { + Some(ref kc) => extenddb_core::expression::collect_key_condition_refs(kc), + None => (HashSet::new(), HashSet::new()), + }; // Collect #name refs from projection paths if let Some(ref proj) = projection { for path in proj { @@ -417,6 +259,177 @@ pub async fn handle_query( )?; } + // P118: Fetch key_info first so we can use table_id for index lookup. + let key_info = ctx + .table_key_info(&input.table_name) + .await + .map_err(storage_err_to_dynamo)?; + + // GSI/LSI: resolve index metadata if querying a secondary index. + // Uses table_id from pre-fetched key_info to skip redundant table lookup (P118 #4). + // A vector index is not a row in the `indexes` catalog, so a not-found + // result is re-resolved against the vector index metadata before the + // name is treated as absent. + let (index_info, vector_index_named) = match input.index_name { + Some(ref idx_name) => { + match ctx + .storage + .index_info_by_table_id(&key_info.table_id, idx_name) + .await + { + // Defense in depth: no in-tree backend stores a vector index in + // `indexes`, but if one ever surfaces here it must be refused, + // not sent down the GSI/LSI data path. + Ok(info) if info.index_type == IndexType::Vector => (None, true), + Ok(info) => (Some(info), false), + Err(err @ StorageError::IndexNotFound(_)) => { + match classify_unresolved_index_read(ctx, &key_info, idx_name).await? { + VectorIndexReadRefusal::NotFound => { + return Err(storage_err_to_dynamo(err)); + } + // Query refuses a backfilling vector index with the same + // message as an active one (measured 2026-08-20; Scan + // differs). The refusal itself fires further down, after + // the KeyConditionExpression checks. + VectorIndexReadRefusal::Backfilling + | VectorIndexReadRefusal::NotSupported => (None, true), + } + } + Err(err) => return Err(storage_err_to_dynamo(err)), + } + } + None => (None, false), + }; + + // ConsistentRead is not supported on GSI queries (tenet 1: fidelity). + // Measured 2026-08-20: it fires for a vector index too, with the same + // wording, and before the vector-index refusal. + if input.consistent_read == Some(true) + && (vector_index_named + || index_info + .as_ref() + .is_some_and(|idx| idx.index_type == IndexType::Gsi)) + { + return Err(DynamoDbError::ValidationException( + "Consistent reads are not supported on global secondary indexes".to_owned(), + )); + } + + // Select=ALL_ATTRIBUTES requires an ALL-projection GSI (shared with Scan). + if let Some(ref idx) = index_info { + extenddb_core::validation::validate_all_attributes_index_support( + input.select, + idx.index_type == IndexType::Gsi, + idx.projection.projection_type == ProjectionType::All, + &idx.index_name, + )?; + } + + // For index queries, build a key_info that reflects the index's key schema + // so the storage layer uses the correct SK column for the index table. + let query_key_info = if let Some(ref idx) = index_info { + TableKeyInfo { + table_name: key_info.table_name.clone(), + account_id: key_info.account_id.clone(), + table_id: key_info.table_id.clone(), + key_schema: idx.key_schema.clone(), + base_key_schema: key_info.key_schema.clone(), + attribute_definitions: key_info.attribute_definitions.clone(), + has_lsi: key_info.has_lsi, + global_secondary_indexes: key_info.global_secondary_indexes.clone(), + local_secondary_indexes: key_info.local_secondary_indexes.clone(), + stream_specification: None, // Queries don't capture stream records + vector_indexes: key_info.vector_indexes.clone(), + } + } else { + key_info.clone() + }; + + // The key condition: parsed above from KeyConditionExpression, or + // desugared here from legacy KeyConditions, which needs the key schema. + let (mut key_condition, legacy_kc_maps) = match parsed_key_condition { + Some(parsed) => (parsed, None), + None => { + let Some(ref kc) = input.key_conditions else { + return Err(DynamoDbError::ValidationException( + "Either the KeyConditions or KeyConditionExpression parameter must be specified in the request." + .to_owned(), + )); + }; + let key_schema_pairs: Vec<(String, bool)> = query_key_info + .key_schema + .iter() + .map(|ks| (ks.attribute_name.clone(), ks.key_type == KeyType::Hash)) + .collect(); + let (kc_parsed, kc_maps) = desugar_key_conditions(kc, &key_schema_pairs)?; + (kc_parsed, Some(kc_maps)) + } + }; + + // A vector index is searched only via the vector search API, never + // queried. Measured 2026-08-20: the service refuses after the + // KeyConditionExpression presence and syntax checks, hence below the parse. + if vector_index_named { + return Err(DynamoDbError::ValidationException( + VECTOR_INDEX_QUERY_NOT_SUPPORTED.to_owned(), + )); + } + + // Use legacy maps for key condition resolution if KeyConditions was used + let effective_maps = if let Some(ref kc_maps) = legacy_kc_maps { + kc_maps + } else { + &maps + }; + + // Correct PK/SK assignment when both clauses are equality comparisons. + // The parser can't distinguish PK from SK without the key schema. + let pk_attr = &query_key_info.key_schema[0].attribute_name; + key_condition.resolve_pk_sk(pk_attr, &effective_maps.names)?; + + // Validate that the partition key is actually referenced in the condition. + let pk_resolved = resolve_path_attr_name(&key_condition.pk_path, &effective_maps.names); + if pk_resolved.as_deref() != Some(pk_attr.as_str()) { + return Err(DynamoDbError::ValidationException(format!( + "Query condition missed key schema element: {pk_attr}" + ))); + } + + // For multi-part key schemas (GSIs with >1 HASH attribute), reclassify + // the parsed conditions so all HASH attributes go to pk_path/extra_pk_conditions + // and the RANGE condition stays as sk_condition. + if extenddb_core::types::is_multipart_key_schema(&query_key_info.key_schema) { + let hash_elements = extenddb_core::types::hash_key_elements(&query_key_info.key_schema); + let hash_attrs: Vec<&str> = hash_elements + .iter() + .map(|ks| ks.attribute_name.as_str()) + .collect(); + key_condition.resolve_multipart(&hash_attrs, &effective_maps.names)?; + + // Validate all HASH attributes are present in the KeyConditionExpression. + let provided_count = 1 + key_condition.extra_pk_conditions.len(); + if provided_count != hash_attrs.len() { + // Find the first missing HASH attribute for the error message. + let missing = hash_attrs + .iter() + .find(|attr| { + let pk_name = + resolve_path_attr_name(&key_condition.pk_path, &effective_maps.names); + if pk_name.as_deref() == Some(*attr) { + return false; + } + !key_condition.extra_pk_conditions.iter().any(|(path, _)| { + resolve_path_attr_name(path, &effective_maps.names).as_deref() + == Some(*attr) + }) + }) + .unwrap_or(&hash_attrs[0]); + return Err(DynamoDbError::ValidationException(format!( + "Query condition missed key schema element: {missing}" + ))); + } + } + // Validate Select vs ProjectionExpression and index requirements (shared // with Scan so both reject the same combinations identically). extenddb_core::validation::validate_select_projection( @@ -461,16 +474,6 @@ pub async fn handle_query( ExpressionMaps::new(names, values) }; - // Validate begins_with operand types upfront (before any rows are read). - if let Some(ref f) = filter { - extenddb_core::expression::validate_begins_with_operands(f, &combined_maps).map_err( - |e| crate::expression_helpers::prefix_expression_error(e, ExpressionKind::Filter), - )?; - extenddb_core::expression::validate_ordering_operand_types(f, &combined_maps).map_err( - |e| crate::expression_helpers::prefix_expression_error(e, ExpressionKind::Filter), - )?; - } - // Validate ExclusiveStartKey matches the key schema if let Some(ref start_key) = input.exclusive_start_key { validate_query_exclusive_start_key(start_key, &key_info, index_info.as_ref())?; @@ -574,67 +577,6 @@ fn resolve_path_attr_name( } } -/// Validate that all `#name` references in an expression AST are defined. -fn validate_name_refs_in_expr( - expr: &extenddb_core::expression::Expr, - names: Option<&HashMap>, - expr_type: ExpressionKind, -) -> Result<(), DynamoDbError> { - use extenddb_core::expression::Expr; - match expr { - Expr::Path(elements) => { - for el in elements { - if let PathElement::Attribute(name) = el - && let Some(ref_name) = name.strip_prefix('#') - { - let key_with_hash = format!("#{ref_name}"); - let defined = names.as_ref().is_some_and(|m| { - m.contains_key(ref_name) || m.contains_key(key_with_hash.as_str()) - }); - if !defined { - return Err(DynamoDbError::ValidationException(format!( - "Invalid {expr_type}: An expression attribute name used in the document path is not defined; attribute name: #{ref_name}" - ))); - } - } - } - Ok(()) - } - Expr::Compare { left, right, .. } => { - validate_name_refs_in_expr(left, names, expr_type)?; - validate_name_refs_in_expr(right, names, expr_type) - } - Expr::And(l, r) | Expr::Or(l, r) => { - validate_name_refs_in_expr(l, names, expr_type)?; - validate_name_refs_in_expr(r, names, expr_type) - } - Expr::Not(inner) => validate_name_refs_in_expr(inner, names, expr_type), - Expr::Between { operand, low, high } => { - validate_name_refs_in_expr(operand, names, expr_type)?; - validate_name_refs_in_expr(low, names, expr_type)?; - validate_name_refs_in_expr(high, names, expr_type) - } - Expr::In { operand, list } => { - validate_name_refs_in_expr(operand, names, expr_type)?; - for item in list { - validate_name_refs_in_expr(item, names, expr_type)?; - } - Ok(()) - } - Expr::Function { args, .. } => { - for arg in args { - validate_name_refs_in_expr(arg, names, expr_type)?; - } - Ok(()) - } - Expr::Arithmetic { left, right, .. } => { - validate_name_refs_in_expr(left, names, expr_type)?; - validate_name_refs_in_expr(right, names, expr_type) - } - Expr::Placeholder(_) => Ok(()), - } -} - /// Reject an explicitly empty `KeyConditionExpression` with DynamoDB's exact /// message. An absent expression (`None`) is allowed here; required-ness is /// enforced later. Only the empty-string case (`Some("")`) is rejected, and it diff --git a/crates/engine/src/read_helpers.rs b/crates/engine/src/read_helpers.rs index 6e3ac7bdd..19b812b67 100755 --- a/crates/engine/src/read_helpers.rs +++ b/crates/engine/src/read_helpers.rs @@ -463,7 +463,8 @@ mod tests { fn a_page_whose_items_all_fail_the_filter_ends_at_the_budget_with_a_key() { let s = store(1_000, 4_000); let filter = - parse_optional_filter(Some("payload = :nomatch"), &LimitsConfig::default()).unwrap(); + parse_optional_filter(Some("payload = :nomatch"), &LimitsConfig::default(), true) + .unwrap(); let mut values = HashMap::new(); values.insert(":nomatch".to_owned(), AttributeValue::S("never".to_owned())); let maps = crate::expression_helpers::build_expression_maps(None, Some(&values)); diff --git a/crates/engine/src/scan.rs b/crates/engine/src/scan.rs index 1c1de2274..c279577f8 100755 --- a/crates/engine/src/scan.rs +++ b/crates/engine/src/scan.rs @@ -18,7 +18,7 @@ use extenddb_storage::error::StorageError; use crate::OperationContext; use crate::capacity_helpers; use crate::create_table::storage_err_to_dynamo; -use crate::expression_helpers::{build_expression_maps, parse_optional_filter}; +use crate::expression_helpers::build_expression_maps; use crate::index_helpers::{ VectorIndexReadRefusal, classify_unresolved_index_read, combined_lek_key_schema, validate_scan_exclusive_start_key, @@ -81,6 +81,64 @@ pub async fn handle_scan( .is_some_and(|a| !a.is_empty()); let has_cond_op = input.conditional_operator.is_some(); + // Limit and Segment/TotalSegments are parameter-level checks the service + // runs ahead of every expression check (measured 2026-10-14: a Scan with + // an undefined filter value and Limit 0 reports the limit; with Segment + // and no TotalSegments it reports TotalSegments). + // Validate Segment/TotalSegments — DynamoDB returns different messages per direction + match (input.segment, input.total_segments) { + (Some(_), None) => { + return Err(DynamoDbError::ValidationException( + "The TotalSegments parameter is required but was not present in the request when Segment parameter is present" + .to_owned(), + )); + } + (None, Some(_)) => { + return Err(DynamoDbError::ValidationException( + "The Segment parameter is required but was not present in the request when parameter TotalSegments is present" + .to_owned(), + )); + } + (Some(seg), Some(total)) => { + if total < 1 { + return Err(DynamoDbError::ValidationException( + "The parameter TotalSegments should be greater than or equal to 1".to_owned(), + )); + } + if total > 1_000_000 { + return Err(DynamoDbError::ValidationException(format!( + "1 validation error detected: Value '{total}' at 'totalSegments' failed to satisfy constraint: Member must have value less than or equal to 1000000" + ))); + } + if seg < 0 { + return Err(DynamoDbError::ValidationException(format!( + "1 validation error detected: Value '{seg}' at 'segment' failed to satisfy constraint: Member must have value greater than or equal to 0" + ))); + } + if seg > 999_999 { + return Err(DynamoDbError::ValidationException(format!( + "1 validation error detected: Value '{seg}' at 'segment' failed to satisfy constraint: Member must have value less than or equal to 999999" + ))); + } + if seg >= total { + return Err(DynamoDbError::ValidationException(format!( + "The Segment parameter is zero-based and must be less than parameter TotalSegments: \ + Segment: {seg} is not less than TotalSegments: {total}" + ))); + } + } + (None, None) => {} + } + + // Validate Limit >= 1 + if let Some(limit) = input.limit + && limit < 1 + { + return Err(DynamoDbError::ValidationException(format!( + "1 validation error detected: Value '{limit}' at 'limit' failed to satisfy constraint: Member must have value greater than or equal to 1" + ))); + } + extenddb_core::validation::validate_no_expression_param_mixing( &[ ("AttributesToGet", has_atg), @@ -120,24 +178,31 @@ pub async fn handle_scan( input.scan_filter.as_ref().map_or(0, HashMap::len), )?; - // Parse FilterExpression or desugar legacy ScanFilter - let (filter, filter_maps) = if let Some(ref sf) = input.scan_filter { - if sf.is_empty() { - ( - parse_optional_filter(input.filter_expression.as_deref(), &ctx.limits)?, - None, - ) - } else { + // Parse FilterExpression or desugar legacy ScanFilter. The expression + // form runs the service's ordered semantic walk (references, functions, + // operand types) here, ahead of the projection, the unused check, and the + // table lookup, so an undefined `:q` is a request error even when no row + // would be evaluated (measured 2026-10-14 on an empty table). + let (filter, filter_maps) = + if let Some(sf) = input.scan_filter.as_ref().filter(|sf| !sf.is_empty()) { let cond_op = input.conditional_operator.unwrap_or_default(); let (expr, fmaps) = desugar_filter(sf, cond_op)?; (Some(expr), Some(fmaps)) - } - } else { - ( - parse_optional_filter(input.filter_expression.as_deref(), &ctx.limits)?, - None, - ) - }; + } else { + match input.filter_expression.as_deref() { + Some(expr) if !expr.is_empty() => ( + Some(crate::expression_helpers::parse_condition_checked( + expr, + &maps, + &ctx.limits, + true, + ExpressionKind::Filter, + )?), + None, + ), + _ => (None, None), + } + }; // Parse ProjectionExpression or desugar legacy AttributesToGet let (effective_projection_str, extra_proj_names) = if input.projection_expression.is_some() { @@ -299,60 +364,6 @@ pub async fn handle_scan( )?; } - // Validate Segment/TotalSegments — DynamoDB returns different messages per direction - match (input.segment, input.total_segments) { - (Some(_), None) => { - return Err(DynamoDbError::ValidationException( - "The TotalSegments parameter is required but was not present in the request when Segment parameter is present" - .to_owned(), - )); - } - (None, Some(_)) => { - return Err(DynamoDbError::ValidationException( - "The Segment parameter is required but was not present in the request when parameter TotalSegments is present" - .to_owned(), - )); - } - (Some(seg), Some(total)) => { - if total < 1 { - return Err(DynamoDbError::ValidationException( - "The parameter TotalSegments should be greater than or equal to 1".to_owned(), - )); - } - if total > 1_000_000 { - return Err(DynamoDbError::ValidationException(format!( - "1 validation error detected: Value '{total}' at 'totalSegments' failed to satisfy constraint: Member must have value less than or equal to 1000000" - ))); - } - if seg < 0 { - return Err(DynamoDbError::ValidationException(format!( - "1 validation error detected: Value '{seg}' at 'segment' failed to satisfy constraint: Member must have value greater than or equal to 0" - ))); - } - if seg > 999_999 { - return Err(DynamoDbError::ValidationException(format!( - "1 validation error detected: Value '{seg}' at 'segment' failed to satisfy constraint: Member must have value less than or equal to 999999" - ))); - } - if seg >= total { - return Err(DynamoDbError::ValidationException(format!( - "The Segment parameter is zero-based and must be less than parameter TotalSegments: \ - Segment: {seg} is not less than TotalSegments: {total}" - ))); - } - } - (None, None) => {} - } - - // Validate Limit >= 1 - if let Some(limit) = input.limit - && limit < 1 - { - return Err(DynamoDbError::ValidationException(format!( - "1 validation error detected: Value '{limit}' at 'limit' failed to satisfy constraint: Member must have value greater than or equal to 1" - ))); - } - // For index scans, build a key_info that reflects the index's key schema. let scan_key_info = if let Some(ref idx) = index_info { TableKeyInfo { @@ -444,16 +455,6 @@ pub async fn handle_scan( } } - // Validate begins_with operand types upfront (before any rows are scanned). - if let Some(ref f) = filter { - extenddb_core::expression::validate_begins_with_operands(f, &combined_maps).map_err( - |e| crate::expression_helpers::prefix_expression_error(e, ExpressionKind::Filter), - )?; - extenddb_core::expression::validate_ordering_operand_types(f, &combined_maps).map_err( - |e| crate::expression_helpers::prefix_expression_error(e, ExpressionKind::Filter), - )?; - } - // Determine which key schema to use for LastEvaluatedKey extraction. // For index scans the LEK carries the index key and the base table key, // and the same shape addresses each chunk read from storage. diff --git a/crates/engine/src/transact_get_items.rs b/crates/engine/src/transact_get_items.rs index 761213172..5cd49afe4 100755 --- a/crates/engine/src/transact_get_items.rs +++ b/crates/engine/src/transact_get_items.rs @@ -60,34 +60,13 @@ pub async fn handle_transact_get_items( ))); } - // Resolve table key info for each item - let mut seen_keys: HashSet> = HashSet::with_capacity(input.transact_items.len()); - for tgi in &input.transact_items { - let dedup_key = - serde_json::to_vec(&(&tgi.get.table_name, &tgi.get.key)).unwrap_or_default(); - if !seen_keys.insert(dedup_key) { - return Err(DynamoDbError::ValidationException( - "Transaction request cannot include multiple operations on one item".to_owned(), - )); - } - } - - let mut key_infos = Vec::with_capacity(input.transact_items.len()); - for tgi in &input.transact_items { - let key_info = ctx - .storage - .table_key_info(&ctx.account_id, &tgi.get.table_name) - .await - .map_err(storage_err_to_dynamo)?; - // Key type validation is deferred to the storage layer so that - // mismatches produce TransactionCanceledException with ValidationError - // cancellation reasons, matching real DynamoDB behavior. - key_infos.push(key_info); - } - - // Validate and compile each projection before the transactional read, so a - // malformed request fails with ValidationException instead of executing the - // transaction. DynamoDB checks overlap before the unused-name check. + // Validation runs in the order the service applies it, each phase over + // every item before the next phase starts: projection grammar and alias + // resolution (top-level ValidationException), then table lookup + // (ResourceNotFoundException), then key schema and key size + // (TransactionCanceledException with a ValidationError reason per + // offending item), then duplicate keys (top-level ValidationException). + // Measured against DynamoDB for #108. let mut projections: Vec> = Vec::with_capacity(input.transact_items.len()); for tgi in &input.transact_items { if let Some(ref proj_str) = tgi.get.projection_expression { @@ -120,26 +99,61 @@ pub async fn handle_transact_get_items( } } - // Reject oversized primary keys as a per-item cancellation, matching real - // DynamoDB (TransactionCanceledException with a ValidationError reason for - // the offending item). Key type/emptiness is validated in the storage - // layer; this enforces the size limit the storage layer does not check. - { - let mut reasons: Vec = Vec::with_capacity(input.transact_items.len()); - let mut any_oversized = false; - for (tgi, ki) in input.transact_items.iter().zip(key_infos.iter()) { - match validation::validate_key_size_limits(&tgi.get.key, &ki.key_schema, &ctx.limits) { - Ok(()) => reasons.push(CancellationReason::none()), - Err(e) => { - any_oversized = true; - reasons.push(CancellationReason::validation_error(e.to_string())); + let mut key_infos = Vec::with_capacity(input.transact_items.len()); + for tgi in &input.transact_items { + let key_info = ctx + .storage + .table_key_info(&ctx.account_id, &tgi.get.table_name) + .await + .map_err(storage_err_to_dynamo)?; + key_infos.push(key_info); + } + + // A key with the wrong type, extra or missing attributes, or over the size + // limit cancels the transaction with a ValidationError reason on the FIRST + // offending item and None on every other, even when several keys fail + // (measured 2026-10-14: items 1 and 3 mistyped report + // [ValidationError, None, None]). + let first_offender = input + .transact_items + .iter() + .zip(key_infos.iter()) + .enumerate() + .find_map(|(i, (tgi, ki))| { + validation::validate_batch_key_only( + &tgi.get.key, + &ki.key_schema, + &ki.attribute_definitions, + ) + .and_then(|()| { + validation::validate_key_size_limits(&tgi.get.key, &ki.key_schema, &ctx.limits) + }) + .err() + .map(|e| (i, e)) + }); + if let Some((index, err)) = first_offender { + let reasons: Vec = (0..input.transact_items.len()) + .map(|i| { + if i == index { + CancellationReason::validation_error(err.to_string()) + } else { + CancellationReason::none() } - } - } - if any_oversized { - return Err(storage_err_to_dynamo(StorageError::TransactionCanceled( - reasons, - ))); + }) + .collect(); + return Err(storage_err_to_dynamo(StorageError::TransactionCanceled( + reasons, + ))); + } + + let mut seen_keys: HashSet> = HashSet::with_capacity(input.transact_items.len()); + for tgi in &input.transact_items { + let dedup_key = + serde_json::to_vec(&(&tgi.get.table_name, &tgi.get.key)).unwrap_or_default(); + if !seen_keys.insert(dedup_key) { + return Err(DynamoDbError::ValidationException( + "Transaction request cannot include multiple operations on one item".to_owned(), + )); } } @@ -163,8 +177,7 @@ pub async fn handle_transact_get_items( // Capacity metering: RCU rounded per item, then summed (M-1). // Capacity metering: TransactGetItems costs 2 RCU per item (transactions // double the read cost). Missing items still cost 2 RCU. - let mut per_table_rcu: std::collections::HashMap = - std::collections::HashMap::new(); + let mut per_table_rcu = capacity_helpers::PerTableCapacity::default(); let rcu: f64 = items .iter() .zip(input.transact_items.iter()) @@ -174,7 +187,7 @@ pub async fn handle_transact_get_items( None => 1.0, // minimum 1 RCU for missing item }; let txn_rcu = base_rcu * 2.0; // transactions cost 2x - *per_table_rcu.entry(tgi.get.table_name.clone()).or_default() += txn_rcu; + per_table_rcu.add(&tgi.get.table_name, txn_rcu); txn_rcu }) .sum(); @@ -200,7 +213,7 @@ pub async fn handle_transact_get_items( let consumed_capacity = capacity_helpers::transact_read_capacity( input.return_consumed_capacity, - per_table_rcu.iter().map(|(t, cu)| (t.as_str(), *cu)), + per_table_rcu.iter(), ); let output = TransactGetItemsOutput { diff --git a/crates/engine/src/transact_write_helpers.rs b/crates/engine/src/transact_write_helpers.rs index 8295ef6db..3a2bcfda6 100755 --- a/crates/engine/src/transact_write_helpers.rs +++ b/crates/engine/src/transact_write_helpers.rs @@ -206,28 +206,46 @@ impl PreparedOp { capacity_helpers::item_metrics(ricm, &key_info.key_schema, item_or_key, key_info.has_lsi) } - /// Return a `ValidationError` cancellation reason if this op's primary key - /// exceeds the size limit (hash > 2048 / range > 1024 bytes by default). + /// The per-item cancellation reason for a key that does not fit the + /// table's key schema, or `None` when the key is acceptable. /// - /// Real `DynamoDB` cancels the whole transaction with a per-item - /// `ValidationError` reason for an oversized key in ANY sub-op (Put, - /// Delete, Update, ConditionCheck). Empty-key values are handled separately - /// as a top-level `ValidationException`, so this checks size only. - pub(crate) fn oversized_key_reason( + /// Covers a key attribute of the wrong type, a `Put` item missing a key + /// attribute, a `Delete`, `Update`, or `ConditionCheck` key with extra or + /// missing attributes, and a hash or range key over its size limit. The + /// service reports all of these as a `ValidationError` cancellation reason + /// on the offending item. The reason text follows the service: a `Put` + /// item keeps the single-item wording ("Type mismatch for key ..."), a + /// `Delete`, `Update`, or `ConditionCheck` key gets "The provided key + /// element does not match the schema". An empty key value is a top-level + /// `ValidationException` and is checked before this runs. + pub(crate) fn key_schema_reason( &self, limits: &extenddb_core::limits::LimitsConfig, ) -> Option { + use extenddb_core::validation::{ + validate_batch_key_only, validate_item_keys, validate_key_size_limits, + }; let (key_info, item_or_key) = match self { Self::Put { key_info, item, .. } => (key_info, item), Self::Delete { key_info, key, .. } | Self::Update { key_info, key, .. } | Self::ConditionCheck { key_info, key, .. } => (key_info, key), }; - match extenddb_core::validation::validate_key_size_limits( - item_or_key, - &key_info.key_schema, - limits, - ) { + let schema_check = match self { + Self::Put { .. } => validate_item_keys( + item_or_key, + &key_info.key_schema, + &key_info.attribute_definitions, + ), + _ => validate_batch_key_only( + item_or_key, + &key_info.key_schema, + &key_info.attribute_definitions, + ), + }; + let outcome = schema_check + .and_then(|()| validate_key_size_limits(item_or_key, &key_info.key_schema, limits)); + match outcome { Ok(()) => None, Err(e) => Some(extenddb_core::types::CancellationReason::validation_error( e.to_string(), diff --git a/crates/engine/src/transact_write_items.rs b/crates/engine/src/transact_write_items.rs index cc4a8951b..cbda1b6f2 100755 --- a/crates/engine/src/transact_write_items.rs +++ b/crates/engine/src/transact_write_items.rs @@ -10,7 +10,7 @@ use serde_json::Value; use crate::OperationContext; use crate::capacity_helpers; use crate::create_table::storage_err_to_dynamo; -use crate::expression_helpers::{build_expression_maps, parse_optional_condition}; +use crate::expression_helpers::build_expression_maps; use crate::serialize_output; use crate::stream_capture; use crate::transact_write_helpers::{ @@ -91,23 +91,32 @@ pub async fn handle_transact_write_items( } } - // Resolve table key info, validate inputs, parse expressions, and check - // for duplicate targets — all in a single pass to avoid redundant - // `table_key_info` lookups. + // Validation runs in the order the service applies it, each phase over + // every item before the next phase starts: + // 1. per item, in order: expression grammar, the semantic walk, unused + // names and values, then the item's size (top-level ValidationException), + // 2. table lookup (ResourceNotFoundException), + // 3. the 4 MB request total (top-level ValidationException), + // 4. key schema and key size (TransactionCanceledException with a + // ValidationError reason on the first offending item and None on + // every other; conditions are not evaluated when any item fails here), + // 5. duplicate targets (top-level ValidationException). + // Measured against DynamoDB for #108; a request with two defects reports + // the earlier phase's error whichever item carries it. + let parsed: Vec = input + .transact_items + .iter() + .map(|twi| parse_write_expressions(twi, &ctx.limits)) + .collect::>()?; + let mut prepared: Vec = Vec::with_capacity(input.transact_items.len()); - let mut seen_targets: HashSet = HashSet::with_capacity(input.transact_items.len()); - for twi in &input.transact_items { - let op = prepare_write_op(twi, ctx).await?; - let target_key = op.canonical_target(); - if !seen_targets.insert(target_key) { - return Err(DynamoDbError::ValidationException( - "Transaction request cannot include multiple operations on one item".to_owned(), - )); - } - prepared.push(op); + for (twi, exprs) in input.transact_items.iter().zip(parsed) { + prepared.push(prepare_write_op(twi, exprs, ctx).await?); } - // Validate total transaction size <= 4MB + // The 4 MB request total comes after every table is looked up and before + // the key schema and duplicate checks (measured: eleven 390 KB items plus + // a mistyped key, a duplicate, or a false condition all report the size). let total_size: usize = prepared .iter() .map(super::transact_write_helpers::PreparedOp::item_size) @@ -118,27 +127,39 @@ pub async fn handle_transact_write_items( )); } - // Reject oversized primary keys as a per-item cancellation, matching real - // DynamoDB: an oversized hash/range key in any sub-op returns - // TransactionCanceledException with a ValidationError cancellation reason - // for the offending item (an EMPTY key value, by contrast, is a top-level - // ValidationException and is handled in prepare_write_op). + // Key schema preflight: a key whose type does not match the schema, or a + // key over the size limit, cancels the transaction with a ValidationError + // reason on the FIRST offending item and None on every other item, even + // when several keys fail (measured: items 1 and 3 mistyped report + // [ValidationError, None, None]). (An EMPTY key value, by contrast, is a + // top-level ValidationException and is handled in prepare_write_op.) + // Running this before duplicate detection and before any condition is + // evaluated matches the service: the other items report None, not + // ConditionalCheckFailed. + if let Some((index, reason)) = prepared + .iter() + .enumerate() + .find_map(|(i, op)| op.key_schema_reason(&ctx.limits).map(|r| (i, r))) { - let mut reasons: Vec = - Vec::with_capacity(prepared.len()); - let mut any_oversized = false; - for op in &prepared { - match op.oversized_key_reason(&ctx.limits) { - Some(reason) => { - any_oversized = true; - reasons.push(reason); + let reasons: Vec = (0..prepared.len()) + .map(|i| { + if i == index { + reason.clone() + } else { + extenddb_core::types::CancellationReason::none() } - None => reasons.push(extenddb_core::types::CancellationReason::none()), - } - } - if any_oversized { - return Err(storage_err_to_dynamo( - extenddb_storage::error::StorageError::TransactionCanceled(reasons), + }) + .collect(); + return Err(storage_err_to_dynamo( + extenddb_storage::error::StorageError::TransactionCanceled(reasons), + )); + } + + let mut seen_targets: HashSet = HashSet::with_capacity(prepared.len()); + for op in &prepared { + if !seen_targets.insert(op.canonical_target()) { + return Err(DynamoDbError::ValidationException( + "Transaction request cannot include multiple operations on one item".to_owned(), )); } } @@ -191,18 +212,18 @@ pub async fn handle_transact_write_items( extenddb_core::types::ReturnConsumedCapacity::None => None, rcc => { let indexes = rcc == extenddb_core::types::ReturnConsumedCapacity::Indexes; - let mut per_table_rcu: HashMap = HashMap::new(); + let mut per_table_rcu = capacity_helpers::PerTableCapacity::default(); for op in &prepared { let rcu = capacity_helpers::read_capacity_units(op.write_bytes(), true) * 2.0; - *per_table_rcu.entry(op.table_name().to_owned()).or_default() += rcu; + per_table_rcu.add(op.table_name(), rcu); } Some( per_table_rcu .iter() .map(|(t, cu)| { extenddb_core::types::ConsumedCapacity::transact_replay_read( - t, *cu, indexes, + t, cu, indexes, ) }) .collect(), @@ -228,12 +249,12 @@ pub async fn handle_transact_write_items( // Stream records are now captured atomically within the storage transaction. // Per-item WCU: round each item individually, then sum (M-1). - let mut per_table_wcu: HashMap = HashMap::new(); + let mut per_table_wcu = capacity_helpers::PerTableCapacity::default(); let wcu: f64 = prepared .iter() .map(|op| { let item_wcu = capacity_helpers::write_capacity_units(op.write_bytes()) * 2.0; // transactions cost 2x - *per_table_wcu.entry(op.table_name().to_owned()).or_default() += item_wcu; + per_table_wcu.add(op.table_name(), item_wcu); item_wcu }) .sum(); @@ -245,10 +266,13 @@ pub async fn handle_transact_write_items( // only for now. let consumed_capacity = capacity_helpers::transact_write_capacity( input.return_consumed_capacity, - per_table_wcu.iter().map(|(t, cu)| (t.as_str(), *cu)), + per_table_wcu.iter(), ); - // Collect ItemCollectionMetrics per table for write operations. + // Collect ItemCollectionMetrics per table for write operations. When SIZE + // was requested the member is present even with nothing to report (no + // table has a local secondary index): the service returns `{}` there + // (measured 2026-10-14), and omits the member only when it was not asked for. let mut all_icm: HashMap> = HashMap::new(); for p in &prepared { @@ -259,14 +283,14 @@ pub async fn handle_transact_write_items( .push(m); } } + let item_collection_metrics = match input.return_item_collection_metrics { + extenddb_core::types::ReturnItemCollectionMetrics::None => None, + extenddb_core::types::ReturnItemCollectionMetrics::Size => Some(all_icm), + }; let output = TransactWriteItemsOutput { consumed_capacity, - item_collection_metrics: if all_icm.is_empty() { - None - } else { - Some(all_icm) - }, + item_collection_metrics, }; let body = serialize_output(&output)?; Ok(DispatchResult { @@ -357,11 +381,120 @@ fn assigned_vector_attributes(actions: &[UpdateAction], maps: &ExpressionMaps) - assigned } -/// Parse and validate a single `TransactWriteItem`, returning a `PreparedOp`. +/// A `TransactWriteItem`'s expressions, parsed and checked without touching +/// storage: grammar, operand types, every `#name` and `:value` reference +/// defined, and no unused names or values. +struct ParsedExpressions { + maps: ExpressionMaps, + condition: Option, + actions: Option>, +} + +/// Phase 1 of `TransactWriteItems` validation: the expressions of one item. +/// +/// Runs for every item before any table is looked up, so a malformed +/// expression on item three is reported ahead of a missing table on item one. +fn parse_write_expressions( + twi: &TransactWriteItem, + limits: &extenddb_core::limits::LimitsConfig, +) -> Result { + use extenddb_core::expression::ExpressionKind; + + let (names, values, condition_expression, update_expression) = if let Some(put) = &twi.put { + ( + put.expression_attribute_names.as_ref(), + put.expression_attribute_values.as_ref(), + put.condition_expression.as_deref(), + None, + ) + } else if let Some(del) = &twi.delete { + ( + del.expression_attribute_names.as_ref(), + del.expression_attribute_values.as_ref(), + del.condition_expression.as_deref(), + None, + ) + } else if let Some(upd) = &twi.update { + ( + upd.expression_attribute_names.as_ref(), + upd.expression_attribute_values.as_ref(), + upd.condition_expression.as_deref(), + Some(upd.update_expression.as_str()), + ) + } else if let Some(cc) = &twi.condition_check { + ( + cc.expression_attribute_names.as_ref(), + cc.expression_attribute_values.as_ref(), + Some(cc.condition_expression.as_str()), + None, + ) + } else { + // Unreachable: the one-operation check above already rejected this item. + return Err(DynamoDbError::ValidationException( + "TransactWriteItem must contain exactly one operation".to_owned(), + )); + }; + + // Inside a transaction the service checks an Update item's + // UpdateExpression (syntax, the ordered semantic walk, overlapping paths) + // before its ConditionExpression, the reverse of UpdateItem; then the + // unused names and values; then the item's size (measured 2026-10-14: + // an oversize Put with an unused value reports the value, and an + // oversize item one reports its size ahead of the next item's defects). + let maps = build_expression_maps(names, values); + let actions = match update_expression { + Some(expr) => Some(crate::expression_helpers::parse_update_expr_checked( + expr, &maps, limits, + )?), + None => None, + }; + let condition = match condition_expression { + Some(expr) if !expr.is_empty() => Some(crate::expression_helpers::parse_condition_checked( + expr, + &maps, + limits, + true, + ExpressionKind::Condition, + )?), + _ => None, + }; + // Transactions accept names/values with no expression (unlike single-item + // APIs); only check for unused refs when some expression is present. + if condition.is_some() || actions.is_some() { + let exprs: Vec<&Expr> = condition.iter().collect(); + let action_refs: Vec<&UpdateAction> = actions.iter().flatten().collect(); + extenddb_core::expression::validate_unused_attributes( + &maps.names, + &maps.values, + &exprs, + &action_refs, + &HashSet::new(), + &HashSet::new(), + )?; + } + if let Some(put) = &twi.put { + validate_item_size(&put.item, limits.max_item_size_bytes)?; + } + Ok(ParsedExpressions { + maps, + condition, + actions, + }) +} + +/// Phase 2 of `TransactWriteItems` validation: look the table up and validate +/// the item or key against it, returning a `PreparedOp`. async fn prepare_write_op( twi: &TransactWriteItem, + exprs: ParsedExpressions, ctx: &OperationContext, ) -> Result { + let ParsedExpressions { + maps, + condition, + actions, + } = exprs; + if let Some(put) = &twi.put { let key_info = ctx .storage @@ -369,37 +502,8 @@ async fn prepare_write_op( .await .map_err(storage_err_to_dynamo)?; validate_item_nesting_depth(&put.item)?; - validate_item_size(&put.item, ctx.limits.max_item_size_bytes)?; validate_attribute_name_sizes(&put.item, &ctx.limits)?; validate_key_not_empty(&put.item, &key_info.key_schema)?; - let maps = build_expression_maps( - put.expression_attribute_names.as_ref(), - put.expression_attribute_values.as_ref(), - ); - let condition = parse_optional_condition(put.condition_expression.as_deref(), &ctx.limits)?; - if let Some(ref expr) = condition { - extenddb_core::expression::validate_ordering_operand_types(expr, &maps).map_err( - |e| { - crate::expression_helpers::prefix_expression_error( - e, - extenddb_core::expression::ExpressionKind::Condition, - ) - }, - )?; - } - // Transactions accept names/values with no expression (unlike single-item - // APIs); only check for unused refs when a condition is present. - if condition.is_some() { - let exprs: Vec<&extenddb_core::expression::Expr> = condition.iter().collect(); - extenddb_core::expression::validate_unused_attributes( - &maps.names, - &maps.values, - &exprs, - &[], - &HashSet::new(), - &HashSet::new(), - )?; - } let stream = stream_capture::stream_view_type(&key_info).map(|vt| extenddb_storage::StreamCapture { view_type: vt, @@ -426,32 +530,6 @@ async fn prepare_write_op( // DynamoDB (a top-level ValidationException), unlike a key type // mismatch which surfaces as a per-item cancellation reason. validate_key_not_empty(&del.key, &key_info.key_schema)?; - let maps = build_expression_maps( - del.expression_attribute_names.as_ref(), - del.expression_attribute_values.as_ref(), - ); - let condition = parse_optional_condition(del.condition_expression.as_deref(), &ctx.limits)?; - if let Some(ref expr) = condition { - extenddb_core::expression::validate_ordering_operand_types(expr, &maps).map_err( - |e| { - crate::expression_helpers::prefix_expression_error( - e, - extenddb_core::expression::ExpressionKind::Condition, - ) - }, - )?; - } - if condition.is_some() { - let exprs: Vec<&extenddb_core::expression::Expr> = condition.iter().collect(); - extenddb_core::expression::validate_unused_attributes( - &maps.names, - &maps.values, - &exprs, - &[], - &HashSet::new(), - &HashSet::new(), - )?; - } let stream = stream_capture::stream_view_type(&key_info).map(|vt| extenddb_storage::StreamCapture { view_type: vt, @@ -478,12 +556,7 @@ async fn prepare_write_op( // DynamoDB (a top-level ValidationException), unlike a key type // mismatch which surfaces as a per-item cancellation reason. validate_key_not_empty(&upd.key, &key_info.key_schema)?; - let maps = build_expression_maps( - upd.expression_attribute_names.as_ref(), - upd.expression_attribute_values.as_ref(), - ); - let actions = - crate::expression_helpers::parse_update_expr(&upd.update_expression, &ctx.limits)?; + let actions = actions.unwrap_or_default(); validate_no_key_updates(&actions, &key_info, &maps)?; // Validate nesting depth of EAV values that get stored via SET actions. @@ -500,28 +573,6 @@ async fn prepare_write_op( .collect(); validate_attribute_values_nesting_depth(stored)?; } - let condition = parse_optional_condition(upd.condition_expression.as_deref(), &ctx.limits)?; - if let Some(ref expr) = condition { - extenddb_core::expression::validate_ordering_operand_types(expr, &maps).map_err( - |e| { - crate::expression_helpers::prefix_expression_error( - e, - extenddb_core::expression::ExpressionKind::Condition, - ) - }, - )?; - } - { - let exprs: Vec<&extenddb_core::expression::Expr> = condition.iter().collect(); - extenddb_core::expression::validate_unused_attributes( - &maps.names, - &maps.values, - &exprs, - &actions.iter().collect::>(), - &HashSet::new(), - &HashSet::new(), - )?; - } let stream = stream_capture::stream_view_type(&key_info).map(|vt| extenddb_storage::StreamCapture { view_type: vt, @@ -549,31 +600,12 @@ async fn prepare_write_op( // DynamoDB (a top-level ValidationException), unlike a key type // mismatch which surfaces as a per-item cancellation reason. validate_key_not_empty(&cc.key, &key_info.key_schema)?; - let maps = build_expression_maps( - cc.expression_attribute_names.as_ref(), - cc.expression_attribute_values.as_ref(), - ); - let condition = - crate::expression_helpers::parse_condition_expr(&cc.condition_expression, &ctx.limits)?; - extenddb_core::expression::validate_ordering_operand_types(&condition, &maps).map_err( - |e| { - crate::expression_helpers::prefix_expression_error( - e, - extenddb_core::expression::ExpressionKind::Condition, - ) - }, - )?; - { - let exprs: Vec<&extenddb_core::expression::Expr> = vec![&condition]; - extenddb_core::expression::validate_unused_attributes( - &maps.names, - &maps.values, - &exprs, - &[], - &HashSet::new(), - &HashSet::new(), - )?; - } + let Some(condition) = condition else { + // Unreachable: parse_write_expressions parsed the required expression. + return Err(DynamoDbError::ValidationException( + "ConditionCheck requires a ConditionExpression".to_owned(), + )); + }; return Ok(PreparedOp::ConditionCheck { key_info, key: cc.key.clone(), diff --git a/crates/engine/src/update_item.rs b/crates/engine/src/update_item.rs index 7f74c205c..150cae933 100755 --- a/crates/engine/src/update_item.rs +++ b/crates/engine/src/update_item.rs @@ -11,7 +11,7 @@ use std::collections::HashMap; use serde_json::Value; use extenddb_core::error::DynamoDbError; -use extenddb_core::expression::{ExpressionKind, ExpressionMaps, PathElement, UpdateAction}; +use extenddb_core::expression::{ExpressionMaps, PathElement, UpdateAction}; use extenddb_core::types::{ AttributeValue, Item, ReturnValues, TableKeyInfo, UpdateItemInput, UpdateItemOutput, item_size_bytes, @@ -20,7 +20,7 @@ use extenddb_core::types::{ use crate::OperationContext; use crate::capacity_helpers; use crate::create_table::{storage_err_to_dynamo, storage_err_to_dynamo_with_ccf}; -use crate::expression_helpers::resolve_condition; +use crate::expression_helpers::{resolve_condition, single_item_write_prefix}; use crate::serialize_output; use crate::stream_capture; use crate::{DispatchMetrics, DispatchResult}; @@ -123,13 +123,16 @@ pub async fn handle_update_item( }; let has_expr = has_update_expr || has_condition; + // The single-item write APIs name no expression in the values message and + // wrap every expression-stage error in `1 validation error detected: `. extenddb_core::expression::validate_expression_param_usage( input.expression_attribute_names.as_ref(), has_expr, input.expression_attribute_values.as_ref(), has_expr, - &[ExpressionKind::Update, ExpressionKind::Condition], - )?; + &[], + ) + .map_err(single_item_write_prefix)?; // ConditionalOperator requires an Expected with two or more conditions. extenddb_core::validation::validate_conditional_operator_usage( @@ -137,6 +140,10 @@ pub async fn handle_update_item( input.expected.as_ref().map_or(0, HashMap::len), )?; + // Amazon DynamoDB checks the whole ConditionExpression (syntax, then + // references, functions, and operand types) before it looks at the + // UpdateExpression: a request with a defect in each reports the + // condition's (measured 2026-10-14). `resolve_condition` runs that pass. let (condition, maps) = resolve_condition( input.condition_expression.as_deref(), effective_expr_names.as_ref(), @@ -144,33 +151,22 @@ pub async fn handle_update_item( input.expected.as_ref(), input.conditional_operator, &ctx.limits, - )?; + ) + .map_err(single_item_write_prefix)?; // No UpdateExpression and no AttributeUpdates: no-op upsert. - // Some("") still errors via parse_update_expr. + // Some("") still errors via parse_update_expr_checked. The checked parse + // runs the service's ordered walk over the parsed actions (function + // names, operand counts and types, undefined references, repeated + // sections) and then the overlapping-path rule, all ahead of the table + // lookup. let actions = if let Some(update_expr) = effective_update_expr.as_deref() { - crate::expression_helpers::parse_update_expr(update_expr, &ctx.limits)? + crate::expression_helpers::parse_update_expr_checked(update_expr, &maps, &ctx.limits) + .map_err(single_item_write_prefix)? } else { Vec::new() }; - - // Reject an undefined #name before the existence check - // (otherwise it is only raised later, during execution). - for action in &actions { - let path = match action { - UpdateAction::Set { path, .. } - | UpdateAction::Remove { path } - | UpdateAction::Add { path, .. } - | UpdateAction::Delete { path, .. } => path, - }; - for el in path { - if let PathElement::Attribute(name) = el - && let Some(ref_name) = name.strip_prefix('#') - { - maps.resolve_name(ref_name)?; - } - } - } + let action_refs: Vec<&UpdateAction> = actions.iter().collect(); if input.expected.is_none() || input @@ -183,10 +179,11 @@ pub async fn handle_update_item( &maps.names, &maps.values, &exprs, - &actions.iter().collect::>(), + &action_refs, &std::collections::HashSet::new(), &std::collections::HashSet::new(), - )?; + ) + .map_err(single_item_write_prefix)?; } let key_info = ctx diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 79ba14056..a6f82ac9a 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -46,6 +46,7 @@ adaptation when switching between ExtendDB and the real service. |------|----------|------| | Table creation delay | Returns `CREATING` immediately; transitions to `ACTIVE` typically within seconds. Same behavior for on-demand and provisioned | Configurable via `control_plane_delay_seconds` runtime setting (default: 5s) | | DeletionProtectionEnabled | Enforced | Enforced (accepted and stored, DeleteTable rejects when enabled) | +| Table ARN in `TableName` | Every data-plane and table operation accepts the table's ARN (`arn:aws:dynamodb:::table/`) in place of its name | Bare table names only; an ARN is reported as a missing table or an invalid name. An ExtendDB table ARN carries ExtendDB's account and region, so a table that streams or replicates to Amazon DynamoDB would answer to two ARNs for one logical table. Keeping names as the only identifier avoids that collision. Transactions that address one item by name and by ARN are therefore two different (missing) targets here, where the service rejects them as duplicates. | ## Time to Live (TTL) diff --git a/tests/test_expression_error_prefix.py b/tests/test_expression_error_prefix.py index e28fad34f..77186dcf2 100644 --- a/tests/test_expression_error_prefix.py +++ b/tests/test_expression_error_prefix.py @@ -59,7 +59,7 @@ def test_condition_prefix_put(self, dynamodb_client, hash_table): ConditionExpression="status = :v", ExpressionAttributeValues={":v": {"S": "x"}}, )) - assert msg == f"Invalid ConditionExpression: {RESERVED}" + assert msg == f"1 validation error detected: Invalid ConditionExpression: {RESERVED}" def test_condition_prefix_delete(self, dynamodb_client, hash_table): msg = _msg(lambda: dynamodb_client.delete_item( @@ -67,7 +67,7 @@ def test_condition_prefix_delete(self, dynamodb_client, hash_table): ConditionExpression="status = :v", ExpressionAttributeValues={":v": {"S": "x"}}, )) - assert msg == f"Invalid ConditionExpression: {RESERVED}" + assert msg == f"1 validation error detected: Invalid ConditionExpression: {RESERVED}" def test_update_prefix(self, dynamodb_client, hash_table): msg = _msg(lambda: dynamodb_client.update_item( @@ -75,7 +75,7 @@ def test_update_prefix(self, dynamodb_client, hash_table): UpdateExpression="SET status = :v", ExpressionAttributeValues={":v": {"S": "x"}}, )) - assert msg == f"Invalid UpdateExpression: {RESERVED}" + assert msg == f"1 validation error detected: Invalid UpdateExpression: {RESERVED}" def test_key_condition_prefix(self, dynamodb_client, hash_table): msg = _msg(lambda: dynamodb_client.query( diff --git a/tests/test_expression_param_usage.py b/tests/test_expression_param_usage.py index 653f38d04..8d5dde93f 100644 --- a/tests/test_expression_param_usage.py +++ b/tests/test_expression_param_usage.py @@ -5,8 +5,11 @@ Amazon DynamoDB rejects a request that provides ExpressionAttributeNames or ExpressionAttributeValues when there is no expression parameter that could -reference them. The names message is always suffix-free; the values message -names the null value-capable expression parameter(s) for that API. +reference them. The names message is always suffix-free. On the read APIs the +values message names the null value-capable expression parameter(s) for that +API (Scan: "FilterExpression is null"; Query: none). PutItem, UpdateItem, and +DeleteItem give both messages suffix-free and wrapped in +"1 validation error detected: " (measured 2026-10-14). Dual-target against Amazon DynamoDB and extenddb. """ @@ -19,6 +22,7 @@ from conftest import scoped_table NAMES_MSG = "ExpressionAttributeNames can only be specified when using expressions" +SINGLE_ITEM = "1 validation error detected: " VALUES_PREFIX = "ExpressionAttributeValues can only be specified when using expressions" @@ -92,7 +96,7 @@ def test_delete_item_names_no_expression(self, dynamodb_client, hash_table): Key={"pk": {"S": "k1"}}, ExpressionAttributeNames={"#a": "foo"}, ), - NAMES_MSG, + SINGLE_ITEM + NAMES_MSG, ) def test_delete_item_values_no_expression(self, dynamodb_client, hash_table): @@ -102,7 +106,7 @@ def test_delete_item_values_no_expression(self, dynamodb_client, hash_table): Key={"pk": {"S": "k1"}}, ExpressionAttributeValues={":v": {"S": "x"}}, ), - f"{VALUES_PREFIX}: ConditionExpression is null", + SINGLE_ITEM + VALUES_PREFIX, ) def test_update_item_names_no_expression(self, dynamodb_client, hash_table): @@ -112,7 +116,7 @@ def test_update_item_names_no_expression(self, dynamodb_client, hash_table): Key={"pk": {"S": "k1"}}, ExpressionAttributeNames={"#a": "foo"}, ), - NAMES_MSG, + SINGLE_ITEM + NAMES_MSG, ) def test_update_item_values_no_expression(self, dynamodb_client, hash_table): @@ -122,7 +126,7 @@ def test_update_item_values_no_expression(self, dynamodb_client, hash_table): Key={"pk": {"S": "k1"}}, ExpressionAttributeValues={":v": {"S": "x"}}, ), - f"{VALUES_PREFIX}: UpdateExpression and ConditionExpression are null", + SINGLE_ITEM + VALUES_PREFIX, ) def test_put_item_names_no_expression(self, dynamodb_client, hash_table): @@ -132,7 +136,7 @@ def test_put_item_names_no_expression(self, dynamodb_client, hash_table): Item={"pk": {"S": "k2"}}, ExpressionAttributeNames={"#a": "foo"}, ), - NAMES_MSG, + SINGLE_ITEM + NAMES_MSG, ) def test_put_item_values_no_expression(self, dynamodb_client, hash_table): @@ -142,7 +146,7 @@ def test_put_item_values_no_expression(self, dynamodb_client, hash_table): Item={"pk": {"S": "k2"}}, ExpressionAttributeValues={":v": {"S": "x"}}, ), - f"{VALUES_PREFIX}: ConditionExpression is null", + SINGLE_ITEM + VALUES_PREFIX, ) def test_query_names_no_expression(self, dynamodb_client, hash_table): diff --git a/tests/test_expression_semantics.py b/tests/test_expression_semantics.py new file mode 100644 index 000000000..5c565c4c7 --- /dev/null +++ b/tests/test_expression_semantics.py @@ -0,0 +1,681 @@ +# SPDX-License-Identifier: Apache-2.0 + +"""Expression semantics Amazon DynamoDB checks before it reads the table. + +After the grammar pass, the service walks the parsed expression once, in +expression order, checking each node after its operands, and only then +checks overlapping paths and unused names and values. Every expectation +below was captured from the service on 2026-10-14 (issue #108). + +UpdateExpression: + - `+` and `-` take numbers; `list_append` takes lists; `ADD` takes a number + or a set; `DELETE` takes a set. A placeholder operand of the wrong type is + refused with its type: `S`, `N`, `L` for the operators and functions, + `STRING`, `NUMBER`, `Binary`, `BOOL`, `NULL`, `LIST`, `MAP` for ADD and + DELETE. + - Function names are matched as written: `IF_NOT_EXISTS(x, :v)` and + `foo(x)` are `Invalid function name`; `size(x)` is `not allowed in an + update expression`; `list_append` is refused as an operand of a function + or of `+`. + - A repeated section is reported after that section's own actions, so + `SET a = :v SET b = :q` reports `:q` and `SET a = :v SET b = :v ADD n :s` + reports the repeated `SET`. + - Every unused name or value is listed in one set. + - UpdateItem checks the ConditionExpression before the UpdateExpression; + inside TransactWriteItems the UpdateExpression comes first. + +ConditionExpression and FilterExpression: + - the six condition functions are matched as written; `if_not_exists` and + `list_append` are `not allowed in a condition expression`; a boolean + function in operand position and `size` in condition position are `not + allowed to be used this way`; operand counts are fixed; `attribute_exists` + and `attribute_not_exists` take a document path; `begins_with` takes a + string or binary and `attribute_type` a string naming a type. + +Query and Scan: + - the FilterExpression, then the KeyConditionExpression, then the + ProjectionExpression are checked for undefined references and function + names, then unused names and values, all before the table is looked up. + +Key-shaped APIs (GetItem, DeleteItem, UpdateItem, the batch key paths, and a +BatchWriteItem Put) refuse a key with a missing, mistyped, or extra attribute +with `The provided key element does not match the schema`; PutItem names the +attribute. BatchWriteItem returns `ItemCollectionMetrics: {}` when SIZE is +requested and no written table has a local secondary index. + +Dual-target: runs against the service and against ExtendDB on every backend. +""" + +from __future__ import annotations + +import re +import uuid + +import pytest +from botocore.exceptions import ClientError + +from conftest import scoped_table + +P = "1 validation error detected: " +U = "Invalid UpdateExpression: " +C = "Invalid ConditionExpression: " +F = "Invalid FilterExpression: " +K = "Invalid KeyConditionExpression: " +PR = "Invalid ProjectionExpression: " + +UNDEF_VALUE = "An expression attribute value used in expression is not defined; attribute value: {}" +UNDEF_NAME = "An expression attribute name used in the document path is not defined; attribute name: {}" +OPERAND_TYPE = "Incorrect operand type for operator or function; operator or function: {}, operand type: {}" +ADD_DELETE_TYPE = "Incorrect operand type for operator or function; operator: {}, operand type: {}, typeSet: ALLOWED_FOR_ADD_OPERAND" +INVALID_FN = "Invalid function name; function: {}" +NOT_THIS_WAY = "The function is not allowed to be used this way in an expression; function: {}" +NOT_IN_UPDATE = "The function is not allowed in an update expression; function: {}" +NOT_IN_CONDITION = "The function is not allowed in a condition expression; function: {}" +OPERAND_COUNT = "Incorrect number of operands for operator or function; operator or function: {}, number of operands: {}" +NEEDS_PATH = "Operator or function requires a document path; operator or function: {}" +SECTION_ONCE = 'The "{}" section can only be used once in an update expression;' +OVERLAP_A_AB = "Two document paths overlap with each other; must remove or rewrite one of these paths; path one: [a], path two: [a, b]" +EVAL_TYPE = "An operand in the update expression has an incorrect data type" +KEY_SCHEMA = "The provided key element does not match the schema" +EMPTY_ARGS_SYNTAX = 'Syntax error; token: ")", near: "()"' + +N = {"N": "1"} +S = {"S": "x"} +L = {"L": []} +SS = {"SS": ["a"]} +KEY = {"pk": {"S": "p"}} +WRITE_KEY = {"pk": {"S": "w"}} + + +@pytest.fixture(scope="module") +def table(dynamodb_client): + with scoped_table(dynamodb_client) as name: + dynamodb_client.put_item( + TableName=name, + Item={"pk": {"S": "p"}, "n": N, "s": SS, "l": L, "m": {"M": {"x": N}}}, + ) + yield name + + +@pytest.fixture(scope="module") +def range_table(dynamodb_client): + with scoped_table( + dynamodb_client, + attribute_definitions=[ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "S"}, + ], + key_schema=[ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + ) as name: + yield name + + +@pytest.fixture(scope="module") +def lsi_table(dynamodb_client): + with scoped_table( + dynamodb_client, + attribute_definitions=[ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "S"}, + {"AttributeName": "alt", "AttributeType": "S"}, + ], + key_schema=[ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + LocalSecondaryIndexes=[ + { + "IndexName": "byalt", + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "alt", "KeyType": "RANGE"}, + ], + "Projection": {"ProjectionType": "ALL"}, + } + ], + ) as name: + yield name + + +@pytest.fixture(scope="module") +def missing_table() -> str: + return f"extenddb-missing-{uuid.uuid4().hex[:12]}" + + +def _expect(func, message: str, code: str = "ValidationException"): + with pytest.raises(ClientError) as exc_info: + func() + err = exc_info.value.response["Error"] + assert err["Code"] == code, err + assert err["Message"] == message, err + + +def _expect_prefix(func, prefix: str, code: str = "ValidationException"): + """For syntax errors, whose inner wording ExtendDB still phrases its own way.""" + with pytest.raises(ClientError) as exc_info: + func() + err = exc_info.value.response["Error"] + assert err["Code"] == code, err + assert err["Message"].startswith(prefix), err + + +def _expect_unused(func, kind: str, keys: set[str]): + """The service lists every unused key in one set, in an order it does not document.""" + with pytest.raises(ClientError) as exc_info: + func() + err = exc_info.value.response["Error"] + assert err["Code"] == "ValidationException", err + match = re.fullmatch( + rf"(?:{re.escape(P)})?Value provided in ExpressionAttribute{kind} unused in expressions: keys: \{{(.*)\}}", + err["Message"], + ) + assert match, err + assert set(match.group(1).split(", ")) == keys, err + + +def _update(client, table, expression, values=None, names=None, condition=None, key=WRITE_KEY): + kwargs = {"TableName": table, "Key": key, "UpdateExpression": expression} + if values is not None: + kwargs["ExpressionAttributeValues"] = values + if names is not None: + kwargs["ExpressionAttributeNames"] = names + if condition is not None: + kwargs["ConditionExpression"] = condition + return lambda: client.update_item(**kwargs) + + +def _transact_update(client, table, expression, values=None, condition=None): + item = {"TableName": table, "Key": WRITE_KEY, "UpdateExpression": expression} + if values is not None: + item["ExpressionAttributeValues"] = values + if condition is not None: + item["ConditionExpression"] = condition + return lambda: client.transact_write_items(TransactItems=[{"Update": item}]) + + +def _put_cond(client, table, condition, values=None, names=None): + kwargs = {"TableName": table, "Item": WRITE_KEY, "ConditionExpression": condition} + if values is not None: + kwargs["ExpressionAttributeValues"] = values + if names is not None: + kwargs["ExpressionAttributeNames"] = names + return lambda: client.put_item(**kwargs) + + +class TestUpdateOperandTypes: + @pytest.mark.parametrize( + ("operator", "value", "type_name"), + [ + ("ADD", S, "STRING"), + ("ADD", {"B": b"x"}, "Binary"), + ("ADD", {"BOOL": True}, "BOOL"), + ("ADD", {"NULL": True}, "NULL"), + ("ADD", L, "LIST"), + ("ADD", {"M": {}}, "MAP"), + ("DELETE", S, "STRING"), + ("DELETE", N, "NUMBER"), + ("DELETE", {"B": b"x"}, "Binary"), + ("DELETE", {"BOOL": True}, "BOOL"), + ("DELETE", {"NULL": True}, "NULL"), + ("DELETE", L, "LIST"), + ("DELETE", {"M": {}}, "MAP"), + ], + ) + def test_add_and_delete_refuse_the_type_at_request_time(self, dynamodb_client, table, operator, value, type_name): + _expect(_update(dynamodb_client, table, f"{operator} attr :v", {":v": value}), + P + U + ADD_DELETE_TYPE.format(operator, type_name)) + + def test_add_takes_numbers_and_sets_delete_takes_sets(self, dynamodb_client, table): + dynamodb_client.update_item(TableName=table, Key=WRITE_KEY, UpdateExpression="ADD n :n, s :ss", + ExpressionAttributeValues={":n": N, ":ss": SS}) + dynamodb_client.update_item(TableName=table, Key=WRITE_KEY, UpdateExpression="DELETE s :ss", + ExpressionAttributeValues={":ss": SS}) + + def test_set_mismatch_against_the_stored_attribute_is_an_evaluation_error(self, dynamodb_client, table): + # Request-time rules accept a set; the stored attribute decides the rest, + # and that message carries no prefix. + _expect(_update(dynamodb_client, table, "ADD n :ss", {":ss": SS}, key=KEY), EVAL_TYPE) + _expect(_update(dynamodb_client, table, "ADD s :ns", {":ns": {"NS": ["1"]}}, key=KEY), EVAL_TYPE) + + @pytest.mark.parametrize( + ("expression", "values", "symbol", "type_code"), + [ + ("SET a = :s + :n", {":s": S, ":n": N}, "+", "S"), + ("SET a = :n + :s", {":s": S, ":n": N}, "+", "S"), + ("SET a = :s + :s2", {":s": S, ":s2": {"S": "y"}}, "+", "S"), + ("SET a = n + :s", {":s": S}, "+", "S"), + ("SET a = :l - :n", {":l": L, ":n": N}, "-", "L"), + ("SET a = :b + :n", {":b": {"B": b"x"}, ":n": N}, "+", "B"), + ("SET a = :z + :n", {":z": {"NULL": True}, ":n": N}, "+", "NULL"), + ("SET a = :m + :n", {":m": {"M": {}}, ":n": N}, "+", "M"), + ("SET a = :t + :n", {":t": {"BOOL": True}, ":n": N}, "+", "BOOL"), + ("SET a = :ss + :n", {":ss": SS, ":n": N}, "+", "SS"), + ], + ) + def test_arithmetic_operands_must_be_numbers(self, dynamodb_client, table, expression, values, symbol, type_code): + _expect(_update(dynamodb_client, table, expression, values), P + U + OPERAND_TYPE.format(symbol, type_code)) + + def test_arithmetic_on_a_function_result_is_checked_at_evaluation(self, dynamodb_client, table): + _expect(_update(dynamodb_client, table, "SET a = if_not_exists(a, :s) + :n", {":s": S, ":n": N}, key=KEY), + EVAL_TYPE) + + @pytest.mark.parametrize( + ("expression", "values", "type_code"), + [ + ("SET l = list_append(:v, :l)", {":v": S, ":l": L}, "S"), + ("SET l = list_append(:v, :l)", {":v": N, ":l": L}, "N"), + ("SET l = list_append(:v, :l)", {":v": {"M": {}}, ":l": L}, "M"), + ("SET l = list_append(:v, :l)", {":v": SS, ":l": L}, "SS"), + ("SET l = list_append(:v, :l)", {":v": {"BOOL": True}, ":l": L}, "BOOL"), + ("SET l = list_append(:v, :l)", {":v": {"B": b"x"}, ":l": L}, "B"), + ("SET l = list_append(:v, :l)", {":v": {"NULL": True}, ":l": L}, "NULL"), + ("SET l = list_append(:v, :l)", {":v": {"NS": ["1"]}, ":l": L}, "NS"), + ("SET l = list_append(:v, :l)", {":v": {"BS": [b"a"]}, ":l": L}, "BS"), + ("SET l = list_append(:l, :v)", {":v": S, ":l": L}, "S"), + ("SET l = list_append(l, :v)", {":v": S}, "S"), + ("SET l = list_append(:s, :n)", {":s": S, ":n": N}, "S"), + ], + ) + def test_list_append_operands_must_be_lists(self, dynamodb_client, table, expression, values, type_code): + _expect(_update(dynamodb_client, table, expression, values), P + U + OPERAND_TYPE.format("list_append", type_code)) + + def test_list_append_of_a_function_result_is_accepted(self, dynamodb_client, table): + dynamodb_client.update_item(TableName=table, Key=WRITE_KEY, + UpdateExpression="SET l = list_append(if_not_exists(l, :e), :l)", + ExpressionAttributeValues={":e": L, ":l": L}) + + +class TestUpdateFunctionNames: + @pytest.mark.parametrize( + ("expression", "values", "name"), + [ + ("SET x = IF_NOT_EXISTS(x, :v)", {":v": N}, "IF_NOT_EXISTS"), + ("SET l = List_Append(:l, :l)", {":l": L}, "List_Append"), + ("SET x = foo(x, :v)", {":v": N}, "foo"), + ("SET x = ATTRIBUTE_EXISTS(x)", None, "ATTRIBUTE_EXISTS"), + ("SET x = Size(x)", None, "Size"), + ("SET a = if_not_exists(foo(x), :v)", {":v": N}, "foo"), + ("SET a = if_not_exists(a, foo(x))", None, "foo"), + ("SET l = list_append(:l, foo(x))", {":l": L}, "foo"), + ], + ) + def test_unknown_or_miscased_names_are_refused(self, dynamodb_client, table, expression, values, name): + _expect(_update(dynamodb_client, table, expression, values), P + U + INVALID_FN.format(name)) + + @pytest.mark.parametrize("name", ["attribute_exists", "size"]) + def test_condition_functions_are_refused_by_name(self, dynamodb_client, table, name): + _expect(_update(dynamodb_client, table, f"SET x = {name}(x), y = :q"), P + U + NOT_IN_UPDATE.format(name)) + + def test_a_call_with_no_operands_is_a_syntax_error(self, dynamodb_client, table): + _expect(_update(dynamodb_client, table, "SET x = foo()"), P + U + EMPTY_ARGS_SYNTAX) + + @pytest.mark.parametrize( + "expression", + ["SET l = list_append(:l, :l) + :n", "SET l = :n + list_append(:l, :l)"], + ) + def test_list_append_is_refused_as_an_arithmetic_operand(self, dynamodb_client, table, expression): + _expect(_update(dynamodb_client, table, expression, {":l": L, ":n": N}), + P + U + NOT_THIS_WAY.format("list_append")) + + def test_nested_list_append_is_reported_where_it_stands(self, dynamodb_client, table): + # The nested call is refused before its sibling operand is resolved, + # but an undefined reference inside it is found first. + _expect(_update(dynamodb_client, table, "SET l = if_not_exists(list_append(:a, :b), :q)", {":a": L, ":b": L}), + P + U + NOT_THIS_WAY.format("list_append")) + _expect(_update(dynamodb_client, table, "SET l = if_not_exists(list_append(:q, :b), :a)", {":a": L, ":b": L}), + P + U + UNDEF_VALUE.format(":q")) + + +class TestUpdateWalkOrder: + def test_references_in_expression_order(self, dynamodb_client, table): + _expect(_update(dynamodb_client, table, "SET a = :q, #n = :v", {":v": N}), P + U + UNDEF_VALUE.format(":q")) + _expect(_update(dynamodb_client, table, "SET #n = :q"), P + U + UNDEF_NAME.format("#n")) + + @pytest.mark.parametrize( + ("expression", "values", "message"), + [ + ("SET a = :q SET b = :v", {":v": N}, UNDEF_VALUE.format(":q")), + ("SET a = :v SET b = :q", {":v": N}, UNDEF_VALUE.format(":q")), + ("SET #n = :v SET b = :v", {":v": N}, UNDEF_NAME.format("#n")), + ("SET a = :v REMOVE #n SET b = :v", {":v": N}, UNDEF_NAME.format("#n")), + ("SET a = :v SET b = :v ADD n :s", {":v": N, ":s": S}, SECTION_ONCE.format("SET")), + ("SET a = :v SET b = :v ADD n :q", {":v": N}, SECTION_ONCE.format("SET")), + ("SET a = :v ADD n :s SET b = :v", {":v": N, ":s": S}, ADD_DELETE_TYPE.format("ADD", "STRING")), + ("ADD n :s ADD m :n", {":s": S, ":n": N}, ADD_DELETE_TYPE.format("ADD", "STRING")), + ("DELETE s :ss DELETE t :s", {":s": S, ":ss": SS}, ADD_DELETE_TYPE.format("DELETE", "STRING")), + ("SET a = :n SET b = :s + :n", {":s": S, ":n": N}, OPERAND_TYPE.format("+", "S")), + ("SET a = :v SET x = foo(x)", {":v": N}, INVALID_FN.format("foo")), + ("SET l = list_append(list_append(:a, :a), :a) SET b = :v", {":v": N, ":a": L}, NOT_THIS_WAY.format("list_append")), + ], + ) + def test_repeated_section_is_reported_after_its_own_actions(self, dynamodb_client, table, expression, values, message): + _expect(_update(dynamodb_client, table, expression, values), P + U + message) + + @pytest.mark.parametrize( + ("expression", "values", "message"), + [ + ("SET a = :q ADD n :s", {":s": S}, UNDEF_VALUE.format(":q")), + ("ADD n :s SET a = if_not_exists(a)", {":s": S}, ADD_DELETE_TYPE.format("ADD", "STRING")), + ("SET a = if_not_exists(a) ADD n :s", {":s": S}, OPERAND_COUNT.format("if_not_exists", 1)), + ("SET a = if_not_exists(a, :q, :q2)", None, UNDEF_VALUE.format(":q")), + ("SET a = if_not_exists(a), b = :q", None, OPERAND_COUNT.format("if_not_exists", 1)), + ("SET a = if_not_exists(:a)", {":a": N}, OPERAND_COUNT.format("if_not_exists", 1)), + ("SET l = list_append(:s)", {":s": S}, OPERAND_COUNT.format("list_append", 1)), + ("SET x = foo(x), y = :q", None, INVALID_FN.format("foo")), + ("SET y = :q, x = foo(x)", None, UNDEF_VALUE.format(":q")), + ("SET x = foo(#n)", None, UNDEF_NAME.format("#n")), + ("SET x = foo(:q)", None, UNDEF_VALUE.format(":q")), + ("ADD n :q", None, UNDEF_VALUE.format(":q")), + ("ADD n :s DELETE s :s", {":s": S}, ADD_DELETE_TYPE.format("ADD", "STRING")), + ("SET a = :n, a.b = :n ADD m :s", {":s": S, ":n": N}, ADD_DELETE_TYPE.format("ADD", "STRING")), + ("SET a = :s + :n ADD n :s", {":s": S, ":n": N}, OPERAND_TYPE.format("+", "S")), + ], + ) + def test_each_node_is_checked_after_its_operands(self, dynamodb_client, table, expression, values, message): + _expect(_update(dynamodb_client, table, expression, values), P + U + message) + + def test_operand_type_and_function_name_beat_unused(self, dynamodb_client, table): + _expect(_update(dynamodb_client, table, "ADD n :s", {":s": S, ":u": N}), P + U + ADD_DELETE_TYPE.format("ADD", "STRING")) + _expect(_update(dynamodb_client, table, "SET x = foo(x)", {":u": N}), P + U + INVALID_FN.format("foo")) + _expect(_update(dynamodb_client, table, "ADD n :s", {":s": S}, {"#u": "u"}), P + U + ADD_DELETE_TYPE.format("ADD", "STRING")) + + def test_a_syntax_error_anywhere_still_wins(self, dynamodb_client, table): + for expression, values in [ + ("SET a = if_not_exists(a), b = ", None), + ("SET l = list_append(list_append(:a, :a), :a), b = ", {":a": L}), + ("SET a = :s + :n, b = ", {":s": S, ":n": N}), + ("SET x = foo(x), b = ", None), + ("ADD n :s SET a = ", {":s": S}), + ]: + _expect_prefix(_update(dynamodb_client, table, expression, values), P + U) + + def test_transaction_reports_the_operand_type_bare_and_top_level(self, dynamodb_client, table): + _expect(_transact_update(dynamodb_client, table, "ADD n :s", {":s": S}), U + ADD_DELETE_TYPE.format("ADD", "STRING")) + _expect(_transact_update(dynamodb_client, table, "SET x = foo(x)"), U + INVALID_FN.format("foo")) + + def test_update_item_checks_the_condition_first(self, dynamodb_client, table): + _expect(_update(dynamodb_client, table, "SET #a = :v", {":v": N}, condition="attribute_exists(#b)"), + P + C + UNDEF_NAME.format("#b")) + _expect(_update(dynamodb_client, table, "SET a = ", condition="attribute_exists(#b)"), + P + C + UNDEF_NAME.format("#b")) + _expect(_update(dynamodb_client, table, "SET a = :v", {":v": N}, condition="b = :q"), + P + C + UNDEF_VALUE.format(":q")) + _expect(_update(dynamodb_client, table, "SET a = :v, a.b = :v", {":v": N}, condition="attribute_exists(#b)"), + P + C + UNDEF_NAME.format("#b")) + _expect_prefix(_update(dynamodb_client, table, "SET #a = :v", {":v": N}, condition="attribute_exists("), P + C) + + def test_transaction_checks_the_update_first(self, dynamodb_client, table): + _expect(_transact_update(dynamodb_client, table, "SET #a = :v", {":v": N}, condition="attribute_exists(#b)"), + U + UNDEF_NAME.format("#a")) + _expect(_transact_update(dynamodb_client, table, "SET #a = :v", {":v": N}, condition="attribute_exists("), + U + UNDEF_NAME.format("#a")) + _expect(_transact_update(dynamodb_client, table, "SET a = :v, a.b = :v", {":v": N}, condition="attribute_exists(#b)"), + U + OVERLAP_A_AB) + _expect(_transact_update(dynamodb_client, table, "ADD n :s", {":s": S}, condition="attribute_exists(#b)"), + U + ADD_DELETE_TYPE.format("ADD", "STRING")) + _expect_prefix(_transact_update(dynamodb_client, table, "SET a = ", condition="attribute_exists(#b)"), U) + + +class TestUnusedKeysAreListedTogether: + def test_values(self, dynamodb_client, table): + _expect_unused(_update(dynamodb_client, table, "SET a = :u", {":u": N, ":x": N, ":y": N, ":z": N}), + "Values", {":x", ":y", ":z"}) + + def test_names(self, dynamodb_client, table): + _expect_unused(_update(dynamodb_client, table, "SET a = :u", {":u": N}, {"#x": "x", "#y": "y", "#z": "z"}), + "Names", {"#x", "#y", "#z"}) + + def test_query_names(self, dynamodb_client, table): + _expect_unused( + lambda: dynamodb_client.query(TableName=table, KeyConditionExpression="pk = :p", + ExpressionAttributeNames={"#b": "b", "#a": "a"}, + ExpressionAttributeValues={":p": S}), + "Names", {"#a", "#b"}, + ) + + +class TestConditionFunctions: + @pytest.mark.parametrize( + ("condition", "values", "message"), + [ + ("ATTRIBUTE_EXISTS(pk)", None, INVALID_FN.format("ATTRIBUTE_EXISTS")), + ("SIZE(a) > :n", {":n": N}, INVALID_FN.format("SIZE")), + ("Begins_With(pk, :v)", {":v": S}, INVALID_FN.format("Begins_With")), + ("foo(pk) AND bar(pk)", None, INVALID_FN.format("foo")), + ("foo(pk) > :n", {":n": N}, INVALID_FN.format("foo")), + ("pk = foo(pk)", None, INVALID_FN.format("foo")), + ("foo(pk, :n)", {":n": N}, INVALID_FN.format("foo")), + ("if_not_exists(a, :v) = :v", {":v": N}, NOT_IN_CONDITION.format("if_not_exists")), + ("list_append(a, :v) = :v", {":v": N}, NOT_IN_CONDITION.format("list_append")), + ("attribute_exists(pk) = :v", {":v": N}, NOT_THIS_WAY.format("attribute_exists")), + ("attribute_exists(pk) > :n", {":n": N}, NOT_THIS_WAY.format("attribute_exists")), + ("begins_with(pk, :v) = :v", {":v": S}, NOT_THIS_WAY.format("begins_with")), + ("contains(pk, :v) = :v", {":v": S}, NOT_THIS_WAY.format("contains")), + ("attribute_type(pk, :v) = :v", {":v": S}, NOT_THIS_WAY.format("attribute_type")), + ("pk IN (:v, attribute_exists(pk))", {":v": S}, NOT_THIS_WAY.format("attribute_exists")), + ("size(attribute_exists(pk)) > :n", {":n": N}, NOT_THIS_WAY.format("attribute_exists")), + ("size(pk)", None, NOT_THIS_WAY.format("size")), + ("NOT size(pk)", None, NOT_THIS_WAY.format("size")), + ("attribute_exists(size(pk))", None, NEEDS_PATH.format("attribute_exists")), + ("attribute_exists(:v)", {":v": S}, NEEDS_PATH.format("attribute_exists")), + ("attribute_not_exists(:v)", {":v": S}, NEEDS_PATH.format("attribute_not_exists")), + ("attribute_exists(pk, pk)", None, OPERAND_COUNT.format("attribute_exists", 2)), + ("attribute_exists(:v, :w)", {":v": S, ":w": S}, OPERAND_COUNT.format("attribute_exists", 2)), + ("begins_with(pk)", None, OPERAND_COUNT.format("begins_with", 1)), + ("contains(pk)", None, OPERAND_COUNT.format("contains", 1)), + ("attribute_type(pk)", None, OPERAND_COUNT.format("attribute_type", 1)), + ("size(pk, pk) > :n", {":n": N}, OPERAND_COUNT.format("size", 2)), + ("foo()", None, EMPTY_ARGS_SYNTAX), + ("attribute_exists()", None, EMPTY_ARGS_SYNTAX), + ("foo(#n)", None, UNDEF_NAME.format("#n")), + ("foo(:q)", None, UNDEF_VALUE.format(":q")), + ("attribute_exists(pk, :q)", None, UNDEF_VALUE.format(":q")), + (":q = #n", None, UNDEF_VALUE.format(":q")), + ("#n = :q", None, UNDEF_NAME.format("#n")), + ("a = :q AND #n = :v", {":v": N}, UNDEF_VALUE.format(":q")), + ("begins_with(pk, :n)", {":n": N}, OPERAND_TYPE.format("begins_with", "N")), + ("attribute_type(pk, :n)", {":n": N}, OPERAND_TYPE.format("attribute_type", "N")), + ("attribute_type(pk, :t)", {":t": {"S": "XX"}}, + "Invalid attribute type name found; type: XX, valid types: {S,SS,N,NS,B,BS,BOOL,NULL,L,M}"), + ("begins_with(pk, :n) AND #x = :v", {":n": N, ":v": S}, OPERAND_TYPE.format("begins_with", "N")), + ("#x = :v AND begins_with(pk, :n)", {":n": N, ":v": S}, UNDEF_NAME.format("#x")), + ("pk > :l AND #x = :v", {":l": L, ":v": S}, OPERAND_TYPE.format(">", "L")), + ("size(pk) BETWEEN :l AND :n", {":l": L, ":n": N}, OPERAND_TYPE.format("BETWEEN", "L")), + ], + ) + def test_put_item_refuses_with_the_prefix(self, dynamodb_client, table, condition, values, message): + _expect(_put_cond(dynamodb_client, table, condition, values), P + C + message) + + def test_function_name_beats_unused(self, dynamodb_client, table): + _expect(_put_cond(dynamodb_client, table, "foo(pk)", {":u": N}), P + C + INVALID_FN.format("foo")) + + @pytest.mark.parametrize( + ("condition", "values"), + [ + ("size(pk) = size(pk)", None), + ("begins_with(:v, pk)", {":v": S}), + ("contains(:v, :w)", {":v": S, ":w": S}), + ("attribute_type(:v, :t)", {":v": S, ":t": {"S": "S"}}), + ("attribute_type(pk, :t)", {":t": {"S": "S"}}), + ("size(pk) > :n AND attribute_exists(pk)", {":n": N}), + ], + ) + def test_accepted_shapes(self, dynamodb_client, table, condition, values): + # The request passes validation; whether the condition then holds on + # the item is a separate matter. + try: + _put_cond(dynamodb_client, table, condition, values)() + except ClientError as exc: + assert exc.response["Error"]["Code"] == "ConditionalCheckFailedException", exc.response["Error"] + + def test_transaction_and_read_apis_carry_their_own_prefixes(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + {"ConditionCheck": {"TableName": table, "Key": KEY, "ConditionExpression": "ATTRIBUTE_EXISTS(x)"}}]), + C + INVALID_FN.format("ATTRIBUTE_EXISTS"), + ) + _expect(lambda: dynamodb_client.query(TableName=table, KeyConditionExpression="pk = :p", + FilterExpression="ATTRIBUTE_EXISTS(x)", ExpressionAttributeValues={":p": S}), + F + INVALID_FN.format("ATTRIBUTE_EXISTS")) + _expect(lambda: dynamodb_client.scan(TableName=table, FilterExpression="foo(pk)"), F + INVALID_FN.format("foo")) + _expect(lambda: dynamodb_client.query(TableName=table, KeyConditionExpression="pk = :p AND Begins_With(sk, :v)", + ExpressionAttributeValues={":p": S, ":v": S}), + K + INVALID_FN.format("Begins_With")) + + +class TestReadExpressionReferences: + """Query and Scan check their expressions before the table, on an empty table too.""" + + @pytest.fixture(scope="class") + def empty(self, dynamodb_client): + with scoped_table(dynamodb_client) as name: + yield name + + def test_scan_and_query_filters(self, dynamodb_client, empty): + _expect(lambda: dynamodb_client.scan(TableName=empty, FilterExpression="x = :q"), F + UNDEF_VALUE.format(":q")) + _expect(lambda: dynamodb_client.scan(TableName=empty, FilterExpression="#n = :q", ExpressionAttributeValues={":q": S}), + F + UNDEF_NAME.format("#n")) + _expect(lambda: dynamodb_client.query(TableName=empty, KeyConditionExpression="pk = :p", FilterExpression="x = :q", + ExpressionAttributeValues={":p": S}), F + UNDEF_VALUE.format(":q")) + + def test_key_condition_references_carry_the_prefix(self, dynamodb_client, empty): + _expect(lambda: dynamodb_client.query(TableName=empty, KeyConditionExpression="pk = :p"), K + UNDEF_VALUE.format(":p")) + _expect(lambda: dynamodb_client.query(TableName=empty, KeyConditionExpression="#n = :p", ExpressionAttributeValues={":p": S}), + K + UNDEF_NAME.format("#n")) + _expect(lambda: dynamodb_client.query(TableName=empty, KeyConditionExpression="pk = :p AND #n = :q", + ExpressionAttributeValues={":p": S}), K + UNDEF_NAME.format("#n")) + + def test_filter_then_key_condition_then_projection(self, dynamodb_client, empty): + q = dynamodb_client.query + _expect(lambda: q(TableName=empty, KeyConditionExpression="pk = :p", FilterExpression="x = :q"), + F + UNDEF_VALUE.format(":q")) + _expect(lambda: q(TableName=empty, KeyConditionExpression="#n = :p", FilterExpression="#m = :p", + ExpressionAttributeValues={":p": S}), F + UNDEF_NAME.format("#m")) + _expect(lambda: q(TableName=empty, KeyConditionExpression="pk = :p", ProjectionExpression="#p"), + K + UNDEF_VALUE.format(":p")) + _expect(lambda: q(TableName=empty, KeyConditionExpression="pk = :p", FilterExpression="x = :q", ProjectionExpression="#p", + ExpressionAttributeValues={":p": S}), F + UNDEF_VALUE.format(":q")) + _expect(lambda: q(TableName=empty, KeyConditionExpression="zz = :p", ProjectionExpression="#p", + ExpressionAttributeValues={":p": S}), PR + UNDEF_NAME.format("#p")) + _expect(lambda: q(TableName=empty, KeyConditionExpression="zz = :p", FilterExpression="x = :q", + ExpressionAttributeValues={":p": S}), F + UNDEF_VALUE.format(":q")) + _expect_prefix(lambda: q(TableName=empty, KeyConditionExpression="pk = :p", FilterExpression="x = "), F) + _expect(lambda: q(TableName=empty, KeyConditionExpression="pk = ", FilterExpression="x = :q"), F + UNDEF_VALUE.format(":q")) + _expect(lambda: q(TableName=empty, KeyConditionExpression="pk = :p", ProjectionExpression="a,,b"), K + UNDEF_VALUE.format(":p")) + _expect(lambda: dynamodb_client.scan(TableName=empty, FilterExpression="x = :q", ProjectionExpression="#p"), + F + UNDEF_VALUE.format(":q")) + _expect_prefix(lambda: dynamodb_client.scan(TableName=empty, FilterExpression="x = ", ProjectionExpression="#p"), F) + + def test_unused_before_the_key_schema_match(self, dynamodb_client, empty): + _expect_unused(lambda: dynamodb_client.query(TableName=empty, KeyConditionExpression="zz = :p", + ExpressionAttributeValues={":p": S, ":u": S}), "Values", {":u"}) + + def test_everything_before_the_table_lookup(self, dynamodb_client, missing_table): + _expect(lambda: dynamodb_client.query(TableName=missing_table, KeyConditionExpression="pk = :p", FilterExpression="x = :q", + ExpressionAttributeValues={":p": S}), F + UNDEF_VALUE.format(":q")) + _expect(lambda: dynamodb_client.query(TableName=missing_table, KeyConditionExpression="pk = :p", ProjectionExpression="#p", + ExpressionAttributeValues={":p": S}), PR + UNDEF_NAME.format("#p")) + _expect(lambda: dynamodb_client.query(TableName=missing_table, KeyConditionExpression="#n = :p", + ExpressionAttributeValues={":p": S}), K + UNDEF_NAME.format("#n")) + _expect(lambda: dynamodb_client.scan(TableName=missing_table, FilterExpression="x = :q"), F + UNDEF_VALUE.format(":q")) + _expect(lambda: dynamodb_client.get_item(TableName=missing_table, Key=KEY, ProjectionExpression="#p"), + PR + UNDEF_NAME.format("#p")) + _expect_unused(lambda: dynamodb_client.scan(TableName=missing_table, FilterExpression="x = :v", + ExpressionAttributeValues={":v": N, ":u": N}), "Values", {":u"}) + _expect_unused(lambda: dynamodb_client.query(TableName=missing_table, KeyConditionExpression="pk = :p", + ExpressionAttributeValues={":p": S, ":u": N}), "Values", {":u"}) + + def test_parameter_checks_come_before_the_expressions(self, dynamodb_client, dynamodb_client_no_validation, empty): + _expect(lambda: dynamodb_client_no_validation.scan(TableName=empty, FilterExpression="#n = :q", + ExpressionAttributeValues={":q": S}, Limit=0), + P + "Value '0' at 'limit' failed to satisfy constraint: Member must have value greater than or equal to 1") + _expect(lambda: dynamodb_client.scan(TableName=empty, FilterExpression="x = :q", Segment=0), + "The TotalSegments parameter is required but was not present in the request when Segment parameter is present") + _expect(lambda: dynamodb_client.scan(TableName=empty, ProjectionExpression="#p", ExpressionAttributeValues={":u": N}), + "ExpressionAttributeValues can only be specified when using expressions: FilterExpression is null") + + +class TestKeyShapeMessages: + BAD_TYPE = {"pk": {"N": "1"}} + MISSING = {"zz": {"S": "x"}} + EXTRA = {"pk": {"S": "x"}, "zz": {"S": "x"}} + + @pytest.mark.parametrize("key", [BAD_TYPE, MISSING, EXTRA], ids=["type", "missing", "extra"]) + def test_key_shaped_apis(self, dynamodb_client, table, key): + _expect(lambda: dynamodb_client.get_item(TableName=table, Key=key), KEY_SCHEMA) + _expect(lambda: dynamodb_client.delete_item(TableName=table, Key=key), KEY_SCHEMA) + _expect(lambda: dynamodb_client.update_item(TableName=table, Key=key, UpdateExpression="SET a = :v", + ExpressionAttributeValues={":v": N}), KEY_SCHEMA) + _expect(lambda: dynamodb_client.batch_get_item(RequestItems={table: {"Keys": [key]}}), KEY_SCHEMA) + _expect(lambda: dynamodb_client.batch_write_item(RequestItems={table: [{"DeleteRequest": {"Key": key}}]}), KEY_SCHEMA) + + def test_missing_range_key(self, dynamodb_client, range_table): + hash_only = {"pk": {"S": "x"}} + _expect(lambda: dynamodb_client.get_item(TableName=range_table, Key=hash_only), KEY_SCHEMA) + _expect(lambda: dynamodb_client.update_item(TableName=range_table, Key=hash_only, UpdateExpression="SET a = :v", + ExpressionAttributeValues={":v": N}), KEY_SCHEMA) + _expect(lambda: dynamodb_client.get_item(TableName=range_table, Key={"pk": {"S": "x"}, "sk": {"N": "1"}}), KEY_SCHEMA) + _expect(lambda: dynamodb_client.batch_write_item(RequestItems={range_table: [{"PutRequest": {"Item": hash_only}}]}), KEY_SCHEMA) + _expect(lambda: dynamodb_client.batch_write_item(RequestItems={range_table: [{"DeleteRequest": {"Key": hash_only}}]}), KEY_SCHEMA) + _expect(lambda: dynamodb_client.put_item(TableName=range_table, Item=hash_only), + "One or more parameter values were invalid: Missing the key sk in the item") + _expect(lambda: dynamodb_client.put_item(TableName=range_table, Item={"pk": {"S": "x"}, "sk": {"N": "1"}}), + "One or more parameter values were invalid: Type mismatch for key sk expected: S actual: N") + + def test_batch_write_put_uses_the_key_message_put_item_names_the_attribute(self, dynamodb_client, table): + bwi = dynamodb_client.batch_write_item + _expect(lambda: bwi(RequestItems={table: [{"PutRequest": {"Item": self.MISSING}}]}), KEY_SCHEMA) + _expect(lambda: bwi(RequestItems={table: [{"PutRequest": {"Item": self.BAD_TYPE}}]}), KEY_SCHEMA) + _expect(lambda: dynamodb_client.put_item(TableName=table, Item=self.MISSING), + "One or more parameter values were invalid: Missing the key pk in the item") + _expect(lambda: dynamodb_client.put_item(TableName=table, Item=self.BAD_TYPE), + "One or more parameter values were invalid: Type mismatch for key pk expected: S actual: N") + + def test_empty_and_oversize_key_values_keep_their_messages(self, dynamodb_client, table): + _expect(lambda: dynamodb_client.get_item(TableName=table, Key={"pk": {"S": ""}}), + "One or more parameter values are not valid. The AttributeValue for a key attribute cannot contain an empty string value. Key: pk") + _expect(lambda: dynamodb_client.delete_item(TableName=table, Key={"pk": {"S": "x" * 3000}}), + "One or more parameter values were invalid: Size of hashkey has exceeded the maximum size limit of2048 bytes") + + +class TestBatchWriteItemCollectionMetrics: + def test_empty_map_when_requested_without_an_lsi(self, dynamodb_client, table): + resp = dynamodb_client.batch_write_item(RequestItems={table: [{"PutRequest": {"Item": {"pk": {"S": "icm"}}}}]}, + ReturnItemCollectionMetrics="SIZE") + assert resp["ItemCollectionMetrics"] == {} + assert resp["UnprocessedItems"] == {} + + def test_absent_when_not_requested(self, dynamodb_client, table): + resp = dynamodb_client.batch_write_item(RequestItems={table: [{"PutRequest": {"Item": {"pk": {"S": "icm2"}}}}]}) + assert "ItemCollectionMetrics" not in resp + + def test_only_lsi_tables_appear(self, dynamodb_client, table, lsi_table): + resp = dynamodb_client.batch_write_item( + RequestItems={table: [{"PutRequest": {"Item": {"pk": {"S": "b"}}}}], + lsi_table: [{"PutRequest": {"Item": {"pk": {"S": "b"}, "sk": {"S": "1"}}}}]}, + ReturnItemCollectionMetrics="SIZE", + ) + assert set(resp["ItemCollectionMetrics"]) == {lsi_table} + [metric] = resp["ItemCollectionMetrics"][lsi_table] + assert metric["ItemCollectionKey"] == {"pk": {"S": "b"}} + assert metric["SizeEstimateRangeGB"] == [0.0, 1.0] + resp = dynamodb_client.batch_write_item( + RequestItems={lsi_table: [{"DeleteRequest": {"Key": {"pk": {"S": "zz"}, "sk": {"S": "1"}}}}]}, + ReturnItemCollectionMetrics="SIZE", + ) + assert set(resp["ItemCollectionMetrics"]) == {lsi_table} + + def test_single_item_apis_omit_the_member_without_an_lsi(self, dynamodb_client, table, lsi_table): + assert "ItemCollectionMetrics" not in dynamodb_client.put_item( + TableName=table, Item={"pk": {"S": "icm3"}}, ReturnItemCollectionMetrics="SIZE") + assert "ItemCollectionMetrics" not in dynamodb_client.update_item( + TableName=table, Key={"pk": {"S": "icm3"}}, UpdateExpression="SET a = :v", + ExpressionAttributeValues={":v": N}, ReturnItemCollectionMetrics="SIZE") + assert "ItemCollectionMetrics" not in dynamodb_client.delete_item( + TableName=table, Key={"pk": {"S": "icm3"}}, ReturnItemCollectionMetrics="SIZE") + resp = dynamodb_client.put_item(TableName=lsi_table, Item={"pk": {"S": "b"}, "sk": {"S": "2"}}, + ReturnItemCollectionMetrics="SIZE") + assert resp["ItemCollectionMetrics"]["ItemCollectionKey"] == {"pk": {"S": "b"}} diff --git a/tests/test_expression_size_limit.py b/tests/test_expression_size_limit.py index 09e321ef4..c298f0093 100644 --- a/tests/test_expression_size_limit.py +++ b/tests/test_expression_size_limit.py @@ -8,12 +8,16 @@ returns ``Invalid : Expression size has exceeded the maximum allowed size;``. -A captured quirk: FilterExpression and ConditionExpression append -`` expression size: `` (N = raw byte length); ProjectionExpression, -UpdateExpression, and KeyConditionExpression do not. - -Messages captured directly from Amazon DynamoDB (asomasun-admin, us-east-1). -Dual-target against Amazon DynamoDB and extenddb. +A captured quirk: the `` expression size: `` suffix (N = raw byte length) +depends on the operation, not only on the expression. A Scan FilterExpression +and a ConditionExpression inside TransactWriteItems carry it; a Query +FilterExpression, a ConditionExpression on PutItem, UpdateItem, or DeleteItem, +ProjectionExpression, UpdateExpression, and KeyConditionExpression do not. +PutItem, UpdateItem, and DeleteItem also wrap the message in +``1 validation error detected: ``. + +Messages captured directly from Amazon DynamoDB (us-east-1, re-measured +2026-10-14). Dual-target against Amazon DynamoDB and extenddb. """ from __future__ import annotations @@ -83,7 +87,7 @@ def test_filter_over_limit_has_size_suffix(self, dynamodb_client, hash_table): f"Invalid FilterExpression: {PREFIX} expression size: {len(expr)}" ) - def test_condition_over_limit_has_size_suffix(self, dynamodb_client, hash_table): + def test_put_condition_over_limit_no_suffix_with_prefix(self, dynamodb_client, hash_table): path = _name_of_len(MAX + 1) expr = f"attribute_not_exists({path})" with pytest.raises(ClientError) as exc_info: @@ -94,6 +98,27 @@ def test_condition_over_limit_has_size_suffix(self, dynamodb_client, hash_table) ) err = exc_info.value.response["Error"] assert err["Code"] == "ValidationException" + assert err["Message"] == ( + f"1 validation error detected: Invalid ConditionExpression: {PREFIX}" + ) + + def test_transact_condition_over_limit_has_size_suffix(self, dynamodb_client, hash_table): + path = _name_of_len(MAX + 1) + expr = f"attribute_not_exists({path})" + with pytest.raises(ClientError) as exc_info: + dynamodb_client.transact_write_items( + TransactItems=[ + { + "ConditionCheck": { + "TableName": hash_table, + "Key": {"pk": {"S": "k1"}}, + "ConditionExpression": expr, + } + } + ] + ) + err = exc_info.value.response["Error"] + assert err["Code"] == "ValidationException" assert err["Message"] == ( f"Invalid ConditionExpression: {PREFIX} expression size: {len(expr)}" ) @@ -123,7 +148,9 @@ def test_update_over_limit_no_suffix(self, dynamodb_client, hash_table): ) err = exc_info.value.response["Error"] assert err["Code"] == "ValidationException" - assert err["Message"] == f"Invalid UpdateExpression: {PREFIX}" + assert err["Message"] == ( + f"1 validation error detected: Invalid UpdateExpression: {PREFIX}" + ) def test_size_measured_on_raw_text_not_substituted(self, dynamodb_client, hash_table): # 686 short #aN placeholders => raw ~4005 bytes (under limit) even though diff --git a/tests/test_transact_validation_order.py b/tests/test_transact_validation_order.py new file mode 100644 index 000000000..038816bfb --- /dev/null +++ b/tests/test_transact_validation_order.py @@ -0,0 +1,664 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""Validation order inside TransactGetItems and TransactWriteItems. + +When a transaction carries two defects, Amazon DynamoDB reports the earlier +of these phases whichever item carries it: + + 1. per item, in order: expression grammar and the semantic walk (a + `#name` or `:value` that is not defined counts here, not as a + cancellation reason), unused names and values, then the item's 400 KB + size, all as a top-level ValidationException; + 2. a table that does not exist, as ResourceNotFoundException; + 3. the 4 MB request total, as a top-level ValidationException; + 4. a key that does not fit the table's key schema, as + TransactionCanceledException with a ValidationError reason on the first + offending item and None on every other item, even when several keys + fail: conditions are not evaluated and runtime update errors are not + reported when any item fails here; + 5. two operations on one item, as a top-level ValidationException. + +Every expected response below was captured from Amazon DynamoDB (issue #108). +Dual-target: runs against the service and against ExtendDB on every backend. +""" + +from __future__ import annotations + +import uuid + +import pytest +from botocore.exceptions import ClientError + +from conftest import scoped_table + +DUPLICATE = "Transaction request cannot include multiple operations on one item" +RESERVED_PROJECTION = ( + "Invalid ProjectionExpression: Attribute name is a reserved keyword; " + "reserved keyword: COMMENT" +) +UNUSED_NAME = "Value provided in ExpressionAttributeNames unused in expressions: keys: {#unused}" +MISSING_PROJECTION_NAME = ( + "Invalid ProjectionExpression: An expression attribute name used in the " + "document path is not defined; attribute name: #missing" +) +MISSING_CONDITION_NAME = ( + "Invalid ConditionExpression: An expression attribute name used in the " + "document path is not defined; attribute name: #missing" +) +MISSING_CONDITION_VALUE = ( + "Invalid ConditionExpression: An expression attribute value used in " + "expression is not defined; attribute value: :missing" +) +# The service's full text is 'Invalid UpdateExpression: Syntax error; token: +# "", near: "+"'. ExtendDB's parser wording differs today (tracked under +# #108 message text); this file asserts the class and prefix, which is what +# the ordering guarantees. +UPDATE_SYNTAX_PREFIX = "Invalid UpdateExpression:" +# A key that does not fit the schema: a Delete, Update, ConditionCheck, or Get +# key gets the first text; a Put item keeps the single-item wording. +KEY_SCHEMA_REASONS = { + "The provided key element does not match the schema", + "One or more parameter values were invalid: Type mismatch for key pk expected: S actual: N", +} + + +@pytest.fixture(scope="module") +def table(dynamodb_client): + """Composite-key table (pk S, sk S) with one item.""" + with scoped_table( + dynamodb_client, + attribute_definitions=[ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "S"}, + ], + key_schema=[ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + ) as name: + dynamodb_client.put_item( + TableName=name, + Item={"pk": {"S": "p"}, "sk": {"S": "s"}, "note": {"S": "old"}}, + ) + yield name + + +@pytest.fixture(scope="module") +def missing_table() -> str: + return f"extenddb-missing-{uuid.uuid4().hex[:12]}" + + +GOOD_KEY = {"pk": {"S": "p"}, "sk": {"S": "s"}} +OTHER_KEY = {"pk": {"S": "other"}, "sk": {"S": "s"}} +BAD_TYPE_KEY = {"pk": {"N": "1"}, "sk": {"S": "s"}} + + +def _get(table_name: str, key: dict, **extra) -> dict: + return {"Get": {"TableName": table_name, "Key": key, **extra}} + + +def _expect(func, code: str, message: str | None = None): + with pytest.raises(ClientError) as exc_info: + func() + err = exc_info.value.response["Error"] + assert err["Code"] == code, err + if message is not None: + assert err["Message"] == message, err + return exc_info.value.response + + +def _expect_prefix(func, prefix: str): + """For syntax errors, whose inner wording ExtendDB still phrases its own way.""" + with pytest.raises(ClientError) as exc_info: + func() + err = exc_info.value.response["Error"] + assert err["Code"] == "ValidationException", err + assert err["Message"].startswith(prefix), err + + +def _expect_cancelled(func, reasons: list[str | None]): + """Assert a cancellation whose per-item codes are exactly `reasons`. + + An item that did not fail carries the literal code "None" and no message. + """ + response = _expect(func, "TransactionCanceledException") + codes = [r.get("Code") for r in response["CancellationReasons"]] + assert codes == ["None" if r is None else r for r in reasons], response["CancellationReasons"] + for reason in response["CancellationReasons"]: + if reason["Code"] == "ValidationError": + assert reason["Message"] in KEY_SCHEMA_REASONS, reason + expected_text = ", ".join("None" if r is None else r for r in reasons) + assert response["Error"]["Message"] == ( + "Transaction cancelled, please refer cancellation reasons for specific " + f"reasons [{expected_text}]" + ) + + +class TestTransactGetItemsOrder: + def test_projection_grammar_beats_duplicate_keys(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_get_items( + TransactItems=[ + _get(table, GOOD_KEY, ProjectionExpression="COMMENT"), + _get(table, GOOD_KEY), + _get(table, GOOD_KEY), + ] + ), + "ValidationException", + RESERVED_PROJECTION, + ) + + def test_later_projection_grammar_beats_earlier_duplicate_keys(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_get_items( + TransactItems=[ + _get(table, GOOD_KEY), + _get(table, GOOD_KEY), + _get(table, OTHER_KEY, ProjectionExpression="COMMENT"), + ] + ), + "ValidationException", + RESERVED_PROJECTION, + ) + + def test_unused_projection_name_beats_duplicate_keys(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_get_items( + TransactItems=[ + _get( + table, + GOOD_KEY, + ProjectionExpression="#ok", + ExpressionAttributeNames={"#ok": "ok", "#unused": "COMMENT"}, + ), + _get(table, GOOD_KEY), + ] + ), + "ValidationException", + UNUSED_NAME, + ) + + def test_undefined_projection_name_beats_duplicate_keys(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_get_items( + TransactItems=[ + _get(table, GOOD_KEY, ProjectionExpression="#missing"), + _get(table, GOOD_KEY), + ] + ), + "ValidationException", + MISSING_PROJECTION_NAME, + ) + + def test_projection_grammar_beats_later_missing_table(self, dynamodb_client, table, missing_table): + _expect( + lambda: dynamodb_client.transact_get_items( + TransactItems=[ + _get(table, GOOD_KEY, ProjectionExpression="COMMENT"), + _get(missing_table, GOOD_KEY), + ] + ), + "ValidationException", + RESERVED_PROJECTION, + ) + + def test_missing_table_beats_duplicate_keys_either_order(self, dynamodb_client, table, missing_table): + for items in ( + [_get(missing_table, GOOD_KEY), _get(table, GOOD_KEY), _get(table, GOOD_KEY)], + [_get(table, GOOD_KEY), _get(table, GOOD_KEY), _get(missing_table, GOOD_KEY)], + ): + _expect( + lambda items=items: dynamodb_client.transact_get_items(TransactItems=items), + "ResourceNotFoundException", + "Requested resource not found", + ) + + def test_key_schema_beats_duplicate_keys_either_order(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_get_items( + TransactItems=[ + _get(table, GOOD_KEY), + _get(table, GOOD_KEY), + _get(table, BAD_TYPE_KEY), + ] + ), + [None, None, "ValidationError"], + ) + _expect_cancelled( + lambda: dynamodb_client.transact_get_items( + TransactItems=[ + _get(table, BAD_TYPE_KEY), + _get(table, GOOD_KEY), + _get(table, GOOD_KEY), + ] + ), + ["ValidationError", None, None], + ) + + def test_duplicate_keys_alone_still_refused(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_get_items( + TransactItems=[_get(table, GOOD_KEY), _get(table, GOOD_KEY)] + ), + "ValidationException", + DUPLICATE, + ) + + +class TestTransactWriteItemsOrder: + def test_undefined_condition_value_is_top_level(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + { + "ConditionCheck": { + "TableName": table, + "Key": GOOD_KEY, + "ConditionExpression": "#c = :v", + "ExpressionAttributeNames": {"#c": "COMMENT"}, + } + } + ] + ), + "ValidationException", + "Invalid ConditionExpression: An expression attribute value used in " + "expression is not defined; attribute value: :v", + ) + + def test_undefined_value_beats_later_false_condition(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + { + "ConditionCheck": { + "TableName": table, + "Key": GOOD_KEY, + "ConditionExpression": "note = :missing", + } + }, + { + "ConditionCheck": { + "TableName": table, + "Key": {"pk": {"S": "q"}, "sk": {"S": "s"}}, + "ConditionExpression": "note = :v", + "ExpressionAttributeValues": {":v": {"S": "different"}}, + } + }, + ] + ), + "ValidationException", + MISSING_CONDITION_VALUE, + ) + + def test_undefined_value_beats_key_schema_mismatch(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + { + "ConditionCheck": { + "TableName": table, + "Key": GOOD_KEY, + "ConditionExpression": "note = :missing", + } + }, + {"Delete": {"TableName": table, "Key": BAD_TYPE_KEY}}, + ] + ), + "ValidationException", + MISSING_CONDITION_VALUE, + ) + + def test_undefined_name_beats_put_key_schema_mismatch(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + { + "Delete": { + "TableName": table, + "Key": GOOD_KEY, + "ConditionExpression": "#missing = :v", + "ExpressionAttributeValues": {":v": {"S": "old"}}, + } + }, + { + "Put": { + "TableName": table, + "Item": {"pk": {"N": "1"}, "sk": {"S": "s"}, "note": {"S": "x"}}, + } + }, + ] + ), + "ValidationException", + MISSING_CONDITION_NAME, + ) + + def test_unused_name_beats_earlier_missing_table(self, dynamodb_client, table, missing_table): + _expect( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + { + "ConditionCheck": { + "TableName": missing_table, + "Key": GOOD_KEY, + "ConditionExpression": "attribute_exists(pk)", + } + }, + { + "Put": { + "TableName": table, + "Item": {"pk": {"S": "p"}, "sk": {"S": "s2"}, "note": {"S": "new"}}, + "ConditionExpression": "attribute_not_exists(pk)", + "ExpressionAttributeNames": {"#unused": "COMMENT"}, + } + }, + ] + ), + "ValidationException", + UNUSED_NAME, + ) + + def test_update_grammar_beats_missing_table_either_order(self, dynamodb_client, table, missing_table): + bad_update = { + "Update": { + "TableName": table, + "Key": GOOD_KEY, + "UpdateExpression": "SET note = :v +", + "ExpressionAttributeValues": {":v": {"N": "1"}}, + } + } + missing = { + "ConditionCheck": { + "TableName": missing_table, + "Key": GOOD_KEY, + "ConditionExpression": "attribute_exists(pk)", + } + } + for items in ([bad_update, missing], [missing, bad_update]): + response = _expect( + lambda items=items: dynamodb_client.transact_write_items(TransactItems=items), + "ValidationException", + ) + assert response["Error"]["Message"].startswith(UPDATE_SYNTAX_PREFIX), response["Error"] + + def test_key_schema_mismatch_suppresses_condition_evaluation(self, dynamodb_client, table): + false_condition = { + "ConditionCheck": { + "TableName": table, + "Key": GOOD_KEY, + "ConditionExpression": "note = :v", + "ExpressionAttributeValues": {":v": {"S": "different"}}, + } + } + bad_delete = {"Delete": {"TableName": table, "Key": BAD_TYPE_KEY}} + _expect_cancelled( + lambda: dynamodb_client.transact_write_items(TransactItems=[false_condition, bad_delete]), + [None, "ValidationError"], + ) + _expect_cancelled( + lambda: dynamodb_client.transact_write_items(TransactItems=[bad_delete, false_condition]), + ["ValidationError", None], + ) + + def test_key_schema_mismatch_suppresses_runtime_update_error(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + { + "Update": { + "TableName": table, + "Key": GOOD_KEY, + "UpdateExpression": "SET n = n + note", + } + }, + {"Delete": {"TableName": table, "Key": BAD_TYPE_KEY}}, + ] + ), + [None, "ValidationError"], + ) + + def test_key_schema_mismatch_beats_duplicate_items(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + { + "Put": { + "TableName": table, + "Item": {"pk": {"S": "p"}, "sk": {"S": "s"}, "note": {"S": "new"}}, + } + }, + { + "ConditionCheck": { + "TableName": table, + "Key": GOOD_KEY, + "ConditionExpression": "attribute_exists(pk)", + } + }, + {"Delete": {"TableName": table, "Key": BAD_TYPE_KEY}}, + ] + ), + [None, None, "ValidationError"], + ) + + def test_duplicate_items_alone_still_refused(self, dynamodb_client, table): + _expect( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + {"Put": {"TableName": table, "Item": {"pk": {"S": "p"}, "sk": {"S": "s"}}}}, + { + "ConditionCheck": { + "TableName": table, + "Key": GOOD_KEY, + "ConditionExpression": "attribute_exists(pk)", + } + }, + ] + ), + "ValidationException", + DUPLICATE, + ) + + def test_key_mismatch_reason_text_follows_the_operation(self, dynamodb_client, table): + """A Put keeps the single-item wording; a Delete gets the key-element wording.""" + response = _expect( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + {"Put": {"TableName": table, "Item": {"pk": {"N": "1"}, "sk": {"S": "s"}}}}, + ] + ), + "TransactionCanceledException", + ) + assert response["CancellationReasons"][0]["Message"] == ( + "One or more parameter values were invalid: Type mismatch for key pk expected: S actual: N" + ) + response = _expect( + lambda: dynamodb_client.transact_write_items( + TransactItems=[{"Delete": {"TableName": table, "Key": BAD_TYPE_KEY}}] + ), + "TransactionCanceledException", + ) + assert response["CancellationReasons"][0]["Message"] == ( + "The provided key element does not match the schema" + ) + + def test_false_condition_alone_still_cancels(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_write_items( + TransactItems=[ + { + "ConditionCheck": { + "TableName": table, + "Key": GOOD_KEY, + "ConditionExpression": "note = :v", + "ExpressionAttributeValues": {":v": {"S": "different"}}, + } + } + ] + ), + ["ConditionalCheckFailed"], + ) + + +class TestTransactWriteItemsResponseShape: + """Measured against Amazon DynamoDB on 2026-10-14.""" + + def test_item_collection_metrics_is_an_empty_map_when_requested_without_lsi(self, dynamodb_client, table): + resp = dynamodb_client.transact_write_items( + ReturnConsumedCapacity="TOTAL", + ReturnItemCollectionMetrics="SIZE", + TransactItems=[{"Put": {"TableName": table, "Item": {"pk": {"S": "icm"}, "sk": {"S": "s"}}}}], + ) + assert resp["ItemCollectionMetrics"] == {} + assert [c["TableName"] for c in resp["ConsumedCapacity"]] == [table] + + def test_item_collection_metrics_absent_when_not_requested(self, dynamodb_client, table): + resp = dynamodb_client.transact_write_items( + TransactItems=[{"Put": {"TableName": table, "Item": {"pk": {"S": "icm2"}, "sk": {"S": "s"}}}}], + ) + assert "ItemCollectionMetrics" not in resp + assert "ConsumedCapacity" not in resp + + +BIG_ITEM = {"pk": {"S": "big"}, "sk": {"S": "s"}, "blob": {"S": "x" * 410_000}} +ITEM_SIZE = "Item size has exceeded the maximum allowed size" +UNDEF_UPDATE_VALUE = ( + "Invalid UpdateExpression: An expression attribute value used in " + "expression is not defined; attribute value: :q" +) + + +def _put(table_name: str, item: dict, **extra) -> dict: + return {"Put": {"TableName": table_name, "Item": item, **extra}} + + +def _update(table_name: str, key: dict, expression: str = "SET a = :v", values: dict | None = None) -> dict: + return {"Update": {"TableName": table_name, "Key": key, "UpdateExpression": expression, + "ExpressionAttributeValues": values if values is not None else {":v": {"N": "1"}}}} + + +class TestKeySchemaFirstOffenderOnly: + """Measured 2026-10-14: only the first item whose key fails carries ValidationError.""" + + def test_write_items_first_and_third(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + _update(table, BAD_TYPE_KEY), _update(table, GOOD_KEY), _update(table, {"pk": {"N": "3"}, "sk": {"S": "s"}})]), + ["ValidationError", None, None], + ) + + def test_write_items_all_three(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + _update(table, BAD_TYPE_KEY), _update(table, {"pk": {"N": "2"}, "sk": {"S": "s"}}), + _update(table, {"pk": {"N": "3"}, "sk": {"S": "s"}})]), + ["ValidationError", None, None], + ) + + def test_write_items_middle_only(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + _update(table, GOOD_KEY), _update(table, BAD_TYPE_KEY), _update(table, OTHER_KEY)]), + [None, "ValidationError", None], + ) + + def test_false_condition_between_two_offenders_is_not_evaluated(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + _update(table, BAD_TYPE_KEY), + {"ConditionCheck": {"TableName": table, "Key": GOOD_KEY, "ConditionExpression": "note = :z", + "ExpressionAttributeValues": {":z": {"S": "never"}}}}, + _update(table, {"pk": {"N": "3"}, "sk": {"S": "s"}})]), + ["ValidationError", None, None], + ) + + def test_get_items(self, dynamodb_client, table): + _expect_cancelled( + lambda: dynamodb_client.transact_get_items(TransactItems=[ + _get(table, BAD_TYPE_KEY), _get(table, GOOD_KEY), _get(table, {"pk": {"N": "3"}, "sk": {"S": "s"}})]), + ["ValidationError", None, None], + ) + _expect_cancelled( + lambda: dynamodb_client.transact_get_items(TransactItems=[ + _get(table, BAD_TYPE_KEY), _get(table, {"pk": {"N": "2"}, "sk": {"S": "s"}}), + _get(table, {"pk": {"N": "3"}, "sk": {"S": "s"}})]), + ["ValidationError", None, None], + ) + + +class TestItemSizePhase: + """The 400 KB item size follows the item's own expressions and precedes the + next item's, the table lookup, the key schema, and duplicates (measured + 2026-10-14).""" + + def test_size_follows_the_same_items_expressions(self, dynamodb_client, table): + _expect_prefix( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, BIG_ITEM, ConditionExpression="attribute_exists(")]), + "Invalid ConditionExpression:", + ) + _expect( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, BIG_ITEM, ConditionExpression="a = :q")]), + "ValidationException", MISSING_CONDITION_VALUE.replace(":missing", ":q"), + ) + _expect( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, BIG_ITEM, ConditionExpression="a = :v", + ExpressionAttributeValues={":v": {"N": "1"}, ":u": {"N": "1"}})]), + "ValidationException", "Value provided in ExpressionAttributeValues unused in expressions: keys: {:u}", + ) + + def test_size_precedes_the_next_items_expressions(self, dynamodb_client, table): + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, BIG_ITEM), _update(table, GOOD_KEY, "SET a = :q")]), "ValidationException", ITEM_SIZE) + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _update(table, GOOD_KEY, "SET a = :q"), _put(table, BIG_ITEM)]), "ValidationException", UNDEF_UPDATE_VALUE) + _expect_prefix(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _update(table, GOOD_KEY, "SET a = "), _put(table, BIG_ITEM)]), UPDATE_SYNTAX_PREFIX) + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, BIG_ITEM), _update(table, GOOD_KEY, "SET a = ")]), "ValidationException", ITEM_SIZE) + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, BIG_ITEM), + _put(table, GOOD_KEY, ConditionExpression="a = :v", + ExpressionAttributeValues={":v": {"N": "1"}, ":u": {"N": "1"}})]), + "ValidationException", ITEM_SIZE) + + def test_size_precedes_table_lookup_key_schema_and_duplicates(self, dynamodb_client, table, missing_table): + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(missing_table, GOOD_KEY), _put(table, BIG_ITEM)]), "ValidationException", ITEM_SIZE) + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _update(table, BAD_TYPE_KEY), _put(table, BIG_ITEM)]), "ValidationException", ITEM_SIZE) + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, GOOD_KEY), _put(table, GOOD_KEY), _put(table, BIG_ITEM)]), "ValidationException", ITEM_SIZE) + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, {"pk": {"N": "1"}, "sk": {"S": "s"}, "blob": {"S": "x" * 410_000}})]), + "ValidationException", ITEM_SIZE) + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=[ + _put(table, {"zz": {"S": "1"}, "blob": {"S": "x" * 410_000}})]), "ValidationException", ITEM_SIZE) + + def test_request_total_follows_table_lookup_and_precedes_key_schema_and_duplicates( + self, dynamodb_client, table, missing_table + ): + # The service's message carries its own serialized payload size + # ("Transaction payload size cannot exceed 4MB. Payload Size: N"); + # ExtendDB states the limit without a size. Both name the 4 MB limit. + mids = [_put(table, {"pk": {"S": f"m{i}"}, "sk": {"S": "s"}, "blob": {"S": "y" * 390_000}}) for i in range(11)] + + def expect_total(items): + with pytest.raises(ClientError) as exc_info: + dynamodb_client.transact_write_items(TransactItems=items) + err = exc_info.value.response["Error"] + assert err["Code"] == "ValidationException", err + assert "4MB" in err["Message"] or "4 MB" in err["Message"], err + + expect_total(mids) + expect_total(mids + [_update(table, BAD_TYPE_KEY)]) + expect_total(mids + [_put(table, GOOD_KEY), _put(table, GOOD_KEY)]) + expect_total(mids + [{"ConditionCheck": {"TableName": table, "Key": GOOD_KEY, "ConditionExpression": "note = :z", + "ExpressionAttributeValues": {":z": {"S": "never"}}}}]) + _expect(lambda: dynamodb_client.transact_write_items(TransactItems=mids + [_put(missing_table, GOOD_KEY)]), + "ResourceNotFoundException") + _expect_prefix(lambda: dynamodb_client.transact_write_items(TransactItems=mids + [_update(table, GOOD_KEY, "SET a = ")]), + UPDATE_SYNTAX_PREFIX) + _expect_prefix(lambda: dynamodb_client.transact_write_items(TransactItems=[_update(table, GOOD_KEY, "SET a = ")] + mids), + UPDATE_SYNTAX_PREFIX) diff --git a/tests/test_update_expression_rules.py b/tests/test_update_expression_rules.py new file mode 100644 index 000000000..ac1874ebf --- /dev/null +++ b/tests/test_update_expression_rules.py @@ -0,0 +1,303 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""UpdateExpression request-level rules, and the single-item write prefix. + +Amazon DynamoDB rejects these before it reads the table: + + - a repeated section (`SET ... SET ...`, likewise REMOVE, ADD, DELETE), + reported after the whole expression has parsed, so a syntax error + anywhere wins over it, and after that section's own actions have been + checked (test_expression_semantics.py pins that order); + - `if_not_exists` and `list_append` with other than two operands; + - `if_not_exists` whose first operand is not a document path; + - `list_append` as an operand of another function + (`list_append(if_not_exists(l, :e), :v)` is allowed; the reverse is not); + - two actions on overlapping paths, across sections, after `#name` + resolution, reported as `path one: [...], path two: [...]`. + +Order on the service when a request carries two of these: syntax error, +then one walk over the actions in expression order (undefined `#name` or +`:value`, function rules, and a repeated section at the end of its section), +then overlap, then unused name or value. + +PutItem, UpdateItem, and DeleteItem wrap every expression-stage error in +`1 validation error detected: `; TransactWriteItems and the read APIs do not. +Every expected message was captured from Amazon DynamoDB on 2026-10-14. +Dual-target. +""" + +from __future__ import annotations + +import pytest +from botocore.exceptions import ClientError + +from conftest import scoped_table + +P = "1 validation error detected: " +U = "Invalid UpdateExpression: " +SECTION_ONCE = U + 'The "{}" section can only be used once in an update expression;' +OPERANDS = U + "Incorrect number of operands for operator or function; operator or function: {}, number of operands: {}" +NEEDS_PATH = U + "Operator or function requires a document path; operator or function: if_not_exists" +NOT_THIS_WAY = U + "The function is not allowed to be used this way in an expression; function: list_append" +OVERLAP = U + "Two document paths overlap with each other; must remove or rewrite one of these paths; path one: {}, path two: {}" + + +@pytest.fixture(scope="module") +def table(dynamodb_client): + with scoped_table(dynamodb_client) as name: + dynamodb_client.put_item( + TableName=name, + Item={"pk": {"S": "p"}, "fieldx": {"N": "1"}, "mapx": {"M": {"num": {"N": "1"}}}, + "lst": {"L": [{"M": {"ok": {"N": "1"}}}, {"M": {"x": {"N": "1"}}}]}}, + ) + yield name + + +KEY = {"pk": {"S": "p"}} +V = {":v": {"N": "1"}} + + +def _expect(func, message: str): + with pytest.raises(ClientError) as exc_info: + func() + err = exc_info.value.response["Error"] + assert err["Code"] == "ValidationException", err + assert err["Message"] == message, err + + +def _update(client, table, expression, names=None, values=None): + kwargs = {"TableName": table, "Key": KEY, "UpdateExpression": expression} + if names: + kwargs["ExpressionAttributeNames"] = names + if values: + kwargs["ExpressionAttributeValues"] = values + return lambda: client.update_item(**kwargs) + + +def _transact_update(client, table, expression, names=None, values=None): + item = {"TableName": table, "Key": KEY, "UpdateExpression": expression} + if names: + item["ExpressionAttributeNames"] = names + if values: + item["ExpressionAttributeValues"] = values + return lambda: client.transact_write_items(TransactItems=[{"Update": item}]) + + +class TestRepeatedSections: + @pytest.mark.parametrize( + ("expression", "section"), + [ + ("SET fieldx = :v SET otherx = :v", "SET"), + ("REMOVE a REMOVE b", "REMOVE"), + ("ADD a :v ADD b :v", "ADD"), + ("DELETE a :s DELETE b :s", "DELETE"), + ], + ) + def test_section_used_twice(self, dynamodb_client, table, expression, section): + values = {":s": {"SS": ["x"]}} if "DELETE" in expression else V + _expect(_update(dynamodb_client, table, expression, values=values), P + SECTION_ONCE.format(section)) + + def test_section_check_follows_the_grammar_pass(self, dynamodb_client, table): + # The service answers 'Syntax error; token: "", near: "+"'; ExtendDB's + # parser wording differs today (tracked under #108 message text). The + # guarantee pinned here is the order: the syntax error wins over the + # repeated-section error. + with pytest.raises(ClientError) as exc_info: + _update(dynamodb_client, table, "SET a = :v SET b = :v +", values=V)() + err = exc_info.value.response["Error"] + assert err["Code"] == "ValidationException", err + assert err["Message"].startswith(P + U), err + assert "section can only be used once" not in err["Message"], err + + def test_repeated_section_beats_overlap(self, dynamodb_client, table): + _expect(_update(dynamodb_client, table, "SET a = :v SET a = :v", values=V), P + SECTION_ONCE.format("SET")) + + def test_transaction_reports_it_without_the_prefix(self, dynamodb_client, table): + _expect( + _transact_update(dynamodb_client, table, "SET fieldx = :v SET otherx = :v", values=V), + SECTION_ONCE.format("SET"), + ) + + +class TestFunctionOperands: + def test_if_not_exists_one_operand(self, dynamodb_client, table): + _expect(_update(dynamodb_client, table, "SET a = if_not_exists(a)"), P + OPERANDS.format("if_not_exists", 1)) + + def test_if_not_exists_three_operands(self, dynamodb_client, table): + _expect( + _update(dynamodb_client, table, "SET a = if_not_exists(a, :v, :v)", values=V), + P + OPERANDS.format("if_not_exists", 3), + ) + + def test_list_append_one_operand(self, dynamodb_client, table): + _expect( + _update(dynamodb_client, table, "SET l = list_append(:l)", values={":l": {"L": []}}), + P + OPERANDS.format("list_append", 1), + ) + + def test_if_not_exists_first_operand_must_be_a_path(self, dynamodb_client, table): + _expect(_update(dynamodb_client, table, "SET a = if_not_exists(:v, :v)", values=V), P + NEEDS_PATH) + + def test_nested_if_not_exists_is_not_a_path(self, dynamodb_client, table): + _expect( + _update( + dynamodb_client, table, "SET #r = if_not_exists(if_not_exists(n, :zero), :one)", + names={"#r": "result"}, values={":zero": {"N": "0"}, ":one": {"N": "1"}}, + ), + P + NEEDS_PATH, + ) + + def test_list_append_inside_list_append(self, dynamodb_client, table): + _expect( + _update( + dynamodb_client, table, "SET #r = list_append(list_append(l, :tail), :tail2)", + names={"#r": "result"}, values={":tail": {"L": [{"S": "b"}]}, ":tail2": {"L": [{"S": "c"}]}}, + ), + P + NOT_THIS_WAY, + ) + + def test_list_append_inside_if_not_exists(self, dynamodb_client, table): + _expect( + _update( + dynamodb_client, table, "SET l = if_not_exists(l, list_append(:e, :v))", + values={":e": {"L": []}, ":v": {"L": [{"S": "x"}]}}, + ), + P + NOT_THIS_WAY, + ) + + def test_if_not_exists_inside_list_append_is_allowed(self, dynamodb_client, table): + dynamodb_client.update_item( + TableName=table, Key=KEY, + UpdateExpression="SET l2 = list_append(if_not_exists(l2, :e), :v)", + ExpressionAttributeValues={":e": {"L": []}, ":v": {"L": [{"S": "x"}]}}, + ) + item = dynamodb_client.get_item(TableName=table, Key=KEY)["Item"] + assert item["l2"] == {"L": [{"S": "x"}]} + + +class TestOverlappingPaths: + @pytest.mark.parametrize( + ("expression", "names", "values", "one", "two"), + [ + ("SET fieldx = :v, fieldx = :w", None, {":v": {"N": "1"}, ":w": {"N": "2"}}, "[fieldx]", "[fieldx]"), + ("SET mapx = :m, mapx.num = :v", None, {":m": {"M": {}}, ":v": {"N": "2"}}, "[mapx]", "[mapx, num]"), + ("SET #a = :v, fieldx = :v", {"#a": "fieldx"}, V, "[fieldx]", "[fieldx]"), + ("SET a = :v REMOVE a", None, V, "[a]", "[a]"), + ("SET a.b = :v REMOVE a", None, V, "[a, b]", "[a]"), + ("REMOVE a, a.b", None, None, "[a]", "[a, b]"), + ("SET a = :v ADD a :v", None, V, "[a]", "[a]"), + ("SET #l[1] = :v, #l[1].x = :v", {"#l": "lst"}, V, "[lst, [1]]", "[lst, [1], x]"), + ], + ) + def test_overlap(self, dynamodb_client, table, expression, names, values, one, two): + _expect(_update(dynamodb_client, table, expression, names=names, values=values), P + OVERLAP.format(one, two)) + + def test_siblings_do_not_overlap(self, dynamodb_client, table): + dynamodb_client.update_item( + TableName=table, Key=KEY, + UpdateExpression="SET mapx.a = :v, mapx.b = :v", + ExpressionAttributeValues=V, + ) + + def test_undefined_value_beats_overlap(self, dynamodb_client, table): + _expect( + _update(dynamodb_client, table, "SET a = :v, a = :missing", values=V), + P + U + "An expression attribute value used in expression is not defined; attribute value: :missing", + ) + + def test_undefined_name_beats_overlap(self, dynamodb_client, table): + _expect( + _update(dynamodb_client, table, "SET a = :v, a = :v, #m = :v", values=V), + P + U + "An expression attribute name used in the document path is not defined; attribute name: #m", + ) + + def test_overlap_beats_unused_name(self, dynamodb_client, table): + _expect( + _update(dynamodb_client, table, "SET a = :v, a = :v", names={"#u": "x"}, values=V), + P + OVERLAP.format("[a]", "[a]"), + ) + + def test_transaction_overlap_without_the_prefix(self, dynamodb_client, table): + _expect( + _transact_update(dynamodb_client, table, "SET fieldx = :s, fieldx = :o", + values={":s": {"S": "new"}, ":o": {"S": "other"}}), + OVERLAP.format("[fieldx]", "[fieldx]"), + ) + _expect( + _transact_update(dynamodb_client, table, "SET lst[0] = :v, lst[0].ok = :v", values=V), + OVERLAP.format("[lst, [0]]", "[lst, [0], ok]"), + ) + + +class TestSingleItemWritePrefix: + """PutItem, UpdateItem, DeleteItem prefix expression-stage errors; the read APIs do not.""" + + RESERVED_C = "Invalid ConditionExpression: Attribute name is a reserved keyword; reserved keyword: count" + UNUSED_N = "Value provided in ExpressionAttributeNames unused in expressions: keys: {#u}" + UNUSED_V = "Value provided in ExpressionAttributeValues unused in expressions: keys: {:u}" + NAMES_NO_EXPR = "ExpressionAttributeNames can only be specified when using expressions" + VALUES_NO_EXPR = "ExpressionAttributeValues can only be specified when using expressions" + UNDEF_V = "Invalid ConditionExpression: An expression attribute value used in expression is not defined; attribute value: :missing" + UNDEF_N = "Invalid ConditionExpression: An expression attribute name used in the document path is not defined; attribute name: #m" + SIZE_C = "Invalid ConditionExpression: Expression size has exceeded the maximum allowed size;" + SIZE_U = "Invalid UpdateExpression: Expression size has exceeded the maximum allowed size;" + SIZE_F = "Invalid FilterExpression: Expression size has exceeded the maximum allowed size;" + + def test_put_reserved_word(self, dynamodb_client, table): + _expect(lambda: dynamodb_client.put_item(TableName=table, Item=KEY, ConditionExpression="count = :v", + ExpressionAttributeValues=V), P + self.RESERVED_C) + + def test_delete_reserved_word(self, dynamodb_client, table): + _expect(lambda: dynamodb_client.delete_item(TableName=table, Key=KEY, ConditionExpression="count = :v", + ExpressionAttributeValues=V), P + self.RESERVED_C) + + def test_unused_names_and_values(self, dynamodb_client, table): + _expect(lambda: dynamodb_client.update_item(TableName=table, Key=KEY, UpdateExpression="SET a = :v", + ExpressionAttributeNames={"#u": "x"}, ExpressionAttributeValues=V), P + self.UNUSED_N) + _expect(lambda: dynamodb_client.put_item(TableName=table, Item=KEY, ConditionExpression="attribute_exists(pk)", + ExpressionAttributeNames={"#u": "x"}), P + self.UNUSED_N) + _expect(lambda: dynamodb_client.delete_item(TableName=table, Key=KEY, ConditionExpression="attribute_exists(pk)", + ExpressionAttributeValues={":u": {"N": "1"}}), P + self.UNUSED_V) + + def test_names_or_values_without_expression(self, dynamodb_client, table): + _expect(lambda: dynamodb_client.put_item(TableName=table, Item=KEY, ExpressionAttributeNames={"#u": "x"}), + P + self.NAMES_NO_EXPR) + _expect(lambda: dynamodb_client.delete_item(TableName=table, Key=KEY, ExpressionAttributeValues=V), + P + self.VALUES_NO_EXPR) + _expect(lambda: dynamodb_client.update_item(TableName=table, Key=KEY, ExpressionAttributeValues=V), + P + self.VALUES_NO_EXPR) + + def test_undefined_references(self, dynamodb_client, table): + _expect(lambda: dynamodb_client.put_item(TableName=table, Item=KEY, ConditionExpression="a = :missing"), + P + self.UNDEF_V) + _expect(lambda: dynamodb_client.delete_item(TableName=table, Key=KEY, ConditionExpression="#m = :v", + ExpressionAttributeValues=V), P + self.UNDEF_N) + + def test_over_size_condition_has_no_length_suffix(self, dynamodb_client, table): + big = "attribute_exists(" + "a" * 4097 + ")" + _expect(lambda: dynamodb_client.put_item(TableName=table, Item=KEY, ConditionExpression=big), P + self.SIZE_C) + _expect(lambda: dynamodb_client.update_item(TableName=table, Key=KEY, UpdateExpression="SET a = :v", + ConditionExpression=big, ExpressionAttributeValues=V), P + self.SIZE_C) + _expect(lambda: dynamodb_client.update_item(TableName=table, Key=KEY, UpdateExpression="SET " + "a" * 4097 + " = :v", + ExpressionAttributeValues=V), P + self.SIZE_U) + + def test_transaction_condition_keeps_the_length_suffix(self, dynamodb_client, table): + big = "attribute_exists(" + "a" * 4097 + ")" + _expect( + lambda: dynamodb_client.transact_write_items(TransactItems=[ + {"ConditionCheck": {"TableName": table, "Key": KEY, "ConditionExpression": big}}]), + self.SIZE_C + f" expression size: {len(big)}", + ) + + def test_query_filter_has_no_suffix_scan_filter_has_one(self, dynamodb_client, table): + big = "attribute_exists(" + "a" * 4097 + ")" + _expect(lambda: dynamodb_client.query(TableName=table, KeyConditionExpression="pk = :v", FilterExpression=big, + ExpressionAttributeValues={":v": {"S": "p"}}), self.SIZE_F) + _expect(lambda: dynamodb_client.scan(TableName=table, FilterExpression=big), self.SIZE_F + f" expression size: {len(big)}") + + def test_read_apis_carry_no_prefix(self, dynamodb_client, table): + _expect(lambda: dynamodb_client.get_item(TableName=table, Key=KEY, ProjectionExpression="#a", + ExpressionAttributeNames={"#a": "a", "#u": "x"}), self.UNUSED_N) + _expect(lambda: dynamodb_client.query(TableName=table, KeyConditionExpression="pk = :v", + ExpressionAttributeValues={":v": {"S": "p"}, ":u": {"N": "1"}}), self.UNUSED_V) From a8d83c620ee8a770e647a906a8975851d4d0ad63 Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 11:22:30 +0000 Subject: [PATCH 02/10] fix(expressions): keep the legacy filter operand-type checks, clarify three comments The move of the FilterExpression operand-type checks into the semantic pass dropped them for the legacy QueryFilter and ScanFilter parameters, which bypass the parser. A ScanFilter with BEGINS_WITH and a number, or LT and a boolean, was accepted at request time and refused only when a row was evaluated, so an empty table returned 200. The desugared legacy filter now runs the same begins_with and ordering operand checks as before, with the FilterExpression prefix, before any row is read. Wire tests cover both parameters on an empty table. Comments reworded after review: the parsers state that grammar errors are reported before semantic errors and why the semantic checks are deferred; the empty operand list comment names the measured cases; the PerTableCapacity doc states that first-appearance order is a firmer guarantee than the service gives, with the measured orders. --- crates/core/src/expression/parser.rs | 18 +++++-- crates/core/src/expression/update_parser.rs | 15 ++++-- crates/engine/src/capacity_helpers.rs | 16 +++--- crates/engine/src/expression_helpers.rs | 23 +++++++++ crates/engine/src/query.rs | 3 ++ crates/engine/src/scan.rs | 5 +- tests/test_expr_operand_type_and_size.py | 56 +++++++++++++++++++++ 7 files changed, 120 insertions(+), 16 deletions(-) diff --git a/crates/core/src/expression/parser.rs b/crates/core/src/expression/parser.rs index 97914f28a..a3b3398c7 100755 --- a/crates/core/src/expression/parser.rs +++ b/crates/core/src/expression/parser.rs @@ -135,10 +135,13 @@ fn parse_primary( // Operand (a path, a placeholder, or a function call), then a comparator, // BETWEEN, or IN. A function call with nothing after it is the condition - // itself; whether that function may stand alone, whether its name exists, - // and how many operands it takes are semantic questions answered after - // the grammar pass (see `validate_condition_semantics`), so a syntax error - // further along still wins, as on Amazon DynamoDB. + // itself. This pass checks grammar only: whether that function may stand + // alone, whether its name exists, and how many operands it takes are + // checked afterwards by `validate_condition_semantics`, over the whole + // parsed tree. Amazon DynamoDB reports every syntax error before any + // semantic error, so `foo(pk) AND` is reported as the syntax error at the + // end, not as the unknown function name that comes first in the text. + // Deferring the semantic checks here gives the same precedence. let left = parse_operand(tokens, pos)?; if *pos < tokens.len() { @@ -235,7 +238,12 @@ fn parse_function_call(tokens: &[Token], pos: &mut usize) -> Result Result Result { if *pos >= tokens.len() { return Err(validation_err("Invalid UpdateExpression: expected operand")); @@ -329,7 +333,10 @@ fn parse_set_function_call(tokens: &[Token], pos: &mut usize) -> Result( /// Per-table capacity totals in the order the tables first appear in the /// request. /// -/// Amazon DynamoDB's own order of the `ConsumedCapacity` entries of a -/// transaction follows no documented rule (measured 2026-10-14: hash order of -/// the table names, which differs between name sets), so no client can depend -/// on it. A `HashMap` here made ExtendDB's order change from one process to -/// the next; first appearance is stable and is what a reader of the request -/// expects. +/// This is a firmer guarantee than Amazon DynamoDB gives, not a copy of its +/// order. The service documents no order for the `ConsumedCapacity` entries +/// of a transaction, and the measured order (2026-10-14) is neither first +/// appearance nor alphabetical: a TransactGetItems naming table b, then a, +/// then b again returned a before b; a TransactWriteItems naming b, a2, b +/// returned b before a2. A client cannot depend on the service's order, so +/// any deterministic order is compatible with it. Before this change ExtendDB +/// aggregated into a `HashMap`, so the order changed from one process to the +/// next and differed between backends; first appearance is stable, matches +/// the request as the caller wrote it, and lets the tests assert the array. #[derive(Default)] pub struct PerTableCapacity(Vec<(String, f64)>); diff --git a/crates/engine/src/expression_helpers.rs b/crates/engine/src/expression_helpers.rs index 5d02dd41d..115380c67 100755 --- a/crates/engine/src/expression_helpers.rs +++ b/crates/engine/src/expression_helpers.rs @@ -378,6 +378,29 @@ pub fn check_key_condition_refs( Ok(()) } +/// Reject a desugared legacy `QueryFilter` or `ScanFilter` whose +/// `BEGINS_WITH` value is not a string or binary, or whose ordering +/// comparison (`LT`, `LE`, `GT`, `GE`, `BETWEEN`) has a non-orderable value, +/// at request time, before any row is read. The expression form gets these +/// checks from [`parse_condition_checked`]; the legacy form bypasses the +/// parser, so they run here over the desugared tree with the maps +/// `desugar_filter` produced. Behaviour and message are unchanged from before +/// the semantic pass existed. +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` with the `Invalid +/// FilterExpression:` prefix. +pub fn check_legacy_filter_operand_types( + filter: &Expr, + maps: &ExpressionMaps, +) -> Result<(), DynamoDbError> { + extenddb_core::expression::validate_begins_with_operands(filter, maps) + .map_err(|e| prefix_expression_error(e, ExpressionKind::Filter))?; + extenddb_core::expression::validate_ordering_operand_types(filter, maps) + .map_err(|e| prefix_expression_error(e, ExpressionKind::Filter)) +} + /// Prefix an expression error with the expression type, matching `DynamoDB`'s format. /// /// `FilterExpression` shares the condition parser, so its errors arrive labelled diff --git a/crates/engine/src/query.rs b/crates/engine/src/query.rs index 1473de845..d0b8058c3 100755 --- a/crates/engine/src/query.rs +++ b/crates/engine/src/query.rs @@ -153,10 +153,13 @@ pub async fn handle_query( // types), the KeyConditionExpression (syntax, then references), the // ProjectionExpression (syntax, then references and overlap), then unused // names and values. A missing table is reported after all of these. + // A legacy QueryFilter skips the parser, so its operand types are checked + // separately, still before any row is read. let (filter, filter_maps) = if let Some(qf) = input.query_filter.as_ref().filter(|qf| !qf.is_empty()) { let cond_op = input.conditional_operator.unwrap_or_default(); let (expr, fmaps) = desugar_filter(qf, cond_op)?; + crate::expression_helpers::check_legacy_filter_operand_types(&expr, &fmaps)?; (Some(expr), Some(fmaps)) } else { match input.filter_expression.as_deref() { diff --git a/crates/engine/src/scan.rs b/crates/engine/src/scan.rs index c279577f8..dee37cc82 100755 --- a/crates/engine/src/scan.rs +++ b/crates/engine/src/scan.rs @@ -182,11 +182,14 @@ pub async fn handle_scan( // form runs the service's ordered semantic walk (references, functions, // operand types) here, ahead of the projection, the unused check, and the // table lookup, so an undefined `:q` is a request error even when no row - // would be evaluated (measured 2026-10-14 on an empty table). + // would be evaluated (measured 2026-10-14 on an empty table). A legacy + // ScanFilter skips the parser, so its operand types are checked + // separately, still before any row is read. let (filter, filter_maps) = if let Some(sf) = input.scan_filter.as_ref().filter(|sf| !sf.is_empty()) { let cond_op = input.conditional_operator.unwrap_or_default(); let (expr, fmaps) = desugar_filter(sf, cond_op)?; + crate::expression_helpers::check_legacy_filter_operand_types(&expr, &fmaps)?; (Some(expr), Some(fmaps)) } else { match input.filter_expression.as_deref() { diff --git a/tests/test_expr_operand_type_and_size.py b/tests/test_expr_operand_type_and_size.py index 2641cb07a..1f9a7a7c1 100644 --- a/tests/test_expr_operand_type_and_size.py +++ b/tests/test_expr_operand_type_and_size.py @@ -350,3 +350,59 @@ def test_query_filter_carries_filter_prefix(self, dynamodb_client, probe_table): err = _err(ei) assert err["Code"] == "ValidationException" assert "Invalid FilterExpression: Incorrect operand type" in err["Message"] + + +@pytest.fixture(scope="class") +def empty_table(dynamodb_client): + name = f"expr_legacy_{uuid.uuid4().hex[:8]}" + dynamodb_client.create_table( + TableName=name, + KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}], + AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}], + BillingMode="PAY_PER_REQUEST", + ) + from conftest import wait_for_active + + wait_for_active(dynamodb_client, name) + yield name + dynamodb_client.delete_table(TableName=name) + + +class TestLegacyFilterOperandType: + """The legacy QueryFilter and ScanFilter parameters are checked at request + time too: on an empty table, where no row is evaluated, BEGINS_WITH with a + number and an ordering comparison with a boolean are refused. Only the + error code is asserted here; the service's message text for the legacy + parameters has not been captured.""" + + def test_scan_filter_begins_with_number_rejected_on_empty_table(self, dynamodb_client, empty_table): + with pytest.raises(ClientError) as ei: + dynamodb_client.scan( + TableName=empty_table, + ScanFilter={"pk": {"ComparisonOperator": "BEGINS_WITH", "AttributeValueList": [{"N": "1"}]}}, + ) + assert _err(ei)["Code"] == "ValidationException" + + def test_scan_filter_ordering_boolean_rejected_on_empty_table(self, dynamodb_client, empty_table): + with pytest.raises(ClientError) as ei: + dynamodb_client.scan( + TableName=empty_table, + ScanFilter={"n": {"ComparisonOperator": "LT", "AttributeValueList": [{"BOOL": True}]}}, + ) + assert _err(ei)["Code"] == "ValidationException" + + def test_query_filter_begins_with_number_rejected_on_empty_table(self, dynamodb_client, empty_table): + with pytest.raises(ClientError) as ei: + dynamodb_client.query( + TableName=empty_table, + KeyConditions={"pk": {"ComparisonOperator": "EQ", "AttributeValueList": [{"S": "absent"}]}}, + QueryFilter={"s": {"ComparisonOperator": "BEGINS_WITH", "AttributeValueList": [{"N": "1"}]}}, + ) + assert _err(ei)["Code"] == "ValidationException" + + def test_scan_filter_begins_with_string_accepted(self, dynamodb_client, empty_table): + resp = dynamodb_client.scan( + TableName=empty_table, + ScanFilter={"pk": {"ComparisonOperator": "BEGINS_WITH", "AttributeValueList": [{"S": "p"}]}}, + ) + assert resp["Count"] == 0 From 343c2fb51ece1e8a571bd6ed7a27a4e1075dd620 Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 13:45:36 +0000 Subject: [PATCH 03/10] fix(expressions): refuse legacy filter operand types with the service's message A ScanFilter or QueryFilter condition whose AttributeValueList holds a value of a type its ComparisonOperator does not take is refused at request time with the service's text, `One or more parameter values were invalid: ComparisonOperator is not valid for AttributeValue type`, naming the first such value. Measured against Amazon DynamoDB on 2026-10-23 on an empty table: BEGINS_WITH with N, LT and GT with BOOL, and BETWEEN with L then N, which names the L. The check runs in desugar_filter on each condition before it is desugared; the `Invalid FilterExpression:` wording that the previous commit restored is gone. attribute_type_code is exported from extenddb-core for the type tag. Pin the condition-expression order for a syntax error after a semantic error: `foo(pk) AND`, `ATTRIBUTE_EXISTS(pk) AND`, `begins_with(pk, :n) AND` and `attribute_exists(#x) AND` each answer the syntax error on the service (measured 2026-10-23), and the test asserts that no semantic message is returned. Document the order of ConsumedCapacity entries in transaction responses. With two tables the service returned the same table first for both request orders (measured 2026-10-23), so the order does not follow the request; ExtendDB keeps first appearance in the request on every backend. No wire test asserts a multi-table order. --- crates/core/src/expression/mod.rs | 8 +- crates/core/src/expression/resolver.rs | 2 +- crates/engine/src/capacity_helpers.rs | 18 ++-- crates/engine/src/expression_helpers.rs | 23 ----- crates/engine/src/legacy_filter.rs | 111 +++++++++++++++++++++++ crates/engine/src/query.rs | 6 +- crates/engine/src/scan.rs | 6 +- docs/differences-from-dynamodb.md | 1 + tests/test_expr_operand_type_and_size.py | 50 ++++++---- tests/test_expression_semantics.py | 23 +++++ 10 files changed, 189 insertions(+), 59 deletions(-) diff --git a/crates/core/src/expression/mod.rs b/crates/core/src/expression/mod.rs index ff34a68b5..3f00f1730 100755 --- a/crates/core/src/expression/mod.rs +++ b/crates/core/src/expression/mod.rs @@ -31,10 +31,10 @@ pub use parser::{parse_condition, parse_condition_with_depth_limit}; pub use projection::{Projection, parse_projection}; pub use reserved_words::validate_no_reserved_words; pub use resolver::{ - ExpressionMaps, collect_key_condition_refs, collect_value_placeholders, resolve_element_name, - resolve_name_ref, resolve_path, validate_begins_with_operands, validate_expression_param_usage, - validate_ordering_operand_types, validate_refs_defined, validate_unused_attributes, - validate_update_paths_disjoint, + ExpressionMaps, attribute_type_code, collect_key_condition_refs, collect_value_placeholders, + resolve_element_name, resolve_name_ref, resolve_path, validate_begins_with_operands, + validate_expression_param_usage, validate_ordering_operand_types, validate_refs_defined, + validate_unused_attributes, validate_update_paths_disjoint, }; pub use search_condition::{ MAX_INLINE_FILTER_CONDITIONS, MAX_PARTITION_KEY_CONDITIONS, MAX_SEARCH_CONDITIONS, diff --git a/crates/core/src/expression/resolver.rs b/crates/core/src/expression/resolver.rs index a29fb09cf..1fbe0fdfb 100755 --- a/crates/core/src/expression/resolver.rs +++ b/crates/core/src/expression/resolver.rs @@ -586,7 +586,7 @@ pub fn validate_begins_with_operands( /// Return the DynamoDB type code for an AttributeValue (S, N, B, BOOL, NULL, /// L, M, SS, NS, BS). Used in operand-type validation messages. -pub(crate) fn attribute_type_code(val: &AttributeValue) -> &'static str { +pub fn attribute_type_code(val: &AttributeValue) -> &'static str { match val { AttributeValue::S(_) => "S", AttributeValue::N(_) => "N", diff --git a/crates/engine/src/capacity_helpers.rs b/crates/engine/src/capacity_helpers.rs index 5bb003304..ff98b1146 100755 --- a/crates/engine/src/capacity_helpers.rs +++ b/crates/engine/src/capacity_helpers.rs @@ -385,14 +385,16 @@ pub fn batch_write_capacity<'a>( /// /// This is a firmer guarantee than Amazon DynamoDB gives, not a copy of its /// order. The service documents no order for the `ConsumedCapacity` entries -/// of a transaction, and the measured order (2026-10-14) is neither first -/// appearance nor alphabetical: a TransactGetItems naming table b, then a, -/// then b again returned a before b; a TransactWriteItems naming b, a2, b -/// returned b before a2. A client cannot depend on the service's order, so -/// any deterministic order is compatible with it. Before this change ExtendDB -/// aggregated into a `HashMap`, so the order changed from one process to the -/// next and differed between backends; first appearance is stable, matches -/// the request as the caller wrote it, and lets the tests assert the array. +/// of a transaction, and the measured order (2026-10-23) does not follow the +/// request: with two tables, a TransactGetItems naming table 2, then 1, then +/// 2 and one naming 1, 2, 1 both returned table 1 before table 2, and the two +/// TransactWriteItems with the same shapes did the same. A client cannot +/// depend on the service's position of an entry, so any deterministic order +/// is compatible with it (docs/differences-from-dynamodb.md, Capacity and +/// Throttling). Before this change ExtendDB aggregated into a `HashMap`, so +/// the order changed from one process to the next and differed between +/// backends; first appearance is stable, matches the request as the caller +/// wrote it, and lets the tests assert the array. #[derive(Default)] pub struct PerTableCapacity(Vec<(String, f64)>); diff --git a/crates/engine/src/expression_helpers.rs b/crates/engine/src/expression_helpers.rs index 115380c67..5d02dd41d 100755 --- a/crates/engine/src/expression_helpers.rs +++ b/crates/engine/src/expression_helpers.rs @@ -378,29 +378,6 @@ pub fn check_key_condition_refs( Ok(()) } -/// Reject a desugared legacy `QueryFilter` or `ScanFilter` whose -/// `BEGINS_WITH` value is not a string or binary, or whose ordering -/// comparison (`LT`, `LE`, `GT`, `GE`, `BETWEEN`) has a non-orderable value, -/// at request time, before any row is read. The expression form gets these -/// checks from [`parse_condition_checked`]; the legacy form bypasses the -/// parser, so they run here over the desugared tree with the maps -/// `desugar_filter` produced. Behaviour and message are unchanged from before -/// the semantic pass existed. -/// -/// # Errors -/// -/// Returns `DynamoDbError::ValidationException` with the `Invalid -/// FilterExpression:` prefix. -pub fn check_legacy_filter_operand_types( - filter: &Expr, - maps: &ExpressionMaps, -) -> Result<(), DynamoDbError> { - extenddb_core::expression::validate_begins_with_operands(filter, maps) - .map_err(|e| prefix_expression_error(e, ExpressionKind::Filter))?; - extenddb_core::expression::validate_ordering_operand_types(filter, maps) - .map_err(|e| prefix_expression_error(e, ExpressionKind::Filter)) -} - /// Prefix an expression error with the expression type, matching `DynamoDB`'s format. /// /// `FilterExpression` shares the condition parser, so its errors arrive labelled diff --git a/crates/engine/src/legacy_filter.rs b/crates/engine/src/legacy_filter.rs index 27dbcd732..4560a8684 100644 --- a/crates/engine/src/legacy_filter.rs +++ b/crates/engine/src/legacy_filter.rs @@ -11,6 +11,7 @@ use std::collections::HashMap; use extenddb_core::error::DynamoDbError; use extenddb_core::expression::{ CompareOp, Expr, ExpressionMaps, KeyCondition, PathElement, SortKeyCondition, + attribute_type_code, }; use extenddb_core::types::{AttributeValue, Condition, ConditionalOperator}; @@ -208,12 +209,54 @@ pub fn desugar_filter( Ok((combined, maps)) } +/// Refuse a legacy `ComparisonOperator` whose `AttributeValueList` holds a +/// value of a type the operator does not take, with the service's message, +/// before the condition is desugared and before any row is read. +/// +/// The service checks this at request time, on an empty table too (measured +/// 2026-10-23: `BEGINS_WITH` with `N`, `LT` and `GT` with `BOOL`, and +/// `BETWEEN` with `L` then `N`, which names the `L`). `EQ` and `NE` take every +/// type; `NULL` and `NOT_NULL` take no value; `BEGINS_WITH` takes `S` and `B`; +/// the ordering operators, `BETWEEN`, `IN`, `CONTAINS` and `NOT_CONTAINS` take +/// `S`, `N` and `B`. The first value of a type the operator does not take is +/// the one named. +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` with the text +/// `One or more parameter values were invalid: ComparisonOperator is not +/// valid for AttributeValue type`. +pub fn check_legacy_operand_types( + operator: &str, + values: &[AttributeValue], +) -> Result<(), DynamoDbError> { + let allowed = |v: &AttributeValue| match operator { + "EQ" | "NE" | "NULL" | "NOT_NULL" => true, + "BEGINS_WITH" => matches!(v, AttributeValue::S(_) | AttributeValue::B(_)), + "LE" | "LT" | "GE" | "GT" | "BETWEEN" | "IN" | "CONTAINS" | "NOT_CONTAINS" => { + matches!( + v, + AttributeValue::S(_) | AttributeValue::N(_) | AttributeValue::B(_) + ) + } + _ => true, + }; + match values.iter().find(|v| !allowed(v)) { + Some(bad) => Err(DynamoDbError::ValidationException(format!( + "One or more parameter values were invalid: ComparisonOperator {operator} is not valid for {} AttributeValue type", + attribute_type_code(bad) + ))), + None => Ok(()), + } +} + fn desugar_one_filter_condition( path: &Expr, cond: &Condition, values: &mut HashMap, counter: &mut u32, ) -> Result { + check_legacy_operand_types(&cond.comparison_operator, &cond.attribute_value_list)?; match cond.comparison_operator.as_str() { "EQ" | "NE" | "LE" | "LT" | "GE" | "GT" => { if cond.attribute_value_list.len() != 1 { @@ -333,3 +376,71 @@ fn desugar_one_filter_condition( ))), } } + +#[cfg(test)] +mod tests { + use super::*; + + fn msg(op: &str, values: &[AttributeValue]) -> Option { + match check_legacy_operand_types(op, values) { + Ok(()) => None, + Err(DynamoDbError::ValidationException(m)) => Some(m), + Err(other) => panic!("unexpected error {other:?}"), + } + } + + fn refusal(op: &str, ty: &str) -> String { + format!( + "One or more parameter values were invalid: ComparisonOperator {op} is not valid for {ty} AttributeValue type" + ) + } + + #[test] + fn measured_rows_carry_the_service_message() { + let n = AttributeValue::N("1".to_owned()); + let b = AttributeValue::Bool(true); + let l = AttributeValue::L(Vec::new()); + assert_eq!( + msg("BEGINS_WITH", std::slice::from_ref(&n)), + Some(refusal("BEGINS_WITH", "N")) + ); + assert_eq!( + msg("LT", std::slice::from_ref(&b)), + Some(refusal("LT", "BOOL")) + ); + assert_eq!(msg("GT", &[b]), Some(refusal("GT", "BOOL"))); + assert_eq!(msg("BETWEEN", &[l, n]), Some(refusal("BETWEEN", "L"))); + } + + #[test] + fn accepted_types_pass() { + let s = AttributeValue::S("p".to_owned()); + let n = AttributeValue::N("1".to_owned()); + let bin = AttributeValue::B(vec![1]); + assert_eq!(msg("BEGINS_WITH", std::slice::from_ref(&s)), None); + assert_eq!(msg("BEGINS_WITH", std::slice::from_ref(&bin)), None); + assert_eq!(msg("BETWEEN", &[n.clone(), n.clone()]), None); + assert_eq!(msg("IN", &[s.clone(), n.clone(), bin]), None); + assert_eq!(msg("EQ", &[AttributeValue::L(Vec::new())]), None); + assert_eq!(msg("NE", &[AttributeValue::Bool(false)]), None); + assert_eq!(msg("NULL", &[]), None); + assert_eq!(msg("CONTAINS", &[n]), None); + } + + #[test] + fn desugar_filter_refuses_before_desugaring() { + let mut filter = HashMap::new(); + filter.insert( + "f".to_owned(), + Condition { + comparison_operator: "BEGINS_WITH".to_owned(), + attribute_value_list: vec![AttributeValue::N("1".to_owned())], + }, + ); + let err = desugar_filter(&filter, ConditionalOperator::And).unwrap_err(); + assert!( + matches!(&err, DynamoDbError::ValidationException(m) if *m == refusal("BEGINS_WITH", "N")), + "{err:?}" + ); + } +} diff --git a/crates/engine/src/query.rs b/crates/engine/src/query.rs index d0b8058c3..1e01ebab0 100755 --- a/crates/engine/src/query.rs +++ b/crates/engine/src/query.rs @@ -153,13 +153,13 @@ pub async fn handle_query( // types), the KeyConditionExpression (syntax, then references), the // ProjectionExpression (syntax, then references and overlap), then unused // names and values. A missing table is reported after all of these. - // A legacy QueryFilter skips the parser, so its operand types are checked - // separately, still before any row is read. + // A legacy QueryFilter skips the parser; desugar_filter checks each + // ComparisonOperator against its value types with the service's message, + // still before any row is read. let (filter, filter_maps) = if let Some(qf) = input.query_filter.as_ref().filter(|qf| !qf.is_empty()) { let cond_op = input.conditional_operator.unwrap_or_default(); let (expr, fmaps) = desugar_filter(qf, cond_op)?; - crate::expression_helpers::check_legacy_filter_operand_types(&expr, &fmaps)?; (Some(expr), Some(fmaps)) } else { match input.filter_expression.as_deref() { diff --git a/crates/engine/src/scan.rs b/crates/engine/src/scan.rs index dee37cc82..ffcfb769e 100755 --- a/crates/engine/src/scan.rs +++ b/crates/engine/src/scan.rs @@ -183,13 +183,13 @@ pub async fn handle_scan( // operand types) here, ahead of the projection, the unused check, and the // table lookup, so an undefined `:q` is a request error even when no row // would be evaluated (measured 2026-10-14 on an empty table). A legacy - // ScanFilter skips the parser, so its operand types are checked - // separately, still before any row is read. + // ScanFilter skips the parser; desugar_filter checks each + // ComparisonOperator against its value types with the service's message, + // still before any row is read. let (filter, filter_maps) = if let Some(sf) = input.scan_filter.as_ref().filter(|sf| !sf.is_empty()) { let cond_op = input.conditional_operator.unwrap_or_default(); let (expr, fmaps) = desugar_filter(sf, cond_op)?; - crate::expression_helpers::check_legacy_filter_operand_types(&expr, &fmaps)?; (Some(expr), Some(fmaps)) } else { match input.filter_expression.as_deref() { diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 2603cc3b5..39ff139ad 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -147,6 +147,7 @@ The same on all four backends. | Provisioned throughput | Token bucket per table/partition | Token bucket per table/partition, matching DynamoDB's burst and refill behavior | | On-demand capacity | Automatic scaling | Fixed initial burst capacity (4000 WCU / 12000 RCU), no auto-scaling | | Throttling | Always on; throttles requests that exceed provisioned/burst capacity. No setting to disable | Configurable via `throttling_enabled` runtime setting (default: `true`) | +| Order of `ConsumedCapacity` entries in `TransactGetItems` and `TransactWriteItems` responses | Undocumented. Measured 2026-10-23 with two tables: the same table came first whether it was named first or second in the request (a request naming `bb-w-M3-2`, `bb-w-M3-1`, `bb-w-M3-2` and one naming `bb-w-M3-1`, `bb-w-M3-2`, `bb-w-M3-1` both returned `bb-w-M3-1` before `bb-w-M3-2`), so the order does not follow the request; it is consistent with the table name order, which is not confirmed | One entry per table in the order the tables first appear in the request, on every backend. A client that reads the array by position works against ExtendDB and may not against DynamoDB; look entries up by `TableName` | ## Operations Not Implemented diff --git a/tests/test_expr_operand_type_and_size.py b/tests/test_expr_operand_type_and_size.py index 1f9a7a7c1..81f2cb966 100644 --- a/tests/test_expr_operand_type_and_size.py +++ b/tests/test_expr_operand_type_and_size.py @@ -368,37 +368,53 @@ def empty_table(dynamodb_client): dynamodb_client.delete_table(TableName=name) +LEGACY_TYPE = ( + "One or more parameter values were invalid: ComparisonOperator {} is not valid for {} AttributeValue type" +) +KC_ABSENT = {"pk": {"ComparisonOperator": "EQ", "AttributeValueList": [{"S": "absent"}]}} + + class TestLegacyFilterOperandType: """The legacy QueryFilter and ScanFilter parameters are checked at request - time too: on an empty table, where no row is evaluated, BEGINS_WITH with a - number and an ordering comparison with a boolean are refused. Only the - error code is asserted here; the service's message text for the legacy - parameters has not been captured.""" - - def test_scan_filter_begins_with_number_rejected_on_empty_table(self, dynamodb_client, empty_table): - with pytest.raises(ClientError) as ei: - dynamodb_client.scan( - TableName=empty_table, - ScanFilter={"pk": {"ComparisonOperator": "BEGINS_WITH", "AttributeValueList": [{"N": "1"}]}}, - ) - assert _err(ei)["Code"] == "ValidationException" + time too: on an empty table, where no row is evaluated, an operator given + a value of a type it does not take is refused with the service's message + (measured against Amazon DynamoDB on 2026-10-23). The value named is the + first one of a type the operator does not take.""" - def test_scan_filter_ordering_boolean_rejected_on_empty_table(self, dynamodb_client, empty_table): + @pytest.mark.parametrize( + ("operator", "values", "named_type"), + [ + ("BEGINS_WITH", [{"N": "1"}], "N"), + ("LT", [{"BOOL": True}], "BOOL"), + ("GT", [{"BOOL": True}], "BOOL"), + ("BETWEEN", [{"L": []}, {"N": "1"}], "L"), + ], + ) + def test_scan_filter_refuses_with_the_service_message(self, dynamodb_client, empty_table, operator, values, named_type): with pytest.raises(ClientError) as ei: dynamodb_client.scan( TableName=empty_table, - ScanFilter={"n": {"ComparisonOperator": "LT", "AttributeValueList": [{"BOOL": True}]}}, + ScanFilter={"f": {"ComparisonOperator": operator, "AttributeValueList": values}}, ) assert _err(ei)["Code"] == "ValidationException" + assert _err(ei)["Message"] == LEGACY_TYPE.format(operator, named_type) - def test_query_filter_begins_with_number_rejected_on_empty_table(self, dynamodb_client, empty_table): + @pytest.mark.parametrize( + ("operator", "values", "named_type"), + [ + ("BEGINS_WITH", [{"N": "1"}], "N"), + ("GT", [{"BOOL": True}], "BOOL"), + ], + ) + def test_query_filter_refuses_with_the_service_message(self, dynamodb_client, empty_table, operator, values, named_type): with pytest.raises(ClientError) as ei: dynamodb_client.query( TableName=empty_table, - KeyConditions={"pk": {"ComparisonOperator": "EQ", "AttributeValueList": [{"S": "absent"}]}}, - QueryFilter={"s": {"ComparisonOperator": "BEGINS_WITH", "AttributeValueList": [{"N": "1"}]}}, + KeyConditions=KC_ABSENT, + QueryFilter={"f": {"ComparisonOperator": operator, "AttributeValueList": values}}, ) assert _err(ei)["Code"] == "ValidationException" + assert _err(ei)["Message"] == LEGACY_TYPE.format(operator, named_type) def test_scan_filter_begins_with_string_accepted(self, dynamodb_client, empty_table): resp = dynamodb_client.scan( diff --git a/tests/test_expression_semantics.py b/tests/test_expression_semantics.py index 5c565c4c7..3006cf608 100644 --- a/tests/test_expression_semantics.py +++ b/tests/test_expression_semantics.py @@ -492,6 +492,29 @@ def test_put_item_refuses_with_the_prefix(self, dynamodb_client, table, conditio def test_function_name_beats_unused(self, dynamodb_client, table): _expect(_put_cond(dynamodb_client, table, "foo(pk)", {":u": N}), P + C + INVALID_FN.format("foo")) + @pytest.mark.parametrize( + ("condition", "values"), + [ + ("foo(pk) AND", None), + ("ATTRIBUTE_EXISTS(pk) AND", None), + ("begins_with(pk, :n) AND", {":n": N}), + ("attribute_exists(#x) AND", None), + ], + ) + def test_a_syntax_error_beats_the_semantic_errors(self, dynamodb_client, table, condition, values): + # The service answers 'Syntax error; token: "", near: "AND"' for each of + # these (measured 2026-10-23): the grammar pass runs over the whole text + # before the function name, the operand type or the undefined name is + # looked at. ExtendDB's syntax wording differs today (tracked under #108 + # message text); the guarantee pinned here is the order. + with pytest.raises(ClientError) as exc_info: + _put_cond(dynamodb_client, table, condition, values)() + err = exc_info.value.response["Error"] + assert err["Code"] == "ValidationException", err + assert err["Message"].startswith(P + C), err + for semantic in ("Invalid function name", "Incorrect operand type", "is not defined"): + assert semantic not in err["Message"], err + @pytest.mark.parametrize( ("condition", "values"), [ From 41323942c984dff21b71e6063a50ec794104078f Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 14:38:41 +0000 Subject: [PATCH 04/10] fix(expressions): validate legacy conditions with the service's rules and messages Measured the full ComparisonOperator by AttributeValue type grid against Amazon DynamoDB on 2026-10-23 for ScanFilter, QueryFilter and KeyConditions, with the value counts, mixed-type lists, inverted BETWEEN ranges and unknown operator names. validate_legacy_condition in legacy_filter.rs now applies the measured rules to every legacy condition before it is desugared and before the table is looked up: the operator name (`Invalid comparison operator`), the value count (`Invalid number of argument(s) for the ComparisonOperator`), the first value's type (EQ and NE take every type; BEGINS_WITH takes S and B; CONTAINS and NOT_CONTAINS take S, N, B, BOOL and NULL; LE, LT, GE, GT, BETWEEN and IN take S, N and B), that every value has the first value's type (`AttributeValues inside AttributeValueList must be of same type`), and that a BETWEEN lower bound is not above its upper bound. Each message carries the `One or more parameter values were invalid: ` prefix. KeyConditions get the same checks ahead of the table lookup, and a range key operator the service does not index (NE, NULL, NOT_NULL, CONTAINS, NOT_CONTAINS, IN) is refused with `Attempted conditional constraint is not an indexable operation`. The engine's own count messages for these parameters are gone. compare_orderable is exported from extenddb-core for the range check. The grid is pinned in legacy_filter_tests.rs and one wire row per message in TestLegacyFilterOperandType. The ConsumedCapacity order note now cites the three-table measurement: 16 transactions in eight request orders all listed the tables in name order. --- crates/core/src/expression/evaluator.rs | 17 ++ crates/core/src/expression/mod.rs | 2 +- crates/engine/src/capacity_helpers.rs | 20 +- crates/engine/src/legacy_filter.rs | 264 ++++++++++------------- crates/engine/src/legacy_filter_tests.rs | 245 +++++++++++++++++++++ crates/engine/src/query.rs | 12 +- crates/engine/src/scan.rs | 4 +- docs/differences-from-dynamodb.md | 2 +- tests/test_expr_operand_type_and_size.py | 202 +++++++++++++---- 9 files changed, 557 insertions(+), 211 deletions(-) create mode 100644 crates/engine/src/legacy_filter_tests.rs diff --git a/crates/core/src/expression/evaluator.rs b/crates/core/src/expression/evaluator.rs index e023af96e..cee1186ea 100755 --- a/crates/core/src/expression/evaluator.rs +++ b/crates/core/src/expression/evaluator.rs @@ -141,6 +141,23 @@ fn placeholder_numeric<'a>( } } +/// Order two values of the same orderable type: `S` and `B` bytewise, `N` +/// numerically. Returns `None` for any other pair (different types, or a type +/// with no order). +#[must_use] +pub fn compare_orderable(left: &AttributeValue, right: &AttributeValue) -> Option { + match (left, right) { + (AttributeValue::S(l), AttributeValue::S(r)) => Some(l.cmp(r)), + (AttributeValue::B(l), AttributeValue::B(r)) => Some(l.cmp(r)), + (AttributeValue::N(l), AttributeValue::N(r)) => { + let l = l.parse::().ok()?; + let r = r.parse::().ok()?; + Some(l.cmp(&r)) + } + _ => None, + } +} + /// Compare two `AttributeValue`s using the given operator. /// /// `DynamoDB` comparison rules: diff --git a/crates/core/src/expression/mod.rs b/crates/core/src/expression/mod.rs index 3f00f1730..b3be127eb 100755 --- a/crates/core/src/expression/mod.rs +++ b/crates/core/src/expression/mod.rs @@ -24,7 +24,7 @@ mod update_evaluator; mod update_parser; pub use ast::{ArithOp, CompareOp, Expr, PathElement, UpdateAction}; -pub use evaluator::evaluate_condition; +pub use evaluator::{compare_orderable, evaluate_condition}; pub use key_condition::{KeyCondition, SortKeyCondition, parse_key_condition}; pub use kind::ExpressionKind; pub use parser::{parse_condition, parse_condition_with_depth_limit}; diff --git a/crates/engine/src/capacity_helpers.rs b/crates/engine/src/capacity_helpers.rs index ff98b1146..a0c113b96 100755 --- a/crates/engine/src/capacity_helpers.rs +++ b/crates/engine/src/capacity_helpers.rs @@ -385,16 +385,16 @@ pub fn batch_write_capacity<'a>( /// /// This is a firmer guarantee than Amazon DynamoDB gives, not a copy of its /// order. The service documents no order for the `ConsumedCapacity` entries -/// of a transaction, and the measured order (2026-10-23) does not follow the -/// request: with two tables, a TransactGetItems naming table 2, then 1, then -/// 2 and one naming 1, 2, 1 both returned table 1 before table 2, and the two -/// TransactWriteItems with the same shapes did the same. A client cannot -/// depend on the service's position of an entry, so any deterministic order -/// is compatible with it (docs/differences-from-dynamodb.md, Capacity and -/// Throttling). Before this change ExtendDB aggregated into a `HashMap`, so -/// the order changed from one process to the next and differed between -/// backends; first appearance is stable, matches the request as the caller -/// wrote it, and lets the tests assert the array. +/// of a transaction, and the measured order (2026-10-23, three tables, eight +/// request orders, TransactGetItems and TransactWriteItems) did not follow +/// the request: every response listed the tables in their name order, which +/// was also their creation order. A client cannot depend on the position of +/// an entry against the service, so any deterministic order is compatible +/// with it (docs/differences-from-dynamodb.md, Capacity and Throttling). +/// Before this change ExtendDB aggregated into a `HashMap`, so the order +/// changed from one process to the next and differed between backends; first +/// appearance is stable, matches the request as the caller wrote it, and lets +/// the tests assert the array. #[derive(Default)] pub struct PerTableCapacity(Vec<(String, f64)>); diff --git a/crates/engine/src/legacy_filter.rs b/crates/engine/src/legacy_filter.rs index 4560a8684..30a3c00a3 100644 --- a/crates/engine/src/legacy_filter.rs +++ b/crates/engine/src/legacy_filter.rs @@ -6,12 +6,13 @@ //! Converts pre-expression parameters (`KeyConditions`, `QueryFilter`, `ScanFilter`) //! into expression-based equivalents. +use std::cmp::Ordering; use std::collections::HashMap; use extenddb_core::error::DynamoDbError; use extenddb_core::expression::{ CompareOp, Expr, ExpressionMaps, KeyCondition, PathElement, SortKeyCondition, - attribute_type_code, + attribute_type_code, compare_orderable, }; use extenddb_core::types::{AttributeValue, Condition, ConditionalOperator}; @@ -32,6 +33,8 @@ pub fn desugar_key_conditions( DynamoDbError::ValidationException("No hash key in key schema".to_owned()) })?; + validate_legacy_conditions(conditions)?; + let hash_cond = conditions.get(hash_attr).ok_or_else(|| { DynamoDbError::ValidationException(format!( "Query condition missed key schema element: {hash_attr}" @@ -44,12 +47,6 @@ pub fn desugar_key_conditions( )); } - if hash_cond.attribute_value_list.len() != 1 { - return Err(DynamoDbError::ValidationException( - "Query key condition must have exactly one value for EQ".to_owned(), - )); - } - let mut values = HashMap::new(); let pk_placeholder = "_kc_pk".to_owned(); values.insert( @@ -107,11 +104,6 @@ fn desugar_sk_condition( let path = vec![PathElement::Attribute(range_name.to_owned())]; match cond.comparison_operator.as_str() { "EQ" | "LE" | "LT" | "GE" | "GT" => { - if cond.attribute_value_list.len() != 1 { - return Err(DynamoDbError::ValidationException( - "Invalid number of values for comparison".to_owned(), - )); - } let placeholder = "_kc_sk".to_owned(); values.insert(placeholder.clone(), cond.attribute_value_list[0].clone()); let op = match cond.comparison_operator.as_str() { @@ -129,11 +121,6 @@ fn desugar_sk_condition( }) } "BETWEEN" => { - if cond.attribute_value_list.len() != 2 { - return Err(DynamoDbError::ValidationException( - "BETWEEN requires exactly 2 values".to_owned(), - )); - } let lo_placeholder = "_kc_sk_lo".to_owned(); let hi_placeholder = "_kc_sk_hi".to_owned(); values.insert(lo_placeholder.clone(), cond.attribute_value_list[0].clone()); @@ -145,11 +132,6 @@ fn desugar_sk_condition( }) } "BEGINS_WITH" => { - if cond.attribute_value_list.len() != 1 { - return Err(DynamoDbError::ValidationException( - "BEGINS_WITH requires exactly 1 value".to_owned(), - )); - } let placeholder = "_kc_sk".to_owned(); values.insert(placeholder.clone(), cond.attribute_value_list[0].clone()); Ok(SortKeyCondition::BeginsWith { @@ -157,8 +139,10 @@ fn desugar_sk_condition( prefix: Expr::Placeholder(placeholder), }) } - other => Err(DynamoDbError::ValidationException(format!( - "Unsupported KeyCondition operator: {other}" + // NE, NULL, NOT_NULL, CONTAINS, NOT_CONTAINS and IN pass the value + // checks and are then refused on a key (measured 2026-10-23). + _ => Err(DynamoDbError::ValidationException(format!( + "{INVALID_PARAMS}Attempted conditional constraint is not an indexable operation" ))), } } @@ -209,45 +193,114 @@ pub fn desugar_filter( Ok((combined, maps)) } -/// Refuse a legacy `ComparisonOperator` whose `AttributeValueList` holds a -/// value of a type the operator does not take, with the service's message, -/// before the condition is desugared and before any row is read. +const INVALID_PARAMS: &str = "One or more parameter values were invalid: "; + +/// Validate one legacy condition (`ScanFilter`, `QueryFilter` or +/// `KeyConditions` entry) the way the service does at request time, before +/// the table is looked up and before any row is read, with its messages +/// (measured 2026-10-23 on empty tables). In order: /// -/// The service checks this at request time, on an empty table too (measured -/// 2026-10-23: `BEGINS_WITH` with `N`, `LT` and `GT` with `BOOL`, and -/// `BETWEEN` with `L` then `N`, which names the `L`). `EQ` and `NE` take every -/// type; `NULL` and `NOT_NULL` take no value; `BEGINS_WITH` takes `S` and `B`; -/// the ordering operators, `BETWEEN`, `IN`, `CONTAINS` and `NOT_CONTAINS` take -/// `S`, `N` and `B`. The first value of a type the operator does not take is -/// the one named. +/// 1. the operator is one of the thirteen (`Invalid comparison operator`); +/// 2. the value count fits the operator: none for `NULL` and `NOT_NULL`, two +/// for `BETWEEN`, one or more for `IN`, one otherwise (`Invalid number of +/// argument(s) for the ComparisonOperator`); +/// 3. the first value's type is one the operator takes: `EQ` and `NE` take +/// every type, `BEGINS_WITH` takes `S` and `B`, `CONTAINS` and +/// `NOT_CONTAINS` take `S`, `N`, `B`, `BOOL` and `NULL`, the ordering +/// operators, `BETWEEN` and `IN` take `S`, `N` and `B` (`ComparisonOperator +/// is not valid for AttributeValue type`); +/// 4. every value has the first value's type (`AttributeValues inside +/// AttributeValueList must be of same type`); +/// 5. a `BETWEEN` lower bound is not above its upper bound (`The BETWEEN +/// condition was provided a range where the lower bound is greater than +/// the upper bound`). +/// +/// The order of step 2 against step 3 within one condition was not measured; +/// each was measured against a well-formed other. /// /// # Errors /// -/// Returns `DynamoDbError::ValidationException` with the text -/// `One or more parameter values were invalid: ComparisonOperator is not -/// valid for AttributeValue type`. -pub fn check_legacy_operand_types( +/// Returns `DynamoDbError::ValidationException` with the service's text, +/// prefixed `One or more parameter values were invalid: `. +pub fn validate_legacy_condition( operator: &str, values: &[AttributeValue], ) -> Result<(), DynamoDbError> { - let allowed = |v: &AttributeValue| match operator { - "EQ" | "NE" | "NULL" | "NOT_NULL" => true, - "BEGINS_WITH" => matches!(v, AttributeValue::S(_) | AttributeValue::B(_)), - "LE" | "LT" | "GE" | "GT" | "BETWEEN" | "IN" | "CONTAINS" | "NOT_CONTAINS" => { - matches!( - v, - AttributeValue::S(_) | AttributeValue::N(_) | AttributeValue::B(_) - ) + let refuse = |text: String| { + Err(DynamoDbError::ValidationException(format!( + "{INVALID_PARAMS}{text}" + ))) + }; + let count_ok = match operator { + "NULL" | "NOT_NULL" => values.is_empty(), + "BETWEEN" => values.len() == 2, + "IN" => !values.is_empty(), + "EQ" | "NE" | "LE" | "LT" | "GE" | "GT" | "CONTAINS" | "NOT_CONTAINS" | "BEGINS_WITH" => { + values.len() == 1 } - _ => true, + _ => return refuse("Invalid comparison operator".to_owned()), }; - match values.iter().find(|v| !allowed(v)) { - Some(bad) => Err(DynamoDbError::ValidationException(format!( - "One or more parameter values were invalid: ComparisonOperator {operator} is not valid for {} AttributeValue type", - attribute_type_code(bad) - ))), - None => Ok(()), + if !count_ok { + return refuse(format!( + "Invalid number of argument(s) for the {operator} ComparisonOperator" + )); + } + let Some(first) = values.first() else { + return Ok(()); + }; + let scalar = matches!( + first, + AttributeValue::S(_) | AttributeValue::N(_) | AttributeValue::B(_) + ); + let type_ok = match operator { + "EQ" | "NE" => true, + "BEGINS_WITH" => matches!(first, AttributeValue::S(_) | AttributeValue::B(_)), + "CONTAINS" | "NOT_CONTAINS" => { + scalar || matches!(first, AttributeValue::Bool(_) | AttributeValue::Null) + } + _ => scalar, + }; + if !type_ok { + return refuse(format!( + "ComparisonOperator {operator} is not valid for {} AttributeValue type", + attribute_type_code(first) + )); + } + if values + .iter() + .any(|v| attribute_type_code(v) != attribute_type_code(first)) + { + return refuse("AttributeValues inside AttributeValueList must be of same type".to_owned()); + } + if operator == "BETWEEN" && compare_orderable(&values[0], &values[1]) == Some(Ordering::Greater) + { + return refuse( + "The BETWEEN condition was provided a range where the lower bound is greater than the upper bound" + .to_owned(), + ); + } + Ok(()) +} + +/// Run [`validate_legacy_condition`] over every entry of a legacy condition +/// map, in attribute name order, so `KeyConditions` can be checked before the +/// table is looked up (the service refuses a bad `QueryFilter` value before it +/// reports a missing table; a bad `KeyConditions` value was measured ahead of +/// every key schema error). +/// +/// # Errors +/// +/// The first entry's error, in attribute name order. +pub fn validate_legacy_conditions( + conditions: &HashMap, +) -> Result<(), DynamoDbError> { + let mut names: Vec<&String> = conditions.keys().collect(); + names.sort(); + for name in names { + let cond = &conditions[name]; + validate_legacy_condition(&cond.comparison_operator, &cond.attribute_value_list)?; } + Ok(()) } fn desugar_one_filter_condition( @@ -256,14 +309,9 @@ fn desugar_one_filter_condition( values: &mut HashMap, counter: &mut u32, ) -> Result { - check_legacy_operand_types(&cond.comparison_operator, &cond.attribute_value_list)?; + validate_legacy_condition(&cond.comparison_operator, &cond.attribute_value_list)?; match cond.comparison_operator.as_str() { "EQ" | "NE" | "LE" | "LT" | "GE" | "GT" => { - if cond.attribute_value_list.len() != 1 { - return Err(DynamoDbError::ValidationException( - "Invalid number of attribute values for comparison".to_owned(), - )); - } let placeholder = format!("_f{counter}"); *counter += 1; values.insert(placeholder.clone(), cond.attribute_value_list[0].clone()); @@ -283,11 +331,6 @@ fn desugar_one_filter_condition( }) } "BETWEEN" => { - if cond.attribute_value_list.len() != 2 { - return Err(DynamoDbError::ValidationException( - "BETWEEN requires exactly 2 values".to_owned(), - )); - } let lo = format!("_f{counter}"); *counter += 1; let hi = format!("_f{counter}"); @@ -301,11 +344,6 @@ fn desugar_one_filter_condition( }) } "BEGINS_WITH" => { - if cond.attribute_value_list.len() != 1 { - return Err(DynamoDbError::ValidationException( - "BEGINS_WITH requires exactly 1 value".to_owned(), - )); - } let placeholder = format!("_f{counter}"); *counter += 1; values.insert(placeholder.clone(), cond.attribute_value_list[0].clone()); @@ -315,11 +353,6 @@ fn desugar_one_filter_condition( }) } "CONTAINS" => { - if cond.attribute_value_list.len() != 1 { - return Err(DynamoDbError::ValidationException( - "CONTAINS requires exactly 1 value".to_owned(), - )); - } let placeholder = format!("_f{counter}"); *counter += 1; values.insert(placeholder.clone(), cond.attribute_value_list[0].clone()); @@ -329,11 +362,6 @@ fn desugar_one_filter_condition( }) } "NOT_CONTAINS" => { - if cond.attribute_value_list.len() != 1 { - return Err(DynamoDbError::ValidationException( - "NOT_CONTAINS requires exactly 1 value".to_owned(), - )); - } let placeholder = format!("_f{counter}"); *counter += 1; values.insert(placeholder.clone(), cond.attribute_value_list[0].clone()); @@ -351,11 +379,6 @@ fn desugar_one_filter_condition( args: vec![path.clone()], }), "IN" => { - if cond.attribute_value_list.is_empty() { - return Err(DynamoDbError::ValidationException( - "IN requires at least 1 value".to_owned(), - )); - } let placeholders: Vec = cond .attribute_value_list .iter() @@ -371,76 +394,13 @@ fn desugar_one_filter_condition( list: placeholders, }) } - other => Err(DynamoDbError::ValidationException(format!( - "Unsupported filter operator: {other}" + // validate_legacy_condition refused every other operator above. + _ => Err(DynamoDbError::ValidationException(format!( + "{INVALID_PARAMS}Invalid comparison operator" ))), } } #[cfg(test)] -mod tests { - use super::*; - - fn msg(op: &str, values: &[AttributeValue]) -> Option { - match check_legacy_operand_types(op, values) { - Ok(()) => None, - Err(DynamoDbError::ValidationException(m)) => Some(m), - Err(other) => panic!("unexpected error {other:?}"), - } - } - - fn refusal(op: &str, ty: &str) -> String { - format!( - "One or more parameter values were invalid: ComparisonOperator {op} is not valid for {ty} AttributeValue type" - ) - } - - #[test] - fn measured_rows_carry_the_service_message() { - let n = AttributeValue::N("1".to_owned()); - let b = AttributeValue::Bool(true); - let l = AttributeValue::L(Vec::new()); - assert_eq!( - msg("BEGINS_WITH", std::slice::from_ref(&n)), - Some(refusal("BEGINS_WITH", "N")) - ); - assert_eq!( - msg("LT", std::slice::from_ref(&b)), - Some(refusal("LT", "BOOL")) - ); - assert_eq!(msg("GT", &[b]), Some(refusal("GT", "BOOL"))); - assert_eq!(msg("BETWEEN", &[l, n]), Some(refusal("BETWEEN", "L"))); - } - - #[test] - fn accepted_types_pass() { - let s = AttributeValue::S("p".to_owned()); - let n = AttributeValue::N("1".to_owned()); - let bin = AttributeValue::B(vec![1]); - assert_eq!(msg("BEGINS_WITH", std::slice::from_ref(&s)), None); - assert_eq!(msg("BEGINS_WITH", std::slice::from_ref(&bin)), None); - assert_eq!(msg("BETWEEN", &[n.clone(), n.clone()]), None); - assert_eq!(msg("IN", &[s.clone(), n.clone(), bin]), None); - assert_eq!(msg("EQ", &[AttributeValue::L(Vec::new())]), None); - assert_eq!(msg("NE", &[AttributeValue::Bool(false)]), None); - assert_eq!(msg("NULL", &[]), None); - assert_eq!(msg("CONTAINS", &[n]), None); - } - - #[test] - fn desugar_filter_refuses_before_desugaring() { - let mut filter = HashMap::new(); - filter.insert( - "f".to_owned(), - Condition { - comparison_operator: "BEGINS_WITH".to_owned(), - attribute_value_list: vec![AttributeValue::N("1".to_owned())], - }, - ); - let err = desugar_filter(&filter, ConditionalOperator::And).unwrap_err(); - assert!( - matches!(&err, DynamoDbError::ValidationException(m) if *m == refusal("BEGINS_WITH", "N")), - "{err:?}" - ); - } -} +#[path = "legacy_filter_tests.rs"] +mod tests; diff --git a/crates/engine/src/legacy_filter_tests.rs b/crates/engine/src/legacy_filter_tests.rs new file mode 100644 index 000000000..2005b4dea --- /dev/null +++ b/crates/engine/src/legacy_filter_tests.rs @@ -0,0 +1,245 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Tests for the legacy condition validator; every expected message was +//! measured against Amazon DynamoDB on 2026-10-23. + +use std::collections::BTreeMap; + +use super::*; + +fn msg(op: &str, values: &[AttributeValue]) -> Option { + match validate_legacy_condition(op, values) { + Ok(()) => None, + Err(DynamoDbError::ValidationException(m)) => Some(m), + Err(other) => panic!("unexpected error {other:?}"), + } +} + +fn refusal(text: &str) -> Option { + Some(format!("{INVALID_PARAMS}{text}")) +} + +fn type_refusal(op: &str, ty: &str) -> Option { + refusal(&format!( + "ComparisonOperator {op} is not valid for {ty} AttributeValue type" + )) +} + +fn one_of_each() -> Vec<(&'static str, AttributeValue)> { + vec![ + ("S", AttributeValue::S("a".to_owned())), + ("N", AttributeValue::N("1".to_owned())), + ("B", AttributeValue::B(vec![1])), + ("SS", AttributeValue::SS(["a".to_owned()].into())), + ("NS", AttributeValue::NS(["1".to_owned()].into())), + ("BS", AttributeValue::BS([vec![1]].into())), + ("BOOL", AttributeValue::Bool(true)), + ("NULL", AttributeValue::Null), + ("L", AttributeValue::L(Vec::new())), + ("M", AttributeValue::M(BTreeMap::new())), + ] +} + +/// The operator by type grid measured on 2026-10-23 (ScanFilter, +/// QueryFilter and KeyConditions gave the same rows). +#[test] +fn operator_by_type_grid() { + let accepted = |op: &str, ty: &str| match op { + "EQ" | "NE" => true, + "BEGINS_WITH" => matches!(ty, "S" | "B"), + "CONTAINS" | "NOT_CONTAINS" => matches!(ty, "S" | "N" | "B" | "BOOL" | "NULL"), + _ => matches!(ty, "S" | "N" | "B"), + }; + for op in [ + "EQ", + "NE", + "LE", + "LT", + "GE", + "GT", + "CONTAINS", + "NOT_CONTAINS", + "BEGINS_WITH", + "IN", + "BETWEEN", + ] { + for (ty, value) in one_of_each() { + let values = if op == "BETWEEN" { + vec![value.clone(), value] + } else { + vec![value] + }; + let expected = if accepted(op, ty) { + None + } else { + type_refusal(op, ty) + }; + assert_eq!(msg(op, &values), expected, "{op} with {ty}"); + } + } + for op in ["NULL", "NOT_NULL"] { + assert_eq!(msg(op, &[]), None); + } +} + +#[test] +fn value_counts() { + let s = AttributeValue::S("a".to_owned()); + let count = |op: &str| { + refusal(&format!( + "Invalid number of argument(s) for the {op} ComparisonOperator" + )) + }; + for op in [ + "EQ", + "NE", + "LE", + "LT", + "GE", + "GT", + "CONTAINS", + "NOT_CONTAINS", + "BEGINS_WITH", + ] { + assert_eq!(msg(op, &[]), count(op), "{op} with 0"); + assert_eq!(msg(op, &[s.clone(), s.clone()]), count(op), "{op} with 2"); + } + for op in ["NULL", "NOT_NULL"] { + assert_eq!(msg(op, std::slice::from_ref(&s)), count(op), "{op} with 1"); + } + assert_eq!(msg("IN", &[]), count("IN")); + assert_eq!(msg("IN", &[s.clone(), s.clone(), s.clone()]), None); + assert_eq!(msg("BETWEEN", std::slice::from_ref(&s)), count("BETWEEN")); + assert_eq!( + msg("BETWEEN", &[s.clone(), s.clone(), s.clone()]), + count("BETWEEN") + ); + assert_eq!( + msg("FOO", std::slice::from_ref(&s)), + refusal("Invalid comparison operator") + ); + assert_eq!( + msg("eq", std::slice::from_ref(&s)), + refusal("Invalid comparison operator") + ); +} + +#[test] +fn mixed_types_and_ranges() { + let s = AttributeValue::S("a".to_owned()); + let n = AttributeValue::N("1".to_owned()); + let l = AttributeValue::L(Vec::new()); + let b = AttributeValue::Bool(true); + let same = refusal("AttributeValues inside AttributeValueList must be of same type"); + // The first value's type is checked; the rest are held to it. + assert_eq!(msg("BETWEEN", &[s.clone(), n.clone()]), same); + assert_eq!(msg("BETWEEN", &[n.clone(), l.clone()]), same); + assert_eq!( + msg("BETWEEN", &[l.clone(), n.clone()]), + type_refusal("BETWEEN", "L") + ); + assert_eq!( + msg("BETWEEN", &[b, l.clone()]), + type_refusal("BETWEEN", "BOOL") + ); + assert_eq!(msg("IN", &[s.clone(), n.clone()]), same); + assert_eq!(msg("IN", &[l, s.clone()]), type_refusal("IN", "L")); + assert_eq!(msg("IN", &[s.clone(), AttributeValue::L(Vec::new())]), same); + let range = refusal( + "The BETWEEN condition was provided a range where the lower bound is greater than the upper bound", + ); + assert_eq!( + msg("BETWEEN", &[AttributeValue::N("2".to_owned()), n.clone()]), + range + ); + assert_eq!( + msg( + "BETWEEN", + &[ + AttributeValue::N("10".to_owned()), + AttributeValue::N("9".to_owned()) + ] + ), + range + ); + assert_eq!( + msg("BETWEEN", &[AttributeValue::S("b".to_owned()), s.clone()]), + range + ); + assert_eq!( + msg("BETWEEN", &[n.clone(), AttributeValue::N("10".to_owned())]), + None + ); + assert_eq!(msg("BETWEEN", &[n.clone(), n]), None); +} + +#[test] +fn desugar_filter_refuses_before_desugaring() { + let mut filter = HashMap::new(); + filter.insert( + "f".to_owned(), + Condition { + comparison_operator: "BEGINS_WITH".to_owned(), + attribute_value_list: vec![AttributeValue::N("1".to_owned())], + }, + ); + let err = desugar_filter(&filter, ConditionalOperator::And).unwrap_err(); + assert!( + matches!(&err, DynamoDbError::ValidationException(m) if Some(m.clone()) == type_refusal("BEGINS_WITH", "N")), + "{err:?}" + ); +} + +#[test] +fn key_conditions_refuse_non_indexable_operators_after_the_value_checks() { + let schema = vec![("pk".to_owned(), true), ("sk".to_owned(), false)]; + let eq = |v: AttributeValue| Condition { + comparison_operator: "EQ".to_owned(), + attribute_value_list: vec![v], + }; + for (op, values) in [ + ("NE", vec![AttributeValue::S("a".to_owned())]), + ("NULL", Vec::new()), + ("NOT_NULL", Vec::new()), + ("CONTAINS", vec![AttributeValue::S("a".to_owned())]), + ("NOT_CONTAINS", vec![AttributeValue::N("1".to_owned())]), + ( + "IN", + vec![ + AttributeValue::S("a".to_owned()), + AttributeValue::S("b".to_owned()), + ], + ), + ] { + let mut kc = HashMap::new(); + kc.insert("pk".to_owned(), eq(AttributeValue::S("p".to_owned()))); + kc.insert( + "sk".to_owned(), + Condition { + comparison_operator: op.to_owned(), + attribute_value_list: values, + }, + ); + let err = desugar_key_conditions(&kc, &schema).unwrap_err(); + assert!( + matches!(&err, DynamoDbError::ValidationException(m) if Some(m.clone()) == refusal("Attempted conditional constraint is not an indexable operation")), + "{op}: {err:?}" + ); + } + // A value the operator does not take is reported first, as is a count. + let mut kc = HashMap::new(); + kc.insert("pk".to_owned(), eq(AttributeValue::S("p".to_owned()))); + kc.insert( + "sk".to_owned(), + Condition { + comparison_operator: "CONTAINS".to_owned(), + attribute_value_list: vec![AttributeValue::L(Vec::new())], + }, + ); + let err = desugar_key_conditions(&kc, &schema).unwrap_err(); + assert!( + matches!(&err, DynamoDbError::ValidationException(m) if Some(m.clone()) == type_refusal("CONTAINS", "L")), + "{err:?}" + ); +} diff --git a/crates/engine/src/query.rs b/crates/engine/src/query.rs index 1e01ebab0..a611f2d49 100755 --- a/crates/engine/src/query.rs +++ b/crates/engine/src/query.rs @@ -24,7 +24,7 @@ use crate::index_helpers::{ VectorIndexReadRefusal, classify_unresolved_index_read, combined_lek_key_schema, validate_query_exclusive_start_key, }; -use crate::legacy_filter::{desugar_filter, desugar_key_conditions}; +use crate::legacy_filter::{desugar_filter, desugar_key_conditions, validate_legacy_conditions}; use crate::read_helpers::{PostRead, read_page}; use crate::serialize_output; use crate::{DispatchMetrics, DispatchResult}; @@ -154,8 +154,11 @@ pub async fn handle_query( // ProjectionExpression (syntax, then references and overlap), then unused // names and values. A missing table is reported after all of these. // A legacy QueryFilter skips the parser; desugar_filter checks each - // ComparisonOperator against its value types with the service's message, - // still before any row is read. + // ComparisonOperator against its value count and types with the service's + // messages, still before any row is read. Legacy KeyConditions get the + // same value checks here, ahead of the table lookup; the service reported + // a bad QueryFilter value before a bad KeyConditions value, and both + // before any key schema error (measured 2026-10-23). let (filter, filter_maps) = if let Some(qf) = input.query_filter.as_ref().filter(|qf| !qf.is_empty()) { let cond_op = input.conditional_operator.unwrap_or_default(); @@ -176,6 +179,9 @@ pub async fn handle_query( _ => (None, None), } }; + if let Some(kc) = input.key_conditions.as_ref() { + validate_legacy_conditions(kc)?; + } let parsed_key_condition = match input.key_condition_expression.as_deref() { Some(kce_str) => { diff --git a/crates/engine/src/scan.rs b/crates/engine/src/scan.rs index ffcfb769e..a71565a48 100755 --- a/crates/engine/src/scan.rs +++ b/crates/engine/src/scan.rs @@ -184,8 +184,8 @@ pub async fn handle_scan( // table lookup, so an undefined `:q` is a request error even when no row // would be evaluated (measured 2026-10-14 on an empty table). A legacy // ScanFilter skips the parser; desugar_filter checks each - // ComparisonOperator against its value types with the service's message, - // still before any row is read. + // ComparisonOperator against its value count and types with the service's + // messages, still before any row is read. let (filter, filter_maps) = if let Some(sf) = input.scan_filter.as_ref().filter(|sf| !sf.is_empty()) { let cond_op = input.conditional_operator.unwrap_or_default(); diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 39ff139ad..38c14c036 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -147,7 +147,7 @@ The same on all four backends. | Provisioned throughput | Token bucket per table/partition | Token bucket per table/partition, matching DynamoDB's burst and refill behavior | | On-demand capacity | Automatic scaling | Fixed initial burst capacity (4000 WCU / 12000 RCU), no auto-scaling | | Throttling | Always on; throttles requests that exceed provisioned/burst capacity. No setting to disable | Configurable via `throttling_enabled` runtime setting (default: `true`) | -| Order of `ConsumedCapacity` entries in `TransactGetItems` and `TransactWriteItems` responses | Undocumented. Measured 2026-10-23 with two tables: the same table came first whether it was named first or second in the request (a request naming `bb-w-M3-2`, `bb-w-M3-1`, `bb-w-M3-2` and one naming `bb-w-M3-1`, `bb-w-M3-2`, `bb-w-M3-1` both returned `bb-w-M3-1` before `bb-w-M3-2`), so the order does not follow the request; it is consistent with the table name order, which is not confirmed | One entry per table in the order the tables first appear in the request, on every backend. A client that reads the array by position works against ExtendDB and may not against DynamoDB; look entries up by `TableName` | +| Order of `ConsumedCapacity` entries in `TransactGetItems` and `TransactWriteItems` responses | Undocumented. Measured 2026-10-23 with three tables in eight request orders (16 transactions): the entries came back in the same order every time, the tables' name order, whatever order the request named them in (a request naming `bb-w-W2-1`, `bb-w-M3-2`, `bb-w-M3-1` returned `bb-w-M3-1`, `bb-w-M3-2`, `bb-w-W2-1`). The three tables were also created in name order, so creation order fits the same data | One entry per table in the order the tables first appear in the request, on every backend. A client that reads the array by position may see a different order against DynamoDB; look entries up by `TableName` | ## Operations Not Implemented diff --git a/tests/test_expr_operand_type_and_size.py b/tests/test_expr_operand_type_and_size.py index 81f2cb966..580f62524 100644 --- a/tests/test_expr_operand_type_and_size.py +++ b/tests/test_expr_operand_type_and_size.py @@ -368,53 +368,171 @@ def empty_table(dynamodb_client): dynamodb_client.delete_table(TableName=name) -LEGACY_TYPE = ( - "One or more parameter values were invalid: ComparisonOperator {} is not valid for {} AttributeValue type" -) +@pytest.fixture(scope="class") +def empty_range_table(dynamodb_client): + name = f"expr_legacy_rk_{uuid.uuid4().hex[:8]}" + dynamodb_client.create_table( + TableName=name, + KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}, {"AttributeName": "sk", "KeyType": "RANGE"}], + AttributeDefinitions=[ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + from conftest import wait_for_active + + wait_for_active(dynamodb_client, name) + yield name + dynamodb_client.delete_table(TableName=name) + + +INVALID_PARAMS = "One or more parameter values were invalid: " +LEGACY_TYPE = INVALID_PARAMS + "ComparisonOperator {} is not valid for {} AttributeValue type" +LEGACY_COUNT = INVALID_PARAMS + "Invalid number of argument(s) for the {} ComparisonOperator" +LEGACY_SAME_TYPE = INVALID_PARAMS + "AttributeValues inside AttributeValueList must be of same type" +LEGACY_RANGE = INVALID_PARAMS + "The BETWEEN condition was provided a range where the lower bound is greater than the upper bound" +LEGACY_OPERATOR = INVALID_PARAMS + "Invalid comparison operator" +LEGACY_NOT_INDEXABLE = INVALID_PARAMS + "Attempted conditional constraint is not an indexable operation" KC_ABSENT = {"pk": {"ComparisonOperator": "EQ", "AttributeValueList": [{"S": "absent"}]}} +S_A = {"S": "a"} +N_1 = {"N": "1"} +B_1 = {"B": b"\x01"} +BOOL_T = {"BOOL": True} +L_E = {"L": []} + +# One refused type and one accepted type per operator (measured 2026-10-23; +# the full grid is in the engine's unit tests). +LEGACY_TYPE_ROWS = [ + ("LE", [{"SS": ["a"]}], "SS"), + ("LT", [BOOL_T], "BOOL"), + ("GE", [{"NULL": True}], "NULL"), + ("GT", [{"M": {}}], "M"), + ("CONTAINS", [L_E], "L"), + ("NOT_CONTAINS", [{"NS": ["1"]}], "NS"), + ("BEGINS_WITH", [N_1], "N"), + ("BEGINS_WITH", [{"BS": [b"\x01"]}], "BS"), + ("IN", [BOOL_T], "BOOL"), + ("BETWEEN", [L_E, L_E], "L"), + ("BETWEEN", [L_E, N_1], "L"), + ("BETWEEN", [BOOL_T, L_E], "BOOL"), + ("IN", [L_E, S_A], "L"), +] +LEGACY_ACCEPTED_ROWS = [ + ("EQ", [L_E]), + ("NE", [BOOL_T]), + ("LT", [B_1]), + ("GE", [N_1]), + ("CONTAINS", [BOOL_T]), + ("NOT_CONTAINS", [{"NULL": True}]), + ("BEGINS_WITH", [B_1]), + ("IN", [N_1, {"N": "2"}, {"N": "3"}]), + ("BETWEEN", [N_1, {"N": "10"}]), + ("NULL", None), + ("NOT_NULL", None), +] +LEGACY_COUNT_ROWS = [ + ("EQ", [], "EQ"), + ("EQ", [S_A, S_A], "EQ"), + ("NULL", [S_A], "NULL"), + ("NOT_NULL", [S_A], "NOT_NULL"), + ("IN", [], "IN"), + ("BETWEEN", [S_A], "BETWEEN"), + ("BETWEEN", [S_A, S_A, S_A], "BETWEEN"), + ("BEGINS_WITH", [S_A, S_A], "BEGINS_WITH"), +] +LEGACY_OTHER_ROWS = [ + ("BETWEEN", [S_A, N_1], LEGACY_SAME_TYPE), + ("BETWEEN", [N_1, L_E], LEGACY_SAME_TYPE), + ("IN", [S_A, N_1], LEGACY_SAME_TYPE), + ("IN", [S_A, L_E], LEGACY_SAME_TYPE), + ("BETWEEN", [{"N": "2"}, N_1], LEGACY_RANGE), + ("BETWEEN", [{"S": "b"}, S_A], LEGACY_RANGE), + ("FOO", [S_A], LEGACY_OPERATOR), + ("eq", [S_A], LEGACY_OPERATOR), +] + + +def _cond(operator, values): + cond = {"ComparisonOperator": operator} + if values is not None: + cond["AttributeValueList"] = values + return cond + + +def _legacy_error(call): + with pytest.raises(ClientError) as ei: + call() + assert _err(ei)["Code"] == "ValidationException" + return _err(ei)["Message"] class TestLegacyFilterOperandType: - """The legacy QueryFilter and ScanFilter parameters are checked at request - time too: on an empty table, where no row is evaluated, an operator given - a value of a type it does not take is refused with the service's message - (measured against Amazon DynamoDB on 2026-10-23). The value named is the - first one of a type the operator does not take.""" - - @pytest.mark.parametrize( - ("operator", "values", "named_type"), - [ - ("BEGINS_WITH", [{"N": "1"}], "N"), - ("LT", [{"BOOL": True}], "BOOL"), - ("GT", [{"BOOL": True}], "BOOL"), - ("BETWEEN", [{"L": []}, {"N": "1"}], "L"), - ], - ) - def test_scan_filter_refuses_with_the_service_message(self, dynamodb_client, empty_table, operator, values, named_type): - with pytest.raises(ClientError) as ei: - dynamodb_client.scan( - TableName=empty_table, - ScanFilter={"f": {"ComparisonOperator": operator, "AttributeValueList": values}}, - ) - assert _err(ei)["Code"] == "ValidationException" - assert _err(ei)["Message"] == LEGACY_TYPE.format(operator, named_type) + """The legacy QueryFilter, ScanFilter and KeyConditions parameters are + checked at request time, on an empty table where no row is evaluated and + before the table is looked up, with the service's messages (measured + against Amazon DynamoDB on 2026-10-23): the operator name, the value + count, the first value's type, that every value has the first value's + type, and that a BETWEEN range is not inverted. The engine's unit tests + hold the full operator by type grid; these rows cover each message on the + wire.""" + + @pytest.mark.parametrize(("operator", "values", "named_type"), LEGACY_TYPE_ROWS) + def test_scan_filter_refuses_a_type_the_operator_does_not_take(self, dynamodb_client_no_validation, empty_table, operator, values, named_type): + msg = _legacy_error(lambda: dynamodb_client_no_validation.scan(TableName=empty_table, ScanFilter={"f": _cond(operator, values)})) + assert msg == LEGACY_TYPE.format(operator, named_type) + + @pytest.mark.parametrize(("operator", "values", "named_type"), LEGACY_TYPE_ROWS) + def test_query_filter_refuses_a_type_the_operator_does_not_take(self, dynamodb_client_no_validation, empty_table, operator, values, named_type): + msg = _legacy_error(lambda: dynamodb_client_no_validation.query(TableName=empty_table, KeyConditions=KC_ABSENT, QueryFilter={"f": _cond(operator, values)})) + assert msg == LEGACY_TYPE.format(operator, named_type) + + @pytest.mark.parametrize(("operator", "values"), LEGACY_ACCEPTED_ROWS) + def test_scan_filter_accepts_the_types_the_operator_takes(self, dynamodb_client_no_validation, empty_table, operator, values): + resp = dynamodb_client_no_validation.scan(TableName=empty_table, ScanFilter={"f": _cond(operator, values)}) + assert resp["Count"] == 0 - @pytest.mark.parametrize( - ("operator", "values", "named_type"), - [ - ("BEGINS_WITH", [{"N": "1"}], "N"), - ("GT", [{"BOOL": True}], "BOOL"), - ], - ) - def test_query_filter_refuses_with_the_service_message(self, dynamodb_client, empty_table, operator, values, named_type): - with pytest.raises(ClientError) as ei: - dynamodb_client.query( - TableName=empty_table, - KeyConditions=KC_ABSENT, - QueryFilter={"f": {"ComparisonOperator": operator, "AttributeValueList": values}}, - ) - assert _err(ei)["Code"] == "ValidationException" - assert _err(ei)["Message"] == LEGACY_TYPE.format(operator, named_type) + @pytest.mark.parametrize(("operator", "values", "named"), LEGACY_COUNT_ROWS) + def test_scan_filter_refuses_a_wrong_value_count(self, dynamodb_client_no_validation, empty_table, operator, values, named): + msg = _legacy_error(lambda: dynamodb_client_no_validation.scan(TableName=empty_table, ScanFilter={"f": _cond(operator, values)})) + assert msg == LEGACY_COUNT.format(named) + + @pytest.mark.parametrize(("operator", "values", "message"), LEGACY_OTHER_ROWS) + def test_scan_filter_mixed_types_ranges_and_operator_names(self, dynamodb_client_no_validation, empty_table, operator, values, message): + msg = _legacy_error(lambda: dynamodb_client_no_validation.scan(TableName=empty_table, ScanFilter={"f": _cond(operator, values)})) + assert msg == message + + @pytest.mark.parametrize(("operator", "values", "message"), LEGACY_OTHER_ROWS) + def test_query_filter_mixed_types_ranges_and_operator_names(self, dynamodb_client_no_validation, empty_table, operator, values, message): + msg = _legacy_error(lambda: dynamodb_client_no_validation.query(TableName=empty_table, KeyConditions=KC_ABSENT, QueryFilter={"f": _cond(operator, values)})) + assert msg == message + + def test_scan_filter_is_checked_before_the_table_lookup(self, dynamodb_client): + msg = _legacy_error(lambda: dynamodb_client.scan(TableName=f"absent_{uuid.uuid4().hex[:8]}", ScanFilter={"f": _cond("BEGINS_WITH", [N_1])})) + assert msg == LEGACY_TYPE.format("BEGINS_WITH", "N") + + def test_query_filter_is_checked_before_the_table_lookup_and_the_key_conditions(self, dynamodb_client): + msg = _legacy_error(lambda: dynamodb_client.query(TableName=f"absent_{uuid.uuid4().hex[:8]}", KeyConditions={"pk": _cond("LT", [S_A])}, QueryFilter={"f": _cond("BEGINS_WITH", [N_1])})) + assert msg == LEGACY_TYPE.format("BEGINS_WITH", "N") + + @pytest.mark.parametrize(("operator", "values", "named_type"), [("BEGINS_WITH", [N_1], "N"), ("LT", [BOOL_T], "BOOL"), ("CONTAINS", [{"SS": ["a"]}], "SS"), ("BETWEEN", [L_E, N_1], "L")]) + def test_key_conditions_get_the_same_value_checks(self, dynamodb_client_no_validation, empty_range_table, operator, values, named_type): + msg = _legacy_error(lambda: dynamodb_client_no_validation.query(TableName=empty_range_table, KeyConditions={"pk": KC_ABSENT["pk"], "sk": _cond(operator, values)})) + assert msg == LEGACY_TYPE.format(operator, named_type) + + @pytest.mark.parametrize(("operator", "values", "named"), [("EQ", [S_A, S_A], "EQ"), ("NULL", [S_A], "NULL"), ("BETWEEN", [S_A], "BETWEEN"), ("IN", [], "IN")]) + def test_key_conditions_refuse_a_wrong_value_count(self, dynamodb_client_no_validation, empty_range_table, operator, values, named): + msg = _legacy_error(lambda: dynamodb_client_no_validation.query(TableName=empty_range_table, KeyConditions={"pk": KC_ABSENT["pk"], "sk": _cond(operator, values)})) + assert msg == LEGACY_COUNT.format(named) + + @pytest.mark.parametrize(("operator", "values"), [("NE", [S_A]), ("NULL", None), ("NOT_NULL", None), ("CONTAINS", [S_A]), ("NOT_CONTAINS", [N_1]), ("IN", [S_A, {"S": "b"}])]) + def test_key_conditions_refuse_non_indexable_operators_after_the_value_checks(self, dynamodb_client_no_validation, empty_range_table, operator, values): + msg = _legacy_error(lambda: dynamodb_client_no_validation.query(TableName=empty_range_table, KeyConditions={"pk": KC_ABSENT["pk"], "sk": _cond(operator, values)})) + assert msg == LEGACY_NOT_INDEXABLE + + def test_key_conditions_inverted_between_range(self, dynamodb_client, empty_range_table): + msg = _legacy_error(lambda: dynamodb_client.query(TableName=empty_range_table, KeyConditions={"pk": KC_ABSENT["pk"], "sk": _cond("BETWEEN", [{"S": "b"}, S_A])})) + assert msg == LEGACY_RANGE def test_scan_filter_begins_with_string_accepted(self, dynamodb_client, empty_table): resp = dynamodb_client.scan( From a437e80f4c0bf56055710c0f2d02dfd77745c1f4 Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 16:17:53 +0000 Subject: [PATCH 05/10] fix(engine): order transaction ConsumedCapacity by table creation time Amazon DynamoDB documents no order for the ConsumedCapacity entries of TransactGetItems and TransactWriteItems. Measured on 2026-10-23 with three single-hash-key tables whose names sort opposite to their creation order, in three request orders: both operations listed the tables by creation time, oldest first, whatever order the request named them in. TableKeyInfo now carries the table's creation time. Every backend reads it from the catalog row or document it already fetches for the key schema (SQLite and PostgreSQL creation_date_time, MongoDB creation_date_time, Cassandra created_at), so no request makes another catalog read. PerTableCapacity moves from capacity_helpers.rs to per_table_capacity.rs, takes the TableKeyInfo of each operation, and lists tables oldest first, with name order breaking a tie on the same creation time. Both transaction handlers use it, including the idempotent replay path of TransactWriteItems, which now has a key_info accessor on PreparedOp. tests/test_transact_capacity_order.py creates three tables whose names sort opposite to their creation order, waiting for ACTIVE between them, and asserts the order for both operations at TOTAL and INDEXES over the three measured request orders; it runs against the service and against ExtendDB. The row in docs/differences-from-dynamodb.md states the measured order and ExtendDB's tie rule. --- crates/core/src/types/key_schema.rs | 10 ++ crates/engine/src/capacity_helpers.rs | 31 ----- crates/engine/src/lib.rs | 1 + crates/engine/src/per_table_capacity.rs | 115 ++++++++++++++++++ crates/engine/src/query.rs | 1 + crates/engine/src/scan.rs | 1 + crates/engine/src/transact_get_items.rs | 11 +- crates/engine/src/transact_write_helpers.rs | 7 +- crates/engine/src/transact_write_items.rs | 14 ++- crates/storage-cassandra/src/backup_engine.rs | 1 + crates/storage-cassandra/src/data/ddl.rs | 7 +- crates/storage-cassandra/src/data/ttl.rs | 1 + crates/storage-cassandra/tests/common/mod.rs | 4 + .../storage-cassandra/tests/direct/streams.rs | 1 + crates/storage-mongodb/src/table_engine.rs | 5 + crates/storage-postgres/src/data/ddl.rs | 9 +- crates/storage-sqlite/src/data/ddl.rs | 14 ++- crates/storage-sqlite/src/data/query_scan.rs | 1 + docs/differences-from-dynamodb.md | 2 +- tests/test_transact_capacity_order.py | 113 +++++++++++++++++ 20 files changed, 297 insertions(+), 52 deletions(-) create mode 100644 crates/engine/src/per_table_capacity.rs create mode 100644 tests/test_transact_capacity_order.py diff --git a/crates/core/src/types/key_schema.rs b/crates/core/src/types/key_schema.rs index 7aa89ef3e..a80c80524 100755 --- a/crates/core/src/types/key_schema.rs +++ b/crates/core/src/types/key_schema.rs @@ -102,6 +102,13 @@ pub struct TableKeyInfo { /// validate vector-valued and search-schema attributes without an extra /// catalog round-trip. pub vector_indexes: Vec, + /// When the table was created, as seconds since the Unix epoch with the + /// catalog's sub-second precision; 0.0 when the source carries none. Read + /// from the same catalog row as the key schema, so it costs no extra + /// round-trip. The transaction handlers order their `ConsumedCapacity` + /// entries by it, oldest table first, as DynamoDB does (measured + /// 2026-10-23). + pub creation_date_time: f64, } /// Vector-index metadata needed by the write path to validate an item's @@ -294,6 +301,7 @@ mod tests { local_secondary_indexes: vec![], stream_specification: None, vector_indexes: vec![], + creation_date_time: 0.0, } } @@ -474,6 +482,7 @@ mod tests { local_secondary_indexes: vec![], stream_specification: None, vector_indexes: vec![], + creation_date_time: 0.0, }; assert_eq!(info.key_schema, info.base_key_schema); } @@ -494,6 +503,7 @@ mod tests { local_secondary_indexes: vec![], stream_specification: None, vector_indexes: vec![], + creation_date_time: 0.0, }; assert_eq!(info.key_schema, index_schema); assert_eq!(info.base_key_schema, base_schema); diff --git a/crates/engine/src/capacity_helpers.rs b/crates/engine/src/capacity_helpers.rs index a0c113b96..73b8c04fb 100755 --- a/crates/engine/src/capacity_helpers.rs +++ b/crates/engine/src/capacity_helpers.rs @@ -380,37 +380,6 @@ pub fn batch_write_capacity<'a>( } } -/// Per-table capacity totals in the order the tables first appear in the -/// request. -/// -/// This is a firmer guarantee than Amazon DynamoDB gives, not a copy of its -/// order. The service documents no order for the `ConsumedCapacity` entries -/// of a transaction, and the measured order (2026-10-23, three tables, eight -/// request orders, TransactGetItems and TransactWriteItems) did not follow -/// the request: every response listed the tables in their name order, which -/// was also their creation order. A client cannot depend on the position of -/// an entry against the service, so any deterministic order is compatible -/// with it (docs/differences-from-dynamodb.md, Capacity and Throttling). -/// Before this change ExtendDB aggregated into a `HashMap`, so the order -/// changed from one process to the next and differed between backends; first -/// appearance is stable, matches the request as the caller wrote it, and lets -/// the tests assert the array. -#[derive(Default)] -pub struct PerTableCapacity(Vec<(String, f64)>); - -impl PerTableCapacity { - pub fn add(&mut self, table: &str, units: f64) { - match self.0.iter_mut().find(|(t, _)| t == table) { - Some((_, total)) => *total += units, - None => self.0.push((table.to_owned(), units)), - } - } - - pub fn iter(&self) -> impl Iterator { - self.0.iter().map(|(t, cu)| (t.as_str(), *cu)) - } -} - /// Build a `Vec` for `TransactGetItems`, or `None` if not requested. /// /// Transactions differ from single-item/batch reads: real DynamoDB emits the diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index 74083fbfa..426bb2993 100755 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -25,6 +25,7 @@ mod import_export_io; mod index_helpers; mod legacy_filter; mod list_tables; +mod per_table_capacity; mod put_item; mod query; mod read_helpers; diff --git a/crates/engine/src/per_table_capacity.rs b/crates/engine/src/per_table_capacity.rs new file mode 100644 index 000000000..a7e9307f8 --- /dev/null +++ b/crates/engine/src/per_table_capacity.rs @@ -0,0 +1,115 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Per-table capacity totals for `TransactGetItems` and `TransactWriteItems`, +//! in the order Amazon DynamoDB lists them. + +use extenddb_core::types::TableKeyInfo; + +/// Per-table capacity totals, listed oldest table first. +/// +/// Amazon DynamoDB documents no order for the `ConsumedCapacity` entries of a +/// transaction. Measured on 2026-10-23 with three single-hash-key tables +/// created on 2026-10-22, 2026-10-23 11:01 and 2026-10-23 14:45 UTC, where the +/// newest table's name sorts first and the oldest table's name sorts last: +/// `TransactGetItems` and `TransactWriteItems` listed the tables oldest first +/// for the request orders [newest, middle, oldest], [middle, newest, oldest] +/// and [oldest, newest, middle]. The order follows neither the request nor +/// the names; it follows the tables' creation time. An earlier measurement +/// the same day (16 transactions over three tables created in name order) is +/// consistent with it. +/// +/// ExtendDB orders the entries the same way, by the `creation_date_time` the +/// catalog carries on [`TableKeyInfo`], oldest first; two tables with the same +/// creation time are listed in name order so the array is deterministic on +/// every backend. Before this change the handlers aggregated into a `HashMap`, +/// so the order changed from one process to the next and differed between +/// backends (docs/differences-from-dynamodb.md, Capacity and Throttling). +#[derive(Default)] +pub struct PerTableCapacity(Vec); + +struct Entry { + table: String, + created: f64, + units: f64, +} + +impl PerTableCapacity { + /// Add `units` to the table `key_info` describes. + pub fn add(&mut self, key_info: &TableKeyInfo, units: f64) { + match self.0.iter_mut().find(|e| e.table == key_info.table_name) { + Some(e) => e.units += units, + None => self.0.push(Entry { + table: key_info.table_name.clone(), + created: key_info.creation_date_time, + units, + }), + } + } + + /// The totals, oldest table first; ties on creation time in name order. + pub fn ordered(&self) -> Vec<(&str, f64)> { + let mut entries: Vec<&Entry> = self.0.iter().collect(); + entries.sort_by(|a, b| { + a.created + .total_cmp(&b.created) + .then_with(|| a.table.cmp(&b.table)) + }); + entries + .into_iter() + .map(|e| (e.table.as_str(), e.units)) + .collect() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn table(name: &str, created: f64) -> TableKeyInfo { + TableKeyInfo { + table_name: name.to_owned(), + creation_date_time: created, + ..Default::default() + } + } + + #[test] + fn orders_by_creation_time_not_by_request_or_name() { + // Names sort opposite to creation order, and the request names the + // newest table first: the service's order is oldest first. + let (c, b, a) = (table("c", 1.0), table("b", 2.0), table("a", 3.0)); + let mut totals = PerTableCapacity::default(); + totals.add(&a, 2.0); + totals.add(&c, 2.0); + totals.add(&b, 2.0); + totals.add(&a, 4.0); + assert_eq!( + totals.ordered(), + vec![("c", 2.0), ("b", 2.0), ("a", 6.0)], + "oldest table first, with the repeated table summed once" + ); + } + + #[test] + fn ties_on_creation_time_fall_back_to_name_order() { + let mut totals = PerTableCapacity::default(); + totals.add(&table("zeta", 5.0), 1.0); + totals.add(&table("alpha", 5.0), 1.0); + totals.add(&table("mid", 5.0), 1.0); + assert_eq!( + totals.ordered(), + vec![("alpha", 1.0), ("mid", 1.0), ("zeta", 1.0)] + ); + } + + #[test] + fn unknown_creation_time_sorts_before_known() { + // A source that carries no creation time reports 0.0 (TableKeyInfo + // doc); it lists first rather than failing the sort. + let mut totals = PerTableCapacity::default(); + totals.add(&table("known", 10.0), 1.0); + totals.add(&table("unknown", 0.0), 1.0); + assert_eq!(totals.ordered(), vec![("unknown", 1.0), ("known", 1.0)]); + } +} diff --git a/crates/engine/src/query.rs b/crates/engine/src/query.rs index a611f2d49..a0c83d539 100755 --- a/crates/engine/src/query.rs +++ b/crates/engine/src/query.rs @@ -349,6 +349,7 @@ pub async fn handle_query( local_secondary_indexes: key_info.local_secondary_indexes.clone(), stream_specification: None, // Queries don't capture stream records vector_indexes: key_info.vector_indexes.clone(), + creation_date_time: key_info.creation_date_time, } } else { key_info.clone() diff --git a/crates/engine/src/scan.rs b/crates/engine/src/scan.rs index a71565a48..7fdd19799 100755 --- a/crates/engine/src/scan.rs +++ b/crates/engine/src/scan.rs @@ -381,6 +381,7 @@ pub async fn handle_scan( local_secondary_indexes: key_info.local_secondary_indexes.clone(), stream_specification: None, // Scans don't capture stream records vector_indexes: key_info.vector_indexes.clone(), + creation_date_time: key_info.creation_date_time, } } else { key_info.clone() diff --git a/crates/engine/src/transact_get_items.rs b/crates/engine/src/transact_get_items.rs index 3c099d68c..2339ea703 100755 --- a/crates/engine/src/transact_get_items.rs +++ b/crates/engine/src/transact_get_items.rs @@ -21,6 +21,7 @@ use crate::OperationContext; use crate::capacity_helpers; use crate::create_table::storage_err_to_dynamo; use crate::expression_helpers::build_expression_maps; +use crate::per_table_capacity::PerTableCapacity; use crate::request_limits; use crate::serialize_output; use crate::{DispatchMetrics, DispatchResult}; @@ -160,17 +161,17 @@ pub async fn handle_transact_get_items( // Capacity metering: RCU rounded per item, then summed (M-1). // Capacity metering: TransactGetItems costs 2 RCU per item (transactions // double the read cost). Missing items still cost 2 RCU. - let mut per_table_rcu = capacity_helpers::PerTableCapacity::default(); + let mut per_table_rcu = PerTableCapacity::default(); let rcu: f64 = items .iter() - .zip(input.transact_items.iter()) - .map(|(opt, tgi)| { + .zip(key_infos.iter()) + .map(|(opt, ki)| { let base_rcu = match opt { Some(item) => capacity_helpers::read_capacity_units(item_size_bytes(item), true), None => 1.0, // minimum 1 RCU for missing item }; let txn_rcu = base_rcu * 2.0; // transactions cost 2x - per_table_rcu.add(&tgi.get.table_name, txn_rcu); + per_table_rcu.add(ki, txn_rcu); txn_rcu }) .sum(); @@ -196,7 +197,7 @@ pub async fn handle_transact_get_items( let consumed_capacity = capacity_helpers::transact_read_capacity( input.return_consumed_capacity, - per_table_rcu.iter(), + per_table_rcu.ordered().into_iter(), ); let output = TransactGetItemsOutput { diff --git a/crates/engine/src/transact_write_helpers.rs b/crates/engine/src/transact_write_helpers.rs index 3a2bcfda6..a68d62298 100755 --- a/crates/engine/src/transact_write_helpers.rs +++ b/crates/engine/src/transact_write_helpers.rs @@ -58,11 +58,16 @@ pub(crate) enum PreparedOp { impl PreparedOp { /// The table name for this operation. pub(crate) fn table_name(&self) -> &str { + &self.key_info().table_name + } + + /// The table this operation targets. + pub(crate) fn key_info(&self) -> &TableKeyInfo { match self { Self::Put { key_info, .. } | Self::Delete { key_info, .. } | Self::Update { key_info, .. } - | Self::ConditionCheck { key_info, .. } => &key_info.table_name, + | Self::ConditionCheck { key_info, .. } => key_info, } } diff --git a/crates/engine/src/transact_write_items.rs b/crates/engine/src/transact_write_items.rs index 8002d6597..d35f80d4b 100755 --- a/crates/engine/src/transact_write_items.rs +++ b/crates/engine/src/transact_write_items.rs @@ -11,6 +11,7 @@ use crate::OperationContext; use crate::capacity_helpers; use crate::create_table::storage_err_to_dynamo; use crate::expression_helpers::build_expression_maps; +use crate::per_table_capacity::PerTableCapacity; use crate::request_limits; use crate::serialize_output; use crate::stream_capture; @@ -196,15 +197,16 @@ pub async fn handle_transact_write_items( extenddb_core::types::ReturnConsumedCapacity::None => None, rcc => { let indexes = rcc == extenddb_core::types::ReturnConsumedCapacity::Indexes; - let mut per_table_rcu = capacity_helpers::PerTableCapacity::default(); + let mut per_table_rcu = PerTableCapacity::default(); for op in &prepared { let rcu = capacity_helpers::read_capacity_units(op.write_bytes(), true) * 2.0; - per_table_rcu.add(op.table_name(), rcu); + per_table_rcu.add(op.key_info(), rcu); } Some( per_table_rcu - .iter() + .ordered() + .into_iter() .map(|(t, cu)| { extenddb_core::types::ConsumedCapacity::transact_replay_read( t, cu, indexes, @@ -233,12 +235,12 @@ pub async fn handle_transact_write_items( // Stream records are now captured atomically within the storage transaction. // Per-item WCU: round each item individually, then sum (M-1). - let mut per_table_wcu = capacity_helpers::PerTableCapacity::default(); + let mut per_table_wcu = PerTableCapacity::default(); let wcu: f64 = prepared .iter() .map(|op| { let item_wcu = capacity_helpers::write_capacity_units(op.write_bytes()) * 2.0; // transactions cost 2x - per_table_wcu.add(op.table_name(), item_wcu); + per_table_wcu.add(op.key_info(), item_wcu); item_wcu }) .sum(); @@ -250,7 +252,7 @@ pub async fn handle_transact_write_items( // only for now. let consumed_capacity = capacity_helpers::transact_write_capacity( input.return_consumed_capacity, - per_table_wcu.iter(), + per_table_wcu.ordered().into_iter(), ); // Collect ItemCollectionMetrics per table for write operations. When SIZE diff --git a/crates/storage-cassandra/src/backup_engine.rs b/crates/storage-cassandra/src/backup_engine.rs index 7c070934e..41b819166 100755 --- a/crates/storage-cassandra/src/backup_engine.rs +++ b/crates/storage-cassandra/src/backup_engine.rs @@ -979,6 +979,7 @@ impl CassandraEngine { local_secondary_indexes: Vec::new(), stream_specification: None, vector_indexes: Vec::new(), + creation_date_time: 0.0, }; let account_keyspace = self.account_keyspace(&job.account_id); let query = format!( diff --git a/crates/storage-cassandra/src/data/ddl.rs b/crates/storage-cassandra/src/data/ddl.rs index 9016422dd..16ab50742 100644 --- a/crates/storage-cassandra/src/data/ddl.rs +++ b/crates/storage-cassandra/src/data/ddl.rs @@ -80,7 +80,8 @@ impl CassandraEngine { let catalog_keyspace = format!("{}_catalog", self.keyspace_prefix); let query = format!( - "SELECT key_schema, attribute_definitions, table_status, table_id, stream_specification \ + "SELECT key_schema, attribute_definitions, table_status, table_id, stream_specification, \ + created_at \ FROM {catalog_keyspace}.tables WHERE account_id = ? AND table_name = ?" ); let values = cdrs_tokio::query_values!(account_id, table_name); @@ -116,6 +117,9 @@ impl CassandraEngine { .map_err(|e| StorageError::Internal(format!("Parse table_id: {e}")))?; let stream_spec_text: Option = row.get_by_name("stream_specification").ok().flatten(); + let created_at: i64 = row + .get_r_by_name("created_at") + .map_err(|e| StorageError::Internal(format!("Parse created_at: {e}")))?; if status == "CREATING" || status == "DELETING" { return Err(StorageError::TableNotFound(table_name.to_owned())); @@ -182,6 +186,7 @@ impl CassandraEngine { local_secondary_indexes, stream_specification, vector_indexes: Vec::new(), + creation_date_time: crate::cassandra_util::millis_to_seconds_f64(created_at), }) } diff --git a/crates/storage-cassandra/src/data/ttl.rs b/crates/storage-cassandra/src/data/ttl.rs index e2b2b0fea..728f8d700 100644 --- a/crates/storage-cassandra/src/data/ttl.rs +++ b/crates/storage-cassandra/src/data/ttl.rs @@ -1345,6 +1345,7 @@ mod tests { local_secondary_indexes: Vec::new(), vector_indexes: Vec::new(), stream_specification: None, + creation_date_time: 0.0, } } diff --git a/crates/storage-cassandra/tests/common/mod.rs b/crates/storage-cassandra/tests/common/mod.rs index c1fd1e971..e88ff74db 100644 --- a/crates/storage-cassandra/tests/common/mod.rs +++ b/crates/storage-cassandra/tests/common/mod.rs @@ -278,6 +278,7 @@ impl TestTable { local_secondary_indexes: Vec::new(), vector_indexes: Vec::new(), stream_specification: None, + creation_date_time: 0.0, }; Self { @@ -360,6 +361,7 @@ impl TestTable { local_secondary_indexes: Vec::new(), vector_indexes: Vec::new(), stream_specification: None, + creation_date_time: 0.0, }; Self { @@ -453,6 +455,7 @@ impl TestTable { local_secondary_indexes: Vec::new(), vector_indexes: Vec::new(), stream_specification: None, + creation_date_time: 0.0, }; Self { @@ -550,6 +553,7 @@ impl TestTable { local_secondary_indexes: Vec::new(), vector_indexes: Vec::new(), stream_specification: None, + creation_date_time: 0.0, }; Self { diff --git a/crates/storage-cassandra/tests/direct/streams.rs b/crates/storage-cassandra/tests/direct/streams.rs index 865774d3a..a896117cd 100644 --- a/crates/storage-cassandra/tests/direct/streams.rs +++ b/crates/storage-cassandra/tests/direct/streams.rs @@ -106,6 +106,7 @@ async fn setup_stream_table( global_secondary_indexes: Vec::new(), local_secondary_indexes: Vec::new(), stream_specification: desc.stream_specification, + creation_date_time: 0.0, }; (key_info, stream_label) diff --git a/crates/storage-mongodb/src/table_engine.rs b/crates/storage-mongodb/src/table_engine.rs index aa8c5272c..44bf0daf4 100644 --- a/crates/storage-mongodb/src/table_engine.rs +++ b/crates/storage-mongodb/src/table_engine.rs @@ -1268,6 +1268,10 @@ impl MongoEngine { let global_secondary_indexes = grouped.gsis; let local_secondary_indexes = grouped.lsis; let has_lsi = !local_secondary_indexes.is_empty(); + let creation_date_time = table_doc + .get_datetime("creation_date_time") + .map(|dt| dt.timestamp_millis() as f64 / 1000.0) + .unwrap_or(0.0); let key_info = TableKeyInfo { table_name, @@ -1280,6 +1284,7 @@ impl MongoEngine { global_secondary_indexes, local_secondary_indexes, stream_specification, + creation_date_time, // Fields for features this backend does not implement, vector // indexes today, take their defaults. Adding one to TableKeyInfo // then does not break this build. diff --git a/crates/storage-postgres/src/data/ddl.rs b/crates/storage-postgres/src/data/ddl.rs index 6edb222f3..bac1d9356 100755 --- a/crates/storage-postgres/src/data/ddl.rs +++ b/crates/storage-postgres/src/data/ddl.rs @@ -31,13 +31,15 @@ use super::{all_sort_key_info, data_table_name, index_table_name, vector_table_n const TEXT_KEY: &str = "TEXT COLLATE \"C\""; use crate::PostgresEngine; -/// Row shape returned by the table-info query: (`key_schema`, `attr_defs`, status, `table_id`, `stream_spec`). +/// Row shape returned by the table-info query: (`key_schema`, `attr_defs`, +/// status, `table_id`, `stream_spec`, creation epoch seconds). type TableInfoRow = ( serde_json::Value, serde_json::Value, String, String, Option, + f64, ); impl PostgresEngine { @@ -407,7 +409,7 @@ impl PostgresEngine { ) -> Result { let row: Option = sqlx::query_as( "SELECT key_schema, attribute_definitions, table_status, table_id, \ - stream_specification \ + stream_specification, EXTRACT(EPOCH FROM creation_date_time)::FLOAT8 \ FROM tables \ WHERE account_id = $1 AND table_name = $2", ) @@ -417,7 +419,7 @@ impl PostgresEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))?; - let (ks_json, ad_json, status, table_id, stream_spec_json) = + let (ks_json, ad_json, status, table_id, stream_spec_json, creation_date_time) = row.ok_or_else(|| StorageError::TableNotFound(table_name.to_owned()))?; match status.as_str() { @@ -470,6 +472,7 @@ impl PostgresEngine { // new core field should break this site and force a decision about // whether the write path needs it, rather than silently defaulting. vector_indexes, + creation_date_time, }; // Catalog metadata that cannot describe its own sort key would make the // keyed read paths fall back to a partition-only lookup and return the diff --git a/crates/storage-sqlite/src/data/ddl.rs b/crates/storage-sqlite/src/data/ddl.rs index 4052c973e..d7a0196bd 100644 --- a/crates/storage-sqlite/src/data/ddl.rs +++ b/crates/storage-sqlite/src/data/ddl.rs @@ -343,10 +343,10 @@ impl SqliteEngine { account_id: &str, table_name: &str, ) -> Result { - let row: Option<(String, String, String, String, Option)> = sqlx::query_as( + let row: Option<(String, String, String, String, Option, String)> = sqlx::query_as( "SELECT key_schema, attribute_definitions, table_status, table_id, \ - stream_specification \ - FROM tables WHERE account_id = ? AND table_name = ?", + stream_specification, creation_date_time \ + FROM tables WHERE account_id = ? AND table_name = ?", ) .bind(account_id) .bind(table_name) @@ -354,7 +354,7 @@ impl SqliteEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))?; - let (ks_str, ad_str, status, table_id, stream_spec_str) = + let (ks_str, ad_str, status, table_id, stream_spec_str, created_str) = row.ok_or_else(|| StorageError::TableNotFound(table_name.to_owned()))?; match status.as_str() { @@ -388,6 +388,11 @@ impl SqliteEngine { // decide whether any maintenance is needed without a query, and so the // engine can validate vector attributes on writes. let vector_indexes = self.fetch_vector_index_key_info(&table_id).await?; + // The column is RFC 3339 with milliseconds (schema.rs); keep them, the + // transaction handlers order tables by this value. + let creation_date_time = crate::sqlite_util::parse_timestamp(&created_str) + .map(|dt| dt.unix_timestamp_nanos() as f64 / 1e9) + .unwrap_or(0.0); let key_info = TableKeyInfo { table_name: table_name.to_owned(), @@ -404,6 +409,7 @@ impl SqliteEngine { // new core field should break this site and force a decision about // whether the write path needs it, rather than silently defaulting. vector_indexes, + creation_date_time, }; // Catalog metadata that cannot describe its own sort key would make the // keyed read paths fall back to a partition-only lookup and return the diff --git a/crates/storage-sqlite/src/data/query_scan.rs b/crates/storage-sqlite/src/data/query_scan.rs index 585debd28..3686b25ba 100644 --- a/crates/storage-sqlite/src/data/query_scan.rs +++ b/crates/storage-sqlite/src/data/query_scan.rs @@ -586,6 +586,7 @@ mod tests { local_secondary_indexes: Vec::new(), vector_indexes: Vec::new(), stream_specification: None, + creation_date_time: 0.0, } } diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 38c14c036..185743da9 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -147,7 +147,7 @@ The same on all four backends. | Provisioned throughput | Token bucket per table/partition | Token bucket per table/partition, matching DynamoDB's burst and refill behavior | | On-demand capacity | Automatic scaling | Fixed initial burst capacity (4000 WCU / 12000 RCU), no auto-scaling | | Throttling | Always on; throttles requests that exceed provisioned/burst capacity. No setting to disable | Configurable via `throttling_enabled` runtime setting (default: `true`) | -| Order of `ConsumedCapacity` entries in `TransactGetItems` and `TransactWriteItems` responses | Undocumented. Measured 2026-10-23 with three tables in eight request orders (16 transactions): the entries came back in the same order every time, the tables' name order, whatever order the request named them in (a request naming `bb-w-W2-1`, `bb-w-M3-2`, `bb-w-M3-1` returned `bb-w-M3-1`, `bb-w-M3-2`, `bb-w-W2-1`). The three tables were also created in name order, so creation order fits the same data | One entry per table in the order the tables first appear in the request, on every backend. A client that reads the array by position may see a different order against DynamoDB; look entries up by `TableName` | +| Order of `ConsumedCapacity` entries in `TransactGetItems` and `TransactWriteItems` responses | Undocumented. Measured 2026-10-23 with three single-hash-key tables whose names sort opposite to their creation order, in three request orders: both operations listed the tables by creation time, oldest first, whatever order the request named them in (a request naming the newest table first still returned the oldest first). The order follows neither the request nor the table names | The same order on every backend: one entry per table, by the table's creation time, oldest first; two tables created in the same millisecond are listed in name order. The creation time comes from the catalog row the handlers already read for the key schema, so it costs no extra round-trip. Earlier builds ordered the entries by hash-map iteration, which varied between processes and backends | ## Operations Not Implemented diff --git a/tests/test_transact_capacity_order.py b/tests/test_transact_capacity_order.py new file mode 100644 index 000000000..f82c69b4d --- /dev/null +++ b/tests/test_transact_capacity_order.py @@ -0,0 +1,113 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""Order of the ConsumedCapacity entries of a multi-table transaction. + +Amazon DynamoDB documents no order for the ``ConsumedCapacity`` array of +``TransactGetItems`` and ``TransactWriteItems``. Measured on 2026-10-23 with +three single-hash-key tables whose names sort opposite to their creation order: +both operations listed the tables oldest first, for every request order tried +([newest, middle, oldest], [middle, newest, oldest], [oldest, newest, middle]). +The order follows neither the request nor the names. + +The fixture below creates its three tables so that the name order is the +reverse of the creation order, waiting for ACTIVE between them, so a server +that ordered by name, or by first appearance in the request, fails these +tests. Dual-target: runs against the service and against ExtendDB on every +backend. +""" + +from __future__ import annotations + +import uuid + +import pytest + +from conftest import wait_for_active, wait_for_deleted + + +@pytest.fixture(scope="module") +def tables_oldest_first(dynamodb_client) -> list[str]: + """Three tables, returned oldest first; their names sort newest first.""" + suffix = uuid.uuid4().hex[:12] + names = [f"extenddb-test-c-{suffix}", f"extenddb-test-b-{suffix}", f"extenddb-test-a-{suffix}"] + created: list[str] = [] + try: + for name in names: + dynamodb_client.create_table( + TableName=name, + AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}], + KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}], + BillingMode="PAY_PER_REQUEST", + ) + created.append(name) + # Each table is ACTIVE before the next is created, so the creation + # times are distinct and in list order. + wait_for_active(dynamodb_client, name) + yield created + finally: + for name in created: + try: + dynamodb_client.delete_table(TableName=name) + except dynamodb_client.exceptions.ResourceNotFoundException: + continue + wait_for_deleted(dynamodb_client, name) + + +def _request_orders(oldest_first: list[str]) -> list[list[str]]: + oldest, middle, newest = oldest_first + return [ + [newest, middle, oldest], + [middle, newest, oldest], + [oldest, newest, middle], + ] + + +def _table_names(resp: dict) -> list[str]: + return [c["TableName"] for c in resp["ConsumedCapacity"]] + + +class TestTransactionConsumedCapacityOrder: + """Measured against Amazon DynamoDB on 2026-10-23.""" + + def test_fixture_names_sort_opposite_to_creation(self, tables_oldest_first): + assert sorted(tables_oldest_first) == list(reversed(tables_oldest_first)) + + @pytest.mark.parametrize("granularity", ["TOTAL", "INDEXES"]) + def test_transact_get_items_lists_tables_oldest_first( + self, dynamodb_client, tables_oldest_first, granularity + ): + for order in _request_orders(tables_oldest_first): + resp = dynamodb_client.transact_get_items( + ReturnConsumedCapacity=granularity, + TransactItems=[{"Get": {"TableName": t, "Key": {"pk": {"S": "k"}}}} for t in order], + ) + assert _table_names(resp) == tables_oldest_first, order + + @pytest.mark.parametrize("granularity", ["TOTAL", "INDEXES"]) + def test_transact_write_items_lists_tables_oldest_first( + self, dynamodb_client, tables_oldest_first, granularity + ): + for n, order in enumerate(_request_orders(tables_oldest_first)): + resp = dynamodb_client.transact_write_items( + ReturnConsumedCapacity=granularity, + TransactItems=[ + {"Put": {"TableName": t, "Item": {"pk": {"S": f"{granularity}-{n}"}}}} for t in order + ], + ) + assert _table_names(resp) == tables_oldest_first, order + + def test_a_table_named_twice_has_one_entry_in_the_same_order( + self, dynamodb_client, tables_oldest_first + ): + oldest, middle, newest = tables_oldest_first + resp = dynamodb_client.transact_get_items( + ReturnConsumedCapacity="TOTAL", + TransactItems=[ + {"Get": {"TableName": newest, "Key": {"pk": {"S": "k1"}}}}, + {"Get": {"TableName": oldest, "Key": {"pk": {"S": "k1"}}}}, + {"Get": {"TableName": newest, "Key": {"pk": {"S": "k2"}}}}, + {"Get": {"TableName": middle, "Key": {"pk": {"S": "k1"}}}}, + ], + ) + assert _table_names(resp) == tables_oldest_first From ad63ef65f7904c016769f9db72c3ac6618d3f8ae Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 18:35:19 +0000 Subject: [PATCH 06/10] fix(storage-mongodb): keep milliseconds on the catalog creation time create_table wrote the catalog creation time as whole seconds, so two tables created inside the same second carried the same value. The transaction ConsumedCapacity order sorts tables by that value and falls back to name order on a tie, which the service does not do, and tests/test_transact_capacity_order.py failed 5 of 6 on this backend. The write now keeps the millisecond part, which is the precision bson::DateTime stores. Every reader already converts through timestamp_millis, so DescribeTable, ListTables, the backup and restore paths and the transaction key-info fetcher return the sub-second value, the same as the PostgreSQL backend. Catalog rows written by earlier builds keep whole seconds and can still tie. --- crates/storage-mongodb/src/table_engine.rs | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/crates/storage-mongodb/src/table_engine.rs b/crates/storage-mongodb/src/table_engine.rs index 44bf0daf4..cc4a9e08a 100644 --- a/crates/storage-mongodb/src/table_engine.rs +++ b/crates/storage-mongodb/src/table_engine.rs @@ -156,7 +156,12 @@ impl MongoEngine { let deletion_protection = input.deletion_protection_enabled.unwrap_or(false); let now = time::OffsetDateTime::now_utc(); - let creation_epoch = now.unix_timestamp() as f64; + // Keep the sub-second part. bson::DateTime stores milliseconds, and + // the transaction ConsumedCapacity order sorts tables by this value, + // so whole seconds make two tables created in the same second tie and + // fall back to name order, which the service does not do. + let creation_millis = now.unix_timestamp() * 1000 + i64::from(now.millisecond()); + let creation_epoch = creation_millis as f64 / 1000.0; // Build the table metadata document let key_schema_bson = @@ -243,7 +248,7 @@ impl MongoEngine { "stream_specification": stream_bson.unwrap_or(bson::Bson::Null), "table_status": table_status, "status_transition_at": status_transition_at, - "creation_date_time": bson::DateTime::from_millis((creation_epoch * 1000.0) as i64), + "creation_date_time": bson::DateTime::from_millis(creation_millis), "table_size_bytes": 0_i64, "item_count": 0_i64, "table_arn": &table_arn_val, From 0aa4c28a06f5b557d1a7db1537d8a7dcd8bb102a Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 18:37:49 +0000 Subject: [PATCH 07/10] docs: state the transaction capacity tie rule per backend The ConsumedCapacity order row said two tables created in the same millisecond are listed in name order. The precision of the stored creation time differs per backend, so the tie does too: nanoseconds on SQLite, microseconds on PostgreSQL, milliseconds on MongoDB and Cassandra, and whole seconds for a MongoDB catalog row written before the precision fix. The row now says which, and names the backends the order was measured on. The service-side measurement no longer claims the three tables had a single hash key, which the recorded run does not state. Same wording in the test docstring and the PerTableCapacity doc comment. The empty-operand-list comment in the condition parser now cites the 2026-10-23 measurement as well as the 2026-10-14 one. --- crates/core/src/expression/parser.rs | 5 +++-- crates/engine/src/per_table_capacity.rs | 15 +++++++++------ docs/differences-from-dynamodb.md | 2 +- tests/test_transact_capacity_order.py | 2 +- 4 files changed, 14 insertions(+), 10 deletions(-) diff --git a/crates/core/src/expression/parser.rs b/crates/core/src/expression/parser.rs index a3b3398c7..c90905a5d 100755 --- a/crates/core/src/expression/parser.rs +++ b/crates/core/src/expression/parser.rs @@ -242,8 +242,9 @@ fn parse_function_call(tokens: &[Token], pos: &mut usize) -> Result); diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 185743da9..bbe0e1bac 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -147,7 +147,7 @@ The same on all four backends. | Provisioned throughput | Token bucket per table/partition | Token bucket per table/partition, matching DynamoDB's burst and refill behavior | | On-demand capacity | Automatic scaling | Fixed initial burst capacity (4000 WCU / 12000 RCU), no auto-scaling | | Throttling | Always on; throttles requests that exceed provisioned/burst capacity. No setting to disable | Configurable via `throttling_enabled` runtime setting (default: `true`) | -| Order of `ConsumedCapacity` entries in `TransactGetItems` and `TransactWriteItems` responses | Undocumented. Measured 2026-10-23 with three single-hash-key tables whose names sort opposite to their creation order, in three request orders: both operations listed the tables by creation time, oldest first, whatever order the request named them in (a request naming the newest table first still returned the oldest first). The order follows neither the request nor the table names | The same order on every backend: one entry per table, by the table's creation time, oldest first; two tables created in the same millisecond are listed in name order. The creation time comes from the catalog row the handlers already read for the key schema, so it costs no extra round-trip. Earlier builds ordered the entries by hash-map iteration, which varied between processes and backends | +| Order of `ConsumedCapacity` entries in `TransactGetItems` and `TransactWriteItems` responses | Undocumented. Measured 2026-10-23 with three tables whose names sort opposite to their creation order, in three request orders: both operations listed the tables by creation time, oldest first, whatever order the request named them in (a request naming the newest table first still returned the oldest first). The order follows neither the request nor the table names | One entry per table, by the table's creation time, oldest first; two tables with the same stored creation time are listed in name order. The precision of the stored time differs per backend, so the tie does too: nanoseconds on SQLite, microseconds on PostgreSQL, milliseconds on MongoDB and Cassandra. A MongoDB catalog row written by a build before this change holds whole seconds, so two tables created in the same second still tie there. Measured on SQLite, PostgreSQL and MongoDB; Cassandra reads the same catalog column and was not served in this round. The creation time comes from the catalog row the handlers already read for the key schema, so it costs no extra round-trip. Earlier builds ordered the entries by hash-map iteration, which varied between processes and backends | ## Operations Not Implemented diff --git a/tests/test_transact_capacity_order.py b/tests/test_transact_capacity_order.py index f82c69b4d..6da1705bf 100644 --- a/tests/test_transact_capacity_order.py +++ b/tests/test_transact_capacity_order.py @@ -5,7 +5,7 @@ Amazon DynamoDB documents no order for the ``ConsumedCapacity`` array of ``TransactGetItems`` and ``TransactWriteItems``. Measured on 2026-10-23 with -three single-hash-key tables whose names sort opposite to their creation order: +three tables whose names sort opposite to their creation order: both operations listed the tables oldest first, for every request order tried ([newest, middle, oldest], [middle, newest, oldest], [oldest, newest, middle]). The order follows neither the request nor the names. From 8621e3fc8ade3453aa7931842cc01b6ce7a7a6b1 Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 18:40:50 +0000 Subject: [PATCH 08/10] refactor(core): split the expression semantic pass by rule family semantics.rs had grown to 819 lines. It is now a module directory with one file per rule family: update_rules.rs for the update-expression rules, condition_rules.rs for the condition-expression rules, refs.rs for the name and value usage checks both passes share, errors.rs for the message builders, and tests.rs for the tests. mod.rs keeps the module documentation, the two function-name tables and the two public entry points. No behaviour change. The functions moved unedited; the only edits are the per-file imports, pub(super) on the helpers the siblings call, and the test module's imports in place of its use super::*. The nine semantic-pass tests keep their names and pass. --- crates/core/src/expression/semantics.rs | 819 ------------------ .../expression/semantics/condition_rules.rs | 159 ++++ .../core/src/expression/semantics/errors.rs | 59 ++ crates/core/src/expression/semantics/mod.rs | 64 ++ crates/core/src/expression/semantics/refs.rs | 49 ++ crates/core/src/expression/semantics/tests.rs | 389 +++++++++ .../src/expression/semantics/update_rules.rs | 162 ++++ 7 files changed, 882 insertions(+), 819 deletions(-) delete mode 100644 crates/core/src/expression/semantics.rs create mode 100644 crates/core/src/expression/semantics/condition_rules.rs create mode 100644 crates/core/src/expression/semantics/errors.rs create mode 100644 crates/core/src/expression/semantics/mod.rs create mode 100644 crates/core/src/expression/semantics/refs.rs create mode 100644 crates/core/src/expression/semantics/tests.rs create mode 100644 crates/core/src/expression/semantics/update_rules.rs diff --git a/crates/core/src/expression/semantics.rs b/crates/core/src/expression/semantics.rs deleted file mode 100644 index c3c8a1e77..000000000 --- a/crates/core/src/expression/semantics.rs +++ /dev/null @@ -1,819 +0,0 @@ -// Copyright 2026 ExtendDB contributors -// SPDX-License-Identifier: Apache-2.0 - -//! Semantic validation of parsed `UpdateExpression` and `ConditionExpression` -//! trees, run after the grammar pass and before any table is read. -//! -//! Amazon DynamoDB reports every syntax error first, then walks the parsed -//! tree once, in expression order, checking each node after its operands -//! (a post-order walk), and only then checks overlapping paths and unused -//! names and values. The order of the messages below was measured against -//! the service on 2026-10-14; each rule names the request that pinned it. -//! -//! Update expression, per node: -//! - a document path: every `#name` element is defined; -//! - a `:value` placeholder: defined; -//! - `+` and `-`: after both operands, a placeholder operand must be a -//! number (`SET a = :s + :n` with a string `:s`); -//! - a function call: after its operands, `list_append` may not be an -//! operand of a function or of `+`/`-` (`if_not_exists(list_append(:a, -//! :b), :v)`); the name must be `if_not_exists` or `list_append` in that -//! case (`IF_NOT_EXISTS(x, :v)` is refused); a condition function is -//! refused by name (`SET x = size(x)`); both take two operands; the first -//! operand of `if_not_exists` is a document path; a placeholder operand of -//! `list_append` is a list; -//! - an `ADD` or `DELETE` action: after the path and the placeholder, the -//! value is a number or a set for `ADD` and a set for `DELETE`; -//! - a section: after its actions, the keyword may not have appeared before -//! (`SET a = :v SET b = :q` reports `:q`, `SET a = :v SET b = :v ADD n :s` -//! reports the repeated `SET`). -//! -//! Condition expression, per node: -//! - operands of a comparison, `BETWEEN`, or `IN` are walked first; a -//! function in operand position must be `size`, and `size` in condition -//! position is refused (`attribute_exists(pk) = :v`, `size(pk)`); -//! - a function call: after its operands, the name must be one of the six -//! condition functions as written (`ATTRIBUTE_EXISTS(pk)` is refused), -//! `if_not_exists` and `list_append` are refused by name, the operand count -//! is fixed per function, `attribute_exists` and `attribute_not_exists` -//! take a document path (`attribute_type(:v, :t)` is accepted), the second operand of -//! `begins_with` is a string or binary, and the second operand of -//! `attribute_type` is a string naming a type; -//! - a comparison or `BETWEEN` with an ordering operator: a placeholder -//! operand is a string, number, or binary. - -use super::ast::{ArithOp, Expr, PathElement, UpdateAction}; -use super::kind::ExpressionKind; -use super::resolver::{ExpressionMaps, attribute_type_code, validate_ordering_operand_types}; -use super::update_parser::{ParsedUpdate, repeated_section_err}; -use crate::error::DynamoDbError; -use crate::types::AttributeValue; - -const CONDITION_FUNCTIONS: [&str; 6] = [ - "attribute_exists", - "attribute_not_exists", - "attribute_type", - "begins_with", - "contains", - "size", -]; - -const UPDATE_FUNCTIONS: [&str; 2] = ["if_not_exists", "list_append"]; - -/// Validate a parsed update expression against the service's semantic rules, -/// in the service's order, then reject overlapping paths. -/// -/// Messages carry the `Invalid UpdateExpression:` prefix. -/// -/// # Errors -/// -/// Returns `DynamoDbError::ValidationException` for the first rule the -/// expression breaks in the order described in the module documentation. -pub fn validate_update_semantics( - parsed: &ParsedUpdate, - maps: &ExpressionMaps, -) -> Result<(), DynamoDbError> { - let kind = ExpressionKind::Update; - let mut seen: Vec<&'static str> = Vec::new(); - for section in &parsed.sections { - for action in &parsed.actions[section.actions.clone()] { - match action { - UpdateAction::Set { path, value } => { - check_path_refs(path, maps, kind)?; - walk_set_value(value, maps, false)?; - } - UpdateAction::Remove { path } => check_path_refs(path, maps, kind)?, - UpdateAction::Add { path, value } => { - check_path_refs(path, maps, kind)?; - check_add_delete_value("ADD", value, maps)?; - } - UpdateAction::Delete { path, value } => { - check_path_refs(path, maps, kind)?; - check_add_delete_value("DELETE", value, maps)?; - } - } - } - if seen.contains(§ion.keyword) { - return Err(repeated_section_err(section.keyword)); - } - seen.push(section.keyword); - } - super::resolver::validate_update_paths_disjoint(&parsed.actions, maps) -} - -/// Walk a SET value. `nested` is true inside a function's operands or an -/// arithmetic operand, where `list_append` is refused. -fn walk_set_value(expr: &Expr, maps: &ExpressionMaps, nested: bool) -> Result<(), DynamoDbError> { - let kind = ExpressionKind::Update; - match expr { - Expr::Path(path) => check_path_refs(path, maps, kind), - Expr::Placeholder(name) => check_value_defined(name, maps, kind), - Expr::Arithmetic { left, op, right } => { - walk_set_value(left, maps, true)?; - walk_set_value(right, maps, true)?; - let symbol = match op { - ArithOp::Add => "+", - ArithOp::Sub => "-", - }; - for operand in [left.as_ref(), right.as_ref()] { - if let Expr::Placeholder(name) = operand - && let Some(val) = maps.values.get(name) - && !matches!(val, AttributeValue::N(_)) - { - return Err(operand_type_err(kind, symbol, attribute_type_code(val))); - } - } - Ok(()) - } - Expr::Function { name, args } => { - for arg in args { - walk_set_value(arg, maps, true)?; - } - if nested && name == "list_append" { - return Err(validation( - kind, - "The function is not allowed to be used this way in an expression; function: list_append", - )); - } - if !UPDATE_FUNCTIONS.contains(&name.as_str()) { - if CONDITION_FUNCTIONS.contains(&name.as_str()) { - return Err(validation( - kind, - &format!( - "The function is not allowed in an update expression; function: {name}" - ), - )); - } - return Err(invalid_function_name(kind, name)); - } - if args.len() != 2 { - return Err(operand_count_err(kind, name, args.len())); - } - if name == "if_not_exists" && !matches!(args[0], Expr::Path(_)) { - return Err(requires_document_path(kind, name)); - } - if name == "list_append" { - for arg in args { - if let Expr::Placeholder(p) = arg - && let Some(val) = maps.values.get(p) - && !matches!(val, AttributeValue::L(_)) - { - return Err(operand_type_err(kind, name, attribute_type_code(val))); - } - } - } - Ok(()) - } - // The update grammar produces no other node kinds in a SET value. - _ => Ok(()), - } -} - -/// `ADD` takes a number or a set; `DELETE` takes a set. The service names -/// the offending type in its own vocabulary and the same allowed set for -/// both operators (measured: `DELETE s :n` reports `NUMBER`, `typeSet: -/// ALLOWED_FOR_ADD_OPERAND`). -fn check_add_delete_value( - operator: &str, - value: &Expr, - maps: &ExpressionMaps, -) -> Result<(), DynamoDbError> { - let kind = ExpressionKind::Update; - let Expr::Placeholder(name) = value else { - return Ok(()); - }; - check_value_defined(name, maps, kind)?; - let Some(val) = maps.values.get(name) else { - return Ok(()); - }; - let type_name = match val { - AttributeValue::SS(_) | AttributeValue::NS(_) | AttributeValue::BS(_) => return Ok(()), - AttributeValue::N(_) if operator == "ADD" => return Ok(()), - AttributeValue::N(_) => "NUMBER", - AttributeValue::S(_) => "STRING", - AttributeValue::B(_) => "Binary", - AttributeValue::Bool(_) => "BOOL", - AttributeValue::Null => "NULL", - AttributeValue::L(_) => "LIST", - AttributeValue::M(_) => "MAP", - }; - Err(validation( - kind, - &format!( - "Incorrect operand type for operator or function; operator: {operator}, operand type: {type_name}, typeSet: ALLOWED_FOR_ADD_OPERAND" - ), - )) -} - -/// Validate a parsed condition, filter, or key-condition tree against the -/// service's semantic rules, in the service's order. -/// -/// `kind` names the parameter in the messages (`Invalid ConditionExpression:`, -/// `Invalid FilterExpression:`). -/// -/// # Errors -/// -/// Returns `DynamoDbError::ValidationException` for the first rule the -/// expression breaks in the order described in the module documentation. -pub fn validate_condition_semantics( - expr: &Expr, - maps: &ExpressionMaps, - kind: ExpressionKind, -) -> Result<(), DynamoDbError> { - walk_condition(expr, maps, kind) -} - -/// A node in boolean position: the root, or an operand of AND, OR, NOT. -fn walk_condition( - expr: &Expr, - maps: &ExpressionMaps, - kind: ExpressionKind, -) -> Result<(), DynamoDbError> { - match expr { - Expr::And(l, r) | Expr::Or(l, r) => { - walk_condition(l, maps, kind)?; - walk_condition(r, maps, kind) - } - Expr::Not(inner) => walk_condition(inner, maps, kind), - Expr::Compare { left, right, .. } => { - walk_condition_operand(left, maps, kind)?; - walk_condition_operand(right, maps, kind)?; - validate_ordering_operand_types(expr, maps).map_err(|e| with_kind(e, kind)) - } - Expr::Between { operand, low, high } => { - walk_condition_operand(operand, maps, kind)?; - walk_condition_operand(low, maps, kind)?; - walk_condition_operand(high, maps, kind)?; - validate_ordering_operand_types(expr, maps).map_err(|e| with_kind(e, kind)) - } - Expr::In { operand, list } => { - walk_condition_operand(operand, maps, kind)?; - for item in list { - walk_condition_operand(item, maps, kind)?; - } - Ok(()) - } - Expr::Function { name, args } => walk_condition_function(name, args, maps, kind, false), - // A bare path or placeholder in condition position is a syntax error - // the grammar pass already reported; `Expected` desugaring never - // produces one either. - Expr::Path(_) | Expr::Placeholder(_) | Expr::Arithmetic { .. } => Ok(()), - } -} - -/// A node in operand position: a side of a comparison, a BETWEEN bound, an -/// IN member, or a function operand. -fn walk_condition_operand( - expr: &Expr, - maps: &ExpressionMaps, - kind: ExpressionKind, -) -> Result<(), DynamoDbError> { - match expr { - Expr::Path(path) => check_path_refs(path, maps, kind), - Expr::Placeholder(name) => check_value_defined(name, maps, kind), - Expr::Function { name, args } => walk_condition_function(name, args, maps, kind, true), - _ => Ok(()), - } -} - -/// The checks on a condition function call, after its operands: the name, -/// then its position (`size` returns a value and is refused in condition -/// position; the five boolean functions are refused in operand position: -/// `attribute_type(pk, :v) = :v` reports the position, not the operand type), -/// then the operand count, the document-path rule, and the operand types. -fn walk_condition_function( - name: &str, - args: &[Expr], - maps: &ExpressionMaps, - kind: ExpressionKind, - operand_position: bool, -) -> Result<(), DynamoDbError> { - for arg in args { - walk_condition_operand(arg, maps, kind)?; - } - if !CONDITION_FUNCTIONS.contains(&name) { - if UPDATE_FUNCTIONS.contains(&name) { - return Err(validation( - kind, - &format!("The function is not allowed in a condition expression; function: {name}"), - )); - } - return Err(invalid_function_name(kind, name)); - } - if operand_position != (name == "size") { - return Err(not_allowed_this_way(kind, name)); - } - let expected = match name { - "attribute_exists" | "attribute_not_exists" | "size" => 1, - _ => 2, - }; - if args.len() != expected { - return Err(operand_count_err(kind, name, args.len())); - } - if matches!(name, "attribute_exists" | "attribute_not_exists") - && !matches!(args[0], Expr::Path(_)) - { - return Err(requires_document_path(kind, name)); - } - if name == "begins_with" - && let Expr::Placeholder(p) = &args[1] - && let Some(val) = maps.values.get(p) - && !matches!(val, AttributeValue::S(_) | AttributeValue::B(_)) - { - return Err(operand_type_err(kind, name, attribute_type_code(val))); - } - if name == "attribute_type" - && let Expr::Placeholder(p) = &args[1] - && let Some(val) = maps.values.get(p) - { - match val { - AttributeValue::S(type_name) => { - const VALID: [&str; 10] = - ["S", "SS", "N", "NS", "B", "BS", "BOOL", "NULL", "L", "M"]; - if !VALID.contains(&type_name.as_str()) { - return Err(validation( - kind, - &format!( - "Invalid attribute type name found; type: {type_name}, valid types: {{S,SS,N,NS,B,BS,BOOL,NULL,L,M}}" - ), - )); - } - } - other => return Err(operand_type_err(kind, name, attribute_type_code(other))), - } - } - Ok(()) -} - -fn check_path_refs( - path: &[PathElement], - maps: &ExpressionMaps, - kind: ExpressionKind, -) -> Result<(), DynamoDbError> { - for element in path { - if let PathElement::Attribute(name) = element - && let Some(ref_name) = name.strip_prefix('#') - && !maps.names.contains_key(ref_name) - { - return Err(validation( - kind, - &format!( - "An expression attribute name used in the document path is not defined; attribute name: #{ref_name}" - ), - )); - } - } - Ok(()) -} - -fn check_value_defined( - name: &str, - maps: &ExpressionMaps, - kind: ExpressionKind, -) -> Result<(), DynamoDbError> { - if maps.values.contains_key(name) { - Ok(()) - } else { - Err(validation( - kind, - &format!( - "An expression attribute value used in expression is not defined; attribute value: :{name}" - ), - )) - } -} - -fn validation(kind: ExpressionKind, msg: &str) -> DynamoDbError { - DynamoDbError::ValidationException(format!("Invalid {kind}: {msg}")) -} - -fn with_kind(err: DynamoDbError, kind: ExpressionKind) -> DynamoDbError { - match err { - DynamoDbError::ValidationException(msg) if !msg.starts_with("Invalid ") => { - validation(kind, &msg) - } - other => other, - } -} - -fn invalid_function_name(kind: ExpressionKind, name: &str) -> DynamoDbError { - validation(kind, &format!("Invalid function name; function: {name}")) -} - -fn not_allowed_this_way(kind: ExpressionKind, name: &str) -> DynamoDbError { - validation( - kind, - &format!( - "The function is not allowed to be used this way in an expression; function: {name}" - ), - ) -} - -fn operand_count_err(kind: ExpressionKind, name: &str, count: usize) -> DynamoDbError { - validation( - kind, - &format!( - "Incorrect number of operands for operator or function; operator or function: {name}, number of operands: {count}" - ), - ) -} - -fn requires_document_path(kind: ExpressionKind, name: &str) -> DynamoDbError { - validation( - kind, - &format!("Operator or function requires a document path; operator or function: {name}"), - ) -} - -fn operand_type_err(kind: ExpressionKind, name: &str, type_code: &str) -> DynamoDbError { - validation( - kind, - &format!( - "Incorrect operand type for operator or function; operator or function: {name}, operand type: {type_code}" - ), - ) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::expression::parser::parse_condition; - use crate::expression::tokenizer::tokenize; - use crate::expression::update_parser::parse_update_sections; - use std::collections::HashMap; - - const U: &str = "Invalid UpdateExpression: "; - const C: &str = "Invalid ConditionExpression: "; - - fn maps(names: &[(&str, &str)], values: &[(&str, AttributeValue)]) -> ExpressionMaps { - ExpressionMaps::new( - names - .iter() - .map(|(k, v)| (k.trim_start_matches('#').to_owned(), (*v).to_owned())) - .collect::>(), - values - .iter() - .map(|(k, v)| (k.trim_start_matches(':').to_owned(), v.clone())) - .collect::>(), - ) - } - - fn n() -> AttributeValue { - AttributeValue::N("1".to_owned()) - } - fn s() -> AttributeValue { - AttributeValue::S("x".to_owned()) - } - fn l() -> AttributeValue { - AttributeValue::L(vec![]) - } - - fn update_err(expr: &str, m: &ExpressionMaps) -> String { - let tokens = tokenize(expr).unwrap(); - let parsed = parse_update_sections(&tokens, Some(expr)).unwrap(); - match validate_update_semantics(&parsed, m) { - Ok(()) => "OK".to_owned(), - Err(DynamoDbError::ValidationException(msg)) => msg, - Err(other) => panic!("unexpected {other:?}"), - } - } - - fn condition_err(expr: &str, m: &ExpressionMaps) -> String { - let tokens = tokenize(expr).unwrap(); - let parsed = parse_condition(&tokens).unwrap(); - match validate_condition_semantics(&parsed, m, ExpressionKind::Condition) { - Ok(()) => "OK".to_owned(), - Err(DynamoDbError::ValidationException(msg)) => msg, - Err(other) => panic!("unexpected {other:?}"), - } - } - - // Every expectation below is a response captured from Amazon DynamoDB - // on 2026-10-14 (issue #108 notes, probe-fix-round-*.py). - - #[test] - fn update_refs_in_expression_order() { - let m = maps(&[], &[(":v", n())]); - assert_eq!( - update_err("SET a = :q, #n = :v", &m), - format!( - "{U}An expression attribute value used in expression is not defined; attribute value: :q" - ) - ); - assert_eq!( - update_err("SET #n = :q", &m), - format!( - "{U}An expression attribute name used in the document path is not defined; attribute name: #n" - ) - ); - } - - #[test] - fn repeated_section_after_its_own_actions() { - let m = maps(&[], &[(":v", n()), (":s", s()), (":n", n())]); - assert_eq!( - update_err("SET a = :v SET b = :q", &m), - format!( - "{U}An expression attribute value used in expression is not defined; attribute value: :q" - ) - ); - assert_eq!( - update_err("SET a = :v SET b = :v ADD n :s", &m), - format!("{U}The \"SET\" section can only be used once in an update expression;") - ); - assert_eq!( - update_err("ADD n :s ADD m :n", &m), - format!( - "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" - ) - ); - assert_eq!( - update_err("SET a = :n SET b = :s + :n", &m), - format!( - "{U}Incorrect operand type for operator or function; operator or function: +, operand type: S" - ) - ); - } - - #[test] - fn function_names_are_case_sensitive() { - let m = maps(&[], &[(":v", n())]); - assert_eq!( - update_err("SET x = IF_NOT_EXISTS(x, :v)", &m), - format!("{U}Invalid function name; function: IF_NOT_EXISTS") - ); - assert_eq!( - update_err("SET x = foo(x, :v)", &m), - format!("{U}Invalid function name; function: foo") - ); - assert_eq!( - update_err("SET x = attribute_exists(x)", &m), - format!( - "{U}The function is not allowed in an update expression; function: attribute_exists" - ) - ); - assert_eq!( - update_err("SET x = IF_NOT_EXISTS(x, :q)", &m), - format!( - "{U}An expression attribute value used in expression is not defined; attribute value: :q" - ) - ); - assert_eq!(update_err("SET x = if_not_exists(x, :v)", &m), "OK"); - } - - #[test] - fn nested_list_append_before_its_siblings() { - let m = maps(&[], &[(":a", l()), (":b", l()), (":n", n())]); - assert_eq!( - update_err("SET l = if_not_exists(list_append(:a, :b), :q)", &m), - format!( - "{U}The function is not allowed to be used this way in an expression; function: list_append" - ) - ); - assert_eq!( - update_err("SET l = if_not_exists(list_append(:q, :b), :a)", &m), - format!( - "{U}An expression attribute value used in expression is not defined; attribute value: :q" - ) - ); - assert_eq!( - update_err("SET l = list_append(:a, :a) + :n", &m), - format!( - "{U}The function is not allowed to be used this way in an expression; function: list_append" - ) - ); - assert_eq!( - update_err("SET l = list_append(if_not_exists(l, :a), :b)", &m), - "OK" - ); - } - - #[test] - fn operand_counts_and_document_path() { - let m = maps(&[], &[(":a", l()), (":s", s())]); - assert_eq!( - update_err("SET a = if_not_exists(:a)", &m), - format!( - "{U}Incorrect number of operands for operator or function; operator or function: if_not_exists, number of operands: 1" - ) - ); - assert_eq!( - update_err("SET a = if_not_exists(a, :q, :q2)", &m), - format!( - "{U}An expression attribute value used in expression is not defined; attribute value: :q" - ) - ); - assert_eq!( - update_err("SET l = if_not_exists(:a, :a)", &m), - format!( - "{U}Operator or function requires a document path; operator or function: if_not_exists" - ) - ); - assert_eq!( - update_err("SET l = list_append(:s, :a)", &m), - format!( - "{U}Incorrect operand type for operator or function; operator or function: list_append, operand type: S" - ) - ); - } - - #[test] - fn add_and_delete_operand_types() { - let m = maps( - &[], - &[ - (":s", s()), - (":n", n()), - (":b", AttributeValue::B(vec![1])), - ( - ":ss", - AttributeValue::SS(std::iter::once("a".to_owned()).collect()), - ), - (":l", l()), - ], - ); - assert_eq!( - update_err("ADD n :s", &m), - format!( - "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" - ) - ); - assert_eq!( - update_err("ADD n :b", &m), - format!( - "{U}Incorrect operand type for operator or function; operator: ADD, operand type: Binary, typeSet: ALLOWED_FOR_ADD_OPERAND" - ) - ); - assert_eq!( - update_err("DELETE s :n", &m), - format!( - "{U}Incorrect operand type for operator or function; operator: DELETE, operand type: NUMBER, typeSet: ALLOWED_FOR_ADD_OPERAND" - ) - ); - assert_eq!( - update_err("DELETE s :l", &m), - format!( - "{U}Incorrect operand type for operator or function; operator: DELETE, operand type: LIST, typeSet: ALLOWED_FOR_ADD_OPERAND" - ) - ); - assert_eq!(update_err("ADD n :n, s :ss DELETE t :ss", &m), "OK"); - assert_eq!( - update_err("ADD n :q", &m), - format!( - "{U}An expression attribute value used in expression is not defined; attribute value: :q" - ) - ); - } - - #[test] - fn overlap_after_the_walk() { - let m = maps(&[], &[(":v", n()), (":s", s())]); - assert_eq!( - update_err("SET a = :v, a.b = :v ADD n :s", &m), - format!( - "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" - ) - ); - assert_eq!( - update_err("SET a = :v, a.b = :v", &m), - format!( - "{U}Two document paths overlap with each other; must remove or rewrite one of these paths; path one: [a], path two: [a, b]" - ) - ); - } - - #[test] - fn condition_function_names_and_positions() { - let m = maps(&[], &[(":v", s()), (":n", n())]); - assert_eq!( - condition_err("ATTRIBUTE_EXISTS(pk)", &m), - format!("{C}Invalid function name; function: ATTRIBUTE_EXISTS") - ); - assert_eq!( - condition_err("foo(pk) AND bar(pk)", &m), - format!("{C}Invalid function name; function: foo") - ); - assert_eq!( - condition_err("if_not_exists(a, :v) = :v", &m), - format!( - "{C}The function is not allowed in a condition expression; function: if_not_exists" - ) - ); - assert_eq!( - condition_err("attribute_exists(pk) = :v", &m), - format!( - "{C}The function is not allowed to be used this way in an expression; function: attribute_exists" - ) - ); - assert_eq!( - condition_err("size(pk)", &m), - format!( - "{C}The function is not allowed to be used this way in an expression; function: size" - ) - ); - assert_eq!( - condition_err("NOT size(pk)", &m), - format!( - "{C}The function is not allowed to be used this way in an expression; function: size" - ) - ); - assert_eq!( - condition_err("size(attribute_exists(pk)) > :n", &m), - format!( - "{C}The function is not allowed to be used this way in an expression; function: attribute_exists" - ) - ); - assert_eq!( - condition_err("attribute_exists(size(pk))", &m), - format!( - "{C}Operator or function requires a document path; operator or function: attribute_exists" - ) - ); - assert_eq!( - condition_err("attribute_exists(:v)", &m), - format!( - "{C}Operator or function requires a document path; operator or function: attribute_exists" - ) - ); - assert_eq!( - condition_err("foo(#n)", &m), - format!( - "{C}An expression attribute name used in the document path is not defined; attribute name: #n" - ) - ); - assert_eq!( - condition_err(":q = #n", &m), - format!( - "{C}An expression attribute value used in expression is not defined; attribute value: :q" - ) - ); - assert_eq!( - condition_err("attribute_type(pk, :v) = :v", &m), - format!( - "{C}The function is not allowed to be used this way in an expression; function: attribute_type" - ) - ); - assert_eq!(condition_err("size(pk) = size(pk)", &m), "OK"); - assert_eq!(condition_err("begins_with(:v, pk)", &m), "OK"); - assert_eq!( - condition_err("size(pk) > :n AND attribute_exists(pk)", &m), - "OK" - ); - } - - #[test] - fn condition_operand_counts_and_types() { - let m = maps( - &[], - &[ - (":n", n()), - (":t", AttributeValue::S("S".to_owned())), - (":xx", AttributeValue::S("XX".to_owned())), - (":l", l()), - ], - ); - assert_eq!( - condition_err("attribute_exists(pk, pk)", &m), - format!( - "{C}Incorrect number of operands for operator or function; operator or function: attribute_exists, number of operands: 2" - ) - ); - assert_eq!( - condition_err("begins_with(pk)", &m), - format!( - "{C}Incorrect number of operands for operator or function; operator or function: begins_with, number of operands: 1" - ) - ); - assert_eq!( - condition_err("size(pk, pk) > :n", &m), - format!( - "{C}Incorrect number of operands for operator or function; operator or function: size, number of operands: 2" - ) - ); - assert_eq!( - condition_err("begins_with(pk, :n)", &m), - format!( - "{C}Incorrect operand type for operator or function; operator or function: begins_with, operand type: N" - ) - ); - assert_eq!( - condition_err("attribute_type(pk, :n)", &m), - format!( - "{C}Incorrect operand type for operator or function; operator or function: attribute_type, operand type: N" - ) - ); - assert_eq!( - condition_err("attribute_type(pk, :xx)", &m), - format!( - "{C}Invalid attribute type name found; type: XX, valid types: {{S,SS,N,NS,B,BS,BOOL,NULL,L,M}}" - ) - ); - assert_eq!( - condition_err("pk > :l", &m), - format!( - "{C}Incorrect operand type for operator or function; operator or function: >, operand type: L" - ) - ); - assert_eq!(condition_err("attribute_type(pk, :t)", &m), "OK"); - } -} diff --git a/crates/core/src/expression/semantics/condition_rules.rs b/crates/core/src/expression/semantics/condition_rules.rs new file mode 100644 index 000000000..a7a55ce93 --- /dev/null +++ b/crates/core/src/expression/semantics/condition_rules.rs @@ -0,0 +1,159 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The condition-expression rules of the semantic pass. See the module +//! documentation in the parent module for the order the service checks them in. + +use super::super::ast::Expr; +use super::super::kind::ExpressionKind; +use super::super::resolver::{ + ExpressionMaps, attribute_type_code, validate_ordering_operand_types, +}; +use super::errors::{ + invalid_function_name, not_allowed_this_way, operand_count_err, operand_type_err, + requires_document_path, validation, with_kind, +}; +use super::refs::{check_path_refs, check_value_defined}; +use super::{CONDITION_FUNCTIONS, UPDATE_FUNCTIONS}; +use crate::error::DynamoDbError; +use crate::types::AttributeValue; + +/// Validate a parsed condition, filter, or key-condition tree against the +/// service's semantic rules, in the service's order. +/// +/// `kind` names the parameter in the messages (`Invalid ConditionExpression:`, +/// `Invalid FilterExpression:`). +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` for the first rule the +/// expression breaks in the order described in the module documentation. +pub fn validate_condition_semantics( + expr: &Expr, + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + walk_condition(expr, maps, kind) +} + +/// A node in boolean position: the root, or an operand of AND, OR, NOT. +fn walk_condition( + expr: &Expr, + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + match expr { + Expr::And(l, r) | Expr::Or(l, r) => { + walk_condition(l, maps, kind)?; + walk_condition(r, maps, kind) + } + Expr::Not(inner) => walk_condition(inner, maps, kind), + Expr::Compare { left, right, .. } => { + walk_condition_operand(left, maps, kind)?; + walk_condition_operand(right, maps, kind)?; + validate_ordering_operand_types(expr, maps).map_err(|e| with_kind(e, kind)) + } + Expr::Between { operand, low, high } => { + walk_condition_operand(operand, maps, kind)?; + walk_condition_operand(low, maps, kind)?; + walk_condition_operand(high, maps, kind)?; + validate_ordering_operand_types(expr, maps).map_err(|e| with_kind(e, kind)) + } + Expr::In { operand, list } => { + walk_condition_operand(operand, maps, kind)?; + for item in list { + walk_condition_operand(item, maps, kind)?; + } + Ok(()) + } + Expr::Function { name, args } => walk_condition_function(name, args, maps, kind, false), + // A bare path or placeholder in condition position is a syntax error + // the grammar pass already reported; `Expected` desugaring never + // produces one either. + Expr::Path(_) | Expr::Placeholder(_) | Expr::Arithmetic { .. } => Ok(()), + } +} + +/// A node in operand position: a side of a comparison, a BETWEEN bound, an +/// IN member, or a function operand. +fn walk_condition_operand( + expr: &Expr, + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + match expr { + Expr::Path(path) => check_path_refs(path, maps, kind), + Expr::Placeholder(name) => check_value_defined(name, maps, kind), + Expr::Function { name, args } => walk_condition_function(name, args, maps, kind, true), + _ => Ok(()), + } +} + +/// The checks on a condition function call, after its operands: the name, +/// then its position (`size` returns a value and is refused in condition +/// position; the five boolean functions are refused in operand position: +/// `attribute_type(pk, :v) = :v` reports the position, not the operand type), +/// then the operand count, the document-path rule, and the operand types. +fn walk_condition_function( + name: &str, + args: &[Expr], + maps: &ExpressionMaps, + kind: ExpressionKind, + operand_position: bool, +) -> Result<(), DynamoDbError> { + for arg in args { + walk_condition_operand(arg, maps, kind)?; + } + if !CONDITION_FUNCTIONS.contains(&name) { + if UPDATE_FUNCTIONS.contains(&name) { + return Err(validation( + kind, + &format!("The function is not allowed in a condition expression; function: {name}"), + )); + } + return Err(invalid_function_name(kind, name)); + } + if operand_position != (name == "size") { + return Err(not_allowed_this_way(kind, name)); + } + let expected = match name { + "attribute_exists" | "attribute_not_exists" | "size" => 1, + _ => 2, + }; + if args.len() != expected { + return Err(operand_count_err(kind, name, args.len())); + } + if matches!(name, "attribute_exists" | "attribute_not_exists") + && !matches!(args[0], Expr::Path(_)) + { + return Err(requires_document_path(kind, name)); + } + if name == "begins_with" + && let Expr::Placeholder(p) = &args[1] + && let Some(val) = maps.values.get(p) + && !matches!(val, AttributeValue::S(_) | AttributeValue::B(_)) + { + return Err(operand_type_err(kind, name, attribute_type_code(val))); + } + if name == "attribute_type" + && let Expr::Placeholder(p) = &args[1] + && let Some(val) = maps.values.get(p) + { + match val { + AttributeValue::S(type_name) => { + const VALID: [&str; 10] = + ["S", "SS", "N", "NS", "B", "BS", "BOOL", "NULL", "L", "M"]; + if !VALID.contains(&type_name.as_str()) { + return Err(validation( + kind, + &format!( + "Invalid attribute type name found; type: {type_name}, valid types: {{S,SS,N,NS,B,BS,BOOL,NULL,L,M}}" + ), + )); + } + } + other => return Err(operand_type_err(kind, name, attribute_type_code(other))), + } + } + Ok(()) +} diff --git a/crates/core/src/expression/semantics/errors.rs b/crates/core/src/expression/semantics/errors.rs new file mode 100644 index 000000000..230a26ea5 --- /dev/null +++ b/crates/core/src/expression/semantics/errors.rs @@ -0,0 +1,59 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The message builders for the semantic rules. Each one carries the +//! service's text for one rule, with the parameter name `kind` supplies. + +use super::super::kind::ExpressionKind; +use crate::error::DynamoDbError; + +pub(super) fn validation(kind: ExpressionKind, msg: &str) -> DynamoDbError { + DynamoDbError::ValidationException(format!("Invalid {kind}: {msg}")) +} + +pub(super) fn with_kind(err: DynamoDbError, kind: ExpressionKind) -> DynamoDbError { + match err { + DynamoDbError::ValidationException(msg) if !msg.starts_with("Invalid ") => { + validation(kind, &msg) + } + other => other, + } +} + +pub(super) fn invalid_function_name(kind: ExpressionKind, name: &str) -> DynamoDbError { + validation(kind, &format!("Invalid function name; function: {name}")) +} + +pub(super) fn not_allowed_this_way(kind: ExpressionKind, name: &str) -> DynamoDbError { + validation( + kind, + &format!( + "The function is not allowed to be used this way in an expression; function: {name}" + ), + ) +} + +pub(super) fn operand_count_err(kind: ExpressionKind, name: &str, count: usize) -> DynamoDbError { + validation( + kind, + &format!( + "Incorrect number of operands for operator or function; operator or function: {name}, number of operands: {count}" + ), + ) +} + +pub(super) fn requires_document_path(kind: ExpressionKind, name: &str) -> DynamoDbError { + validation( + kind, + &format!("Operator or function requires a document path; operator or function: {name}"), + ) +} + +pub(super) fn operand_type_err(kind: ExpressionKind, name: &str, type_code: &str) -> DynamoDbError { + validation( + kind, + &format!( + "Incorrect operand type for operator or function; operator or function: {name}, operand type: {type_code}" + ), + ) +} diff --git a/crates/core/src/expression/semantics/mod.rs b/crates/core/src/expression/semantics/mod.rs new file mode 100644 index 000000000..f02149661 --- /dev/null +++ b/crates/core/src/expression/semantics/mod.rs @@ -0,0 +1,64 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Semantic validation of parsed `UpdateExpression` and `ConditionExpression` +//! trees, run after the grammar pass and before any table is read. +//! +//! Amazon DynamoDB reports every syntax error first, then walks the parsed +//! tree once, in expression order, checking each node after its operands +//! (a post-order walk), and only then checks overlapping paths and unused +//! names and values. The order of the messages below was measured against +//! the service on 2026-10-14; each rule names the request that pinned it. +//! +//! Update expression, per node: +//! - a document path: every `#name` element is defined; +//! - a `:value` placeholder: defined; +//! - `+` and `-`: after both operands, a placeholder operand must be a +//! number (`SET a = :s + :n` with a string `:s`); +//! - a function call: after its operands, `list_append` may not be an +//! operand of a function or of `+`/`-` (`if_not_exists(list_append(:a, +//! :b), :v)`); the name must be `if_not_exists` or `list_append` in that +//! case (`IF_NOT_EXISTS(x, :v)` is refused); a condition function is +//! refused by name (`SET x = size(x)`); both take two operands; the first +//! operand of `if_not_exists` is a document path; a placeholder operand of +//! `list_append` is a list; +//! - an `ADD` or `DELETE` action: after the path and the placeholder, the +//! value is a number or a set for `ADD` and a set for `DELETE`; +//! - a section: after its actions, the keyword may not have appeared before +//! (`SET a = :v SET b = :q` reports `:q`, `SET a = :v SET b = :v ADD n :s` +//! reports the repeated `SET`). +//! +//! Condition expression, per node: +//! - operands of a comparison, `BETWEEN`, or `IN` are walked first; a +//! function in operand position must be `size`, and `size` in condition +//! position is refused (`attribute_exists(pk) = :v`, `size(pk)`); +//! - a function call: after its operands, the name must be one of the six +//! condition functions as written (`ATTRIBUTE_EXISTS(pk)` is refused), +//! `if_not_exists` and `list_append` are refused by name, the operand count +//! is fixed per function, `attribute_exists` and `attribute_not_exists` +//! take a document path (`attribute_type(:v, :t)` is accepted), the second operand of +//! `begins_with` is a string or binary, and the second operand of +//! `attribute_type` is a string naming a type; +//! - a comparison or `BETWEEN` with an ordering operator: a placeholder +//! operand is a string, number, or binary. + +const CONDITION_FUNCTIONS: [&str; 6] = [ + "attribute_exists", + "attribute_not_exists", + "attribute_type", + "begins_with", + "contains", + "size", +]; + +const UPDATE_FUNCTIONS: [&str; 2] = ["if_not_exists", "list_append"]; + +mod condition_rules; +mod errors; +mod refs; +#[cfg(test)] +mod tests; +mod update_rules; + +pub use condition_rules::validate_condition_semantics; +pub use update_rules::validate_update_semantics; diff --git a/crates/core/src/expression/semantics/refs.rs b/crates/core/src/expression/semantics/refs.rs new file mode 100644 index 000000000..157e60573 --- /dev/null +++ b/crates/core/src/expression/semantics/refs.rs @@ -0,0 +1,49 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The name and value usage checks shared by both passes: every `#name` +//! element of a document path, and every `:value` placeholder, is defined. + +use super::super::ast::PathElement; +use super::super::kind::ExpressionKind; +use super::super::resolver::ExpressionMaps; +use super::errors::validation; +use crate::error::DynamoDbError; + +pub(super) fn check_path_refs( + path: &[PathElement], + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + for element in path { + if let PathElement::Attribute(name) = element + && let Some(ref_name) = name.strip_prefix('#') + && !maps.names.contains_key(ref_name) + { + return Err(validation( + kind, + &format!( + "An expression attribute name used in the document path is not defined; attribute name: #{ref_name}" + ), + )); + } + } + Ok(()) +} + +pub(super) fn check_value_defined( + name: &str, + maps: &ExpressionMaps, + kind: ExpressionKind, +) -> Result<(), DynamoDbError> { + if maps.values.contains_key(name) { + Ok(()) + } else { + Err(validation( + kind, + &format!( + "An expression attribute value used in expression is not defined; attribute value: :{name}" + ), + )) + } +} diff --git a/crates/core/src/expression/semantics/tests.rs b/crates/core/src/expression/semantics/tests.rs new file mode 100644 index 000000000..a4344337d --- /dev/null +++ b/crates/core/src/expression/semantics/tests.rs @@ -0,0 +1,389 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The semantic-pass tests. Each one names the request that pinned it. + +use super::{validate_condition_semantics, validate_update_semantics}; +use crate::error::DynamoDbError; +use crate::expression::kind::ExpressionKind; +use crate::expression::parser::parse_condition; +use crate::expression::resolver::ExpressionMaps; +use crate::expression::tokenizer::tokenize; +use crate::expression::update_parser::parse_update_sections; +use crate::types::AttributeValue; +use std::collections::HashMap; + +const U: &str = "Invalid UpdateExpression: "; +const C: &str = "Invalid ConditionExpression: "; + +fn maps(names: &[(&str, &str)], values: &[(&str, AttributeValue)]) -> ExpressionMaps { + ExpressionMaps::new( + names + .iter() + .map(|(k, v)| (k.trim_start_matches('#').to_owned(), (*v).to_owned())) + .collect::>(), + values + .iter() + .map(|(k, v)| (k.trim_start_matches(':').to_owned(), v.clone())) + .collect::>(), + ) +} + +fn n() -> AttributeValue { + AttributeValue::N("1".to_owned()) +} +fn s() -> AttributeValue { + AttributeValue::S("x".to_owned()) +} +fn l() -> AttributeValue { + AttributeValue::L(vec![]) +} + +fn update_err(expr: &str, m: &ExpressionMaps) -> String { + let tokens = tokenize(expr).unwrap(); + let parsed = parse_update_sections(&tokens, Some(expr)).unwrap(); + match validate_update_semantics(&parsed, m) { + Ok(()) => "OK".to_owned(), + Err(DynamoDbError::ValidationException(msg)) => msg, + Err(other) => panic!("unexpected {other:?}"), + } +} + +fn condition_err(expr: &str, m: &ExpressionMaps) -> String { + let tokens = tokenize(expr).unwrap(); + let parsed = parse_condition(&tokens).unwrap(); + match validate_condition_semantics(&parsed, m, ExpressionKind::Condition) { + Ok(()) => "OK".to_owned(), + Err(DynamoDbError::ValidationException(msg)) => msg, + Err(other) => panic!("unexpected {other:?}"), + } +} + +// Every expectation below is a response captured from Amazon DynamoDB +// on 2026-10-14 (issue #108 notes, probe-fix-round-*.py). + +#[test] +fn update_refs_in_expression_order() { + let m = maps(&[], &[(":v", n())]); + assert_eq!( + update_err("SET a = :q, #n = :v", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + update_err("SET #n = :q", &m), + format!( + "{U}An expression attribute name used in the document path is not defined; attribute name: #n" + ) + ); +} + +#[test] +fn repeated_section_after_its_own_actions() { + let m = maps(&[], &[(":v", n()), (":s", s()), (":n", n())]); + assert_eq!( + update_err("SET a = :v SET b = :q", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + update_err("SET a = :v SET b = :v ADD n :s", &m), + format!("{U}The \"SET\" section can only be used once in an update expression;") + ); + assert_eq!( + update_err("ADD n :s ADD m :n", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("SET a = :n SET b = :s + :n", &m), + format!( + "{U}Incorrect operand type for operator or function; operator or function: +, operand type: S" + ) + ); +} + +#[test] +fn function_names_are_case_sensitive() { + let m = maps(&[], &[(":v", n())]); + assert_eq!( + update_err("SET x = IF_NOT_EXISTS(x, :v)", &m), + format!("{U}Invalid function name; function: IF_NOT_EXISTS") + ); + assert_eq!( + update_err("SET x = foo(x, :v)", &m), + format!("{U}Invalid function name; function: foo") + ); + assert_eq!( + update_err("SET x = attribute_exists(x)", &m), + format!( + "{U}The function is not allowed in an update expression; function: attribute_exists" + ) + ); + assert_eq!( + update_err("SET x = IF_NOT_EXISTS(x, :q)", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!(update_err("SET x = if_not_exists(x, :v)", &m), "OK"); +} + +#[test] +fn nested_list_append_before_its_siblings() { + let m = maps(&[], &[(":a", l()), (":b", l()), (":n", n())]); + assert_eq!( + update_err("SET l = if_not_exists(list_append(:a, :b), :q)", &m), + format!( + "{U}The function is not allowed to be used this way in an expression; function: list_append" + ) + ); + assert_eq!( + update_err("SET l = if_not_exists(list_append(:q, :b), :a)", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + update_err("SET l = list_append(:a, :a) + :n", &m), + format!( + "{U}The function is not allowed to be used this way in an expression; function: list_append" + ) + ); + assert_eq!( + update_err("SET l = list_append(if_not_exists(l, :a), :b)", &m), + "OK" + ); +} + +#[test] +fn operand_counts_and_document_path() { + let m = maps(&[], &[(":a", l()), (":s", s())]); + assert_eq!( + update_err("SET a = if_not_exists(:a)", &m), + format!( + "{U}Incorrect number of operands for operator or function; operator or function: if_not_exists, number of operands: 1" + ) + ); + assert_eq!( + update_err("SET a = if_not_exists(a, :q, :q2)", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + update_err("SET l = if_not_exists(:a, :a)", &m), + format!( + "{U}Operator or function requires a document path; operator or function: if_not_exists" + ) + ); + assert_eq!( + update_err("SET l = list_append(:s, :a)", &m), + format!( + "{U}Incorrect operand type for operator or function; operator or function: list_append, operand type: S" + ) + ); +} + +#[test] +fn add_and_delete_operand_types() { + let m = maps( + &[], + &[ + (":s", s()), + (":n", n()), + (":b", AttributeValue::B(vec![1])), + ( + ":ss", + AttributeValue::SS(std::iter::once("a".to_owned()).collect()), + ), + (":l", l()), + ], + ); + assert_eq!( + update_err("ADD n :s", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("ADD n :b", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: ADD, operand type: Binary, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("DELETE s :n", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: DELETE, operand type: NUMBER, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("DELETE s :l", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: DELETE, operand type: LIST, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!(update_err("ADD n :n, s :ss DELETE t :ss", &m), "OK"); + assert_eq!( + update_err("ADD n :q", &m), + format!( + "{U}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); +} + +#[test] +fn overlap_after_the_walk() { + let m = maps(&[], &[(":v", n()), (":s", s())]); + assert_eq!( + update_err("SET a = :v, a.b = :v ADD n :s", &m), + format!( + "{U}Incorrect operand type for operator or function; operator: ADD, operand type: STRING, typeSet: ALLOWED_FOR_ADD_OPERAND" + ) + ); + assert_eq!( + update_err("SET a = :v, a.b = :v", &m), + format!( + "{U}Two document paths overlap with each other; must remove or rewrite one of these paths; path one: [a], path two: [a, b]" + ) + ); +} + +#[test] +fn condition_function_names_and_positions() { + let m = maps(&[], &[(":v", s()), (":n", n())]); + assert_eq!( + condition_err("ATTRIBUTE_EXISTS(pk)", &m), + format!("{C}Invalid function name; function: ATTRIBUTE_EXISTS") + ); + assert_eq!( + condition_err("foo(pk) AND bar(pk)", &m), + format!("{C}Invalid function name; function: foo") + ); + assert_eq!( + condition_err("if_not_exists(a, :v) = :v", &m), + format!( + "{C}The function is not allowed in a condition expression; function: if_not_exists" + ) + ); + assert_eq!( + condition_err("attribute_exists(pk) = :v", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: attribute_exists" + ) + ); + assert_eq!( + condition_err("size(pk)", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: size" + ) + ); + assert_eq!( + condition_err("NOT size(pk)", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: size" + ) + ); + assert_eq!( + condition_err("size(attribute_exists(pk)) > :n", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: attribute_exists" + ) + ); + assert_eq!( + condition_err("attribute_exists(size(pk))", &m), + format!( + "{C}Operator or function requires a document path; operator or function: attribute_exists" + ) + ); + assert_eq!( + condition_err("attribute_exists(:v)", &m), + format!( + "{C}Operator or function requires a document path; operator or function: attribute_exists" + ) + ); + assert_eq!( + condition_err("foo(#n)", &m), + format!( + "{C}An expression attribute name used in the document path is not defined; attribute name: #n" + ) + ); + assert_eq!( + condition_err(":q = #n", &m), + format!( + "{C}An expression attribute value used in expression is not defined; attribute value: :q" + ) + ); + assert_eq!( + condition_err("attribute_type(pk, :v) = :v", &m), + format!( + "{C}The function is not allowed to be used this way in an expression; function: attribute_type" + ) + ); + assert_eq!(condition_err("size(pk) = size(pk)", &m), "OK"); + assert_eq!(condition_err("begins_with(:v, pk)", &m), "OK"); + assert_eq!( + condition_err("size(pk) > :n AND attribute_exists(pk)", &m), + "OK" + ); +} + +#[test] +fn condition_operand_counts_and_types() { + let m = maps( + &[], + &[ + (":n", n()), + (":t", AttributeValue::S("S".to_owned())), + (":xx", AttributeValue::S("XX".to_owned())), + (":l", l()), + ], + ); + assert_eq!( + condition_err("attribute_exists(pk, pk)", &m), + format!( + "{C}Incorrect number of operands for operator or function; operator or function: attribute_exists, number of operands: 2" + ) + ); + assert_eq!( + condition_err("begins_with(pk)", &m), + format!( + "{C}Incorrect number of operands for operator or function; operator or function: begins_with, number of operands: 1" + ) + ); + assert_eq!( + condition_err("size(pk, pk) > :n", &m), + format!( + "{C}Incorrect number of operands for operator or function; operator or function: size, number of operands: 2" + ) + ); + assert_eq!( + condition_err("begins_with(pk, :n)", &m), + format!( + "{C}Incorrect operand type for operator or function; operator or function: begins_with, operand type: N" + ) + ); + assert_eq!( + condition_err("attribute_type(pk, :n)", &m), + format!( + "{C}Incorrect operand type for operator or function; operator or function: attribute_type, operand type: N" + ) + ); + assert_eq!( + condition_err("attribute_type(pk, :xx)", &m), + format!( + "{C}Invalid attribute type name found; type: XX, valid types: {{S,SS,N,NS,B,BS,BOOL,NULL,L,M}}" + ) + ); + assert_eq!( + condition_err("pk > :l", &m), + format!( + "{C}Incorrect operand type for operator or function; operator or function: >, operand type: L" + ) + ); + assert_eq!(condition_err("attribute_type(pk, :t)", &m), "OK"); +} diff --git a/crates/core/src/expression/semantics/update_rules.rs b/crates/core/src/expression/semantics/update_rules.rs new file mode 100644 index 000000000..3a63bc47e --- /dev/null +++ b/crates/core/src/expression/semantics/update_rules.rs @@ -0,0 +1,162 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The update-expression rules of the semantic pass. See the module +//! documentation in the parent module for the order the service checks them in. + +use super::super::ast::{ArithOp, Expr, UpdateAction}; +use super::super::kind::ExpressionKind; +use super::super::resolver::{ExpressionMaps, attribute_type_code}; +use super::super::update_parser::{ParsedUpdate, repeated_section_err}; +use super::errors::{ + invalid_function_name, operand_count_err, operand_type_err, requires_document_path, validation, +}; +use super::refs::{check_path_refs, check_value_defined}; +use super::{CONDITION_FUNCTIONS, UPDATE_FUNCTIONS}; +use crate::error::DynamoDbError; +use crate::types::AttributeValue; + +/// Validate a parsed update expression against the service's semantic rules, +/// in the service's order, then reject overlapping paths. +/// +/// Messages carry the `Invalid UpdateExpression:` prefix. +/// +/// # Errors +/// +/// Returns `DynamoDbError::ValidationException` for the first rule the +/// expression breaks in the order described in the module documentation. +pub fn validate_update_semantics( + parsed: &ParsedUpdate, + maps: &ExpressionMaps, +) -> Result<(), DynamoDbError> { + let kind = ExpressionKind::Update; + let mut seen: Vec<&'static str> = Vec::new(); + for section in &parsed.sections { + for action in &parsed.actions[section.actions.clone()] { + match action { + UpdateAction::Set { path, value } => { + check_path_refs(path, maps, kind)?; + walk_set_value(value, maps, false)?; + } + UpdateAction::Remove { path } => check_path_refs(path, maps, kind)?, + UpdateAction::Add { path, value } => { + check_path_refs(path, maps, kind)?; + check_add_delete_value("ADD", value, maps)?; + } + UpdateAction::Delete { path, value } => { + check_path_refs(path, maps, kind)?; + check_add_delete_value("DELETE", value, maps)?; + } + } + } + if seen.contains(§ion.keyword) { + return Err(repeated_section_err(section.keyword)); + } + seen.push(section.keyword); + } + super::super::resolver::validate_update_paths_disjoint(&parsed.actions, maps) +} + +/// Walk a SET value. `nested` is true inside a function's operands or an +/// arithmetic operand, where `list_append` is refused. +fn walk_set_value(expr: &Expr, maps: &ExpressionMaps, nested: bool) -> Result<(), DynamoDbError> { + let kind = ExpressionKind::Update; + match expr { + Expr::Path(path) => check_path_refs(path, maps, kind), + Expr::Placeholder(name) => check_value_defined(name, maps, kind), + Expr::Arithmetic { left, op, right } => { + walk_set_value(left, maps, true)?; + walk_set_value(right, maps, true)?; + let symbol = match op { + ArithOp::Add => "+", + ArithOp::Sub => "-", + }; + for operand in [left.as_ref(), right.as_ref()] { + if let Expr::Placeholder(name) = operand + && let Some(val) = maps.values.get(name) + && !matches!(val, AttributeValue::N(_)) + { + return Err(operand_type_err(kind, symbol, attribute_type_code(val))); + } + } + Ok(()) + } + Expr::Function { name, args } => { + for arg in args { + walk_set_value(arg, maps, true)?; + } + if nested && name == "list_append" { + return Err(validation( + kind, + "The function is not allowed to be used this way in an expression; function: list_append", + )); + } + if !UPDATE_FUNCTIONS.contains(&name.as_str()) { + if CONDITION_FUNCTIONS.contains(&name.as_str()) { + return Err(validation( + kind, + &format!( + "The function is not allowed in an update expression; function: {name}" + ), + )); + } + return Err(invalid_function_name(kind, name)); + } + if args.len() != 2 { + return Err(operand_count_err(kind, name, args.len())); + } + if name == "if_not_exists" && !matches!(args[0], Expr::Path(_)) { + return Err(requires_document_path(kind, name)); + } + if name == "list_append" { + for arg in args { + if let Expr::Placeholder(p) = arg + && let Some(val) = maps.values.get(p) + && !matches!(val, AttributeValue::L(_)) + { + return Err(operand_type_err(kind, name, attribute_type_code(val))); + } + } + } + Ok(()) + } + // The update grammar produces no other node kinds in a SET value. + _ => Ok(()), + } +} + +/// `ADD` takes a number or a set; `DELETE` takes a set. The service names +/// the offending type in its own vocabulary and the same allowed set for +/// both operators (measured: `DELETE s :n` reports `NUMBER`, `typeSet: +/// ALLOWED_FOR_ADD_OPERAND`). +fn check_add_delete_value( + operator: &str, + value: &Expr, + maps: &ExpressionMaps, +) -> Result<(), DynamoDbError> { + let kind = ExpressionKind::Update; + let Expr::Placeholder(name) = value else { + return Ok(()); + }; + check_value_defined(name, maps, kind)?; + let Some(val) = maps.values.get(name) else { + return Ok(()); + }; + let type_name = match val { + AttributeValue::SS(_) | AttributeValue::NS(_) | AttributeValue::BS(_) => return Ok(()), + AttributeValue::N(_) if operator == "ADD" => return Ok(()), + AttributeValue::N(_) => "NUMBER", + AttributeValue::S(_) => "STRING", + AttributeValue::B(_) => "Binary", + AttributeValue::Bool(_) => "BOOL", + AttributeValue::Null => "NULL", + AttributeValue::L(_) => "LIST", + AttributeValue::M(_) => "MAP", + }; + Err(validation( + kind, + &format!( + "Incorrect operand type for operator or function; operator: {operator}, operand type: {type_name}, typeSet: ALLOWED_FOR_ADD_OPERAND" + ), + )) +} From 000e218548f4027c24d4e371f6c469229c0a3813 Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 18:41:33 +0000 Subject: [PATCH 09/10] refactor(core): move the update-parser tests into their own file update_parser.rs had grown from 571 lines to 630. It is now a module directory: mod.rs holds the parser at 417 lines and tests.rs holds the 21 tests at 215 lines. No behaviour change. The test module moved unedited, keeping its use super::*, and the tests keep their names. One comment inside it lost an em dash. --- .../mod.rs} | 215 +----------------- .../src/expression/update_parser/tests.rs | 215 ++++++++++++++++++ 2 files changed, 216 insertions(+), 214 deletions(-) rename crates/core/src/expression/{update_parser.rs => update_parser/mod.rs} (67%) mode change 100755 => 100644 create mode 100644 crates/core/src/expression/update_parser/tests.rs diff --git a/crates/core/src/expression/update_parser.rs b/crates/core/src/expression/update_parser/mod.rs old mode 100755 new mode 100644 similarity index 67% rename from crates/core/src/expression/update_parser.rs rename to crates/core/src/expression/update_parser/mod.rs index a3c280753..7e2fd3f0a --- a/crates/core/src/expression/update_parser.rs +++ b/crates/core/src/expression/update_parser/mod.rs @@ -414,217 +414,4 @@ fn validation_err(msg: &str) -> DynamoDbError { } #[cfg(test)] -mod tests { - use super::*; - use crate::expression::ast::{ArithOp, Expr, PathElement, UpdateAction}; - use crate::expression::tokenizer::tokenize; - - fn parse(input: &str) -> Result, DynamoDbError> { - let tokens = tokenize(input)?; - parse_update(&tokens) - } - - #[test] - fn parse_simple_set() { - let actions = parse("SET price = :p").unwrap(); - assert_eq!(actions.len(), 1); - assert!(matches!(&actions[0], UpdateAction::Set { path, .. } if path.len() == 1)); - } - - #[test] - fn parse_multiple_set_actions() { - let actions = parse("SET a = :v1, b = :v2, c = :v3").unwrap(); - assert_eq!(actions.len(), 3); - assert!( - actions - .iter() - .all(|a| matches!(a, UpdateAction::Set { .. })) - ); - } - - #[test] - fn parse_set_with_arithmetic_add() { - let actions = parse("SET price = price + :inc").unwrap(); - assert_eq!(actions.len(), 1); - if let UpdateAction::Set { value, .. } = &actions[0] { - assert!(matches!( - value, - Expr::Arithmetic { - op: ArithOp::Add, - .. - } - )); - } else { - panic!("Expected Set action"); - } - } - - #[test] - fn parse_set_with_arithmetic_sub() { - let actions = parse("SET stock = stock - :dec").unwrap(); - if let UpdateAction::Set { value, .. } = &actions[0] { - assert!(matches!( - value, - Expr::Arithmetic { - op: ArithOp::Sub, - .. - } - )); - } else { - panic!("Expected Set action"); - } - } - - #[test] - fn parse_set_with_if_not_exists() { - let actions = parse("SET price = if_not_exists(price, :default)").unwrap(); - if let UpdateAction::Set { value, .. } = &actions[0] { - if let Expr::Function { name, args } = value { - assert_eq!(name, "if_not_exists"); - assert_eq!(args.len(), 2); - } else { - panic!("Expected Function expression"); - } - } else { - panic!("Expected Set action"); - } - } - - #[test] - fn parse_set_with_list_append() { - let actions = parse("SET tags = list_append(tags, :newtags)").unwrap(); - if let UpdateAction::Set { value, .. } = &actions[0] { - assert!(matches!(value, Expr::Function { name, .. } if name == "list_append")); - } else { - panic!("Expected Set action"); - } - } - - #[test] - fn parse_simple_remove() { - let actions = parse("REMOVE oldattr").unwrap(); - assert_eq!(actions.len(), 1); - assert!(matches!(&actions[0], UpdateAction::Remove { .. })); - } - - #[test] - fn parse_multiple_remove() { - let actions = parse("REMOVE a, b, c").unwrap(); - assert_eq!(actions.len(), 3); - } - - #[test] - fn parse_set_and_remove_combined() { - let actions = parse("SET a = :v REMOVE b").unwrap(); - assert_eq!(actions.len(), 2); - assert!(matches!(&actions[0], UpdateAction::Set { .. })); - assert!(matches!(&actions[1], UpdateAction::Remove { .. })); - } - - #[test] - fn parse_add_action() { - let actions = parse("ADD counter :inc").unwrap(); - assert_eq!(actions.len(), 1); - assert!(matches!(&actions[0], UpdateAction::Add { .. })); - } - - #[test] - fn parse_delete_action() { - let actions = parse("DELETE colors :remove_colors").unwrap(); - assert_eq!(actions.len(), 1); - assert!(matches!(&actions[0], UpdateAction::Delete { .. })); - } - - #[test] - fn parse_all_four_action_types() { - let actions = parse("SET a = :v ADD b :inc DELETE c :rm REMOVE d").unwrap(); - assert_eq!(actions.len(), 4); - } - - #[test] - fn parse_nested_path_in_set() { - let actions = parse("SET address.city = :city").unwrap(); - if let UpdateAction::Set { path, .. } = &actions[0] { - assert_eq!(path.len(), 2); - assert_eq!(path[0], PathElement::Attribute("address".into())); - assert_eq!(path[1], PathElement::Attribute("city".into())); - } else { - panic!("Expected Set action"); - } - } - - #[test] - fn parse_list_index_in_set() { - let actions = parse("SET items[0] = :v").unwrap(); - if let UpdateAction::Set { path, .. } = &actions[0] { - assert_eq!(path.len(), 2); - assert_eq!(path[1], PathElement::Index(0)); - } else { - panic!("Expected Set action"); - } - } - - #[test] - fn parse_empty_expression_rejected() { - let tokens = tokenize("").unwrap(); - // Empty tokens — parse_update should fail - let err = parse_update(&tokens).unwrap_err(); - assert!( - matches!(err, DynamoDbError::ValidationException(msg) if msg.contains("at least one action")) - ); - } - - #[test] - fn parse_invalid_action_keyword_rejected() { - let err = parse("price = :v").unwrap_err(); - assert!( - matches!(err, DynamoDbError::ValidationException(msg) if msg.contains("expected SET")) - ); - } - - #[test] - fn parse_name_ref_in_path() { - let actions = parse("SET #n = :v").unwrap(); - if let UpdateAction::Set { path, .. } = &actions[0] { - assert_eq!(path[0], PathElement::Attribute("#n".into())); - } else { - panic!("Expected Set action"); - } - } - - #[test] - fn redundant_parens_rejected() { - let err = parse("SET c = ((c - :v))").unwrap_err(); - assert!( - matches!(&err, DynamoDbError::ValidationException(msg) if msg.contains("redundant parentheses")), - "Expected redundant parens error, got: {err:?}" - ); - } - - #[test] - fn single_parens_accepted() { - let actions = parse("SET c = (c - :v)").unwrap(); - assert!(matches!(&actions[0], UpdateAction::Set { .. })); - } - - #[test] - fn unmatched_paren_rejected() { - // Opening paren with no matching close must be a ValidationException, - // not a panic or a silently-accepted expression. - let err = parse("SET c = (c - :v").unwrap_err(); - assert!( - matches!(&err, DynamoDbError::ValidationException(msg) if msg.contains("closing")), - "Expected unmatched-paren error, got: {err:?}" - ); - } - - #[test] - fn unmatched_close_paren_rejected() { - // A stray closing paren with no matching open must also be rejected. - let err = parse("SET c = c - :v)").unwrap_err(); - assert!( - matches!(&err, DynamoDbError::ValidationException(_)), - "Expected validation error for stray ')', got: {err:?}" - ); - } -} +mod tests; diff --git a/crates/core/src/expression/update_parser/tests.rs b/crates/core/src/expression/update_parser/tests.rs new file mode 100644 index 000000000..41758716a --- /dev/null +++ b/crates/core/src/expression/update_parser/tests.rs @@ -0,0 +1,215 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The update-expression parser tests. + +use super::*; +use crate::expression::ast::{ArithOp, Expr, PathElement, UpdateAction}; +use crate::expression::tokenizer::tokenize; + +fn parse(input: &str) -> Result, DynamoDbError> { + let tokens = tokenize(input)?; + parse_update(&tokens) +} + +#[test] +fn parse_simple_set() { + let actions = parse("SET price = :p").unwrap(); + assert_eq!(actions.len(), 1); + assert!(matches!(&actions[0], UpdateAction::Set { path, .. } if path.len() == 1)); +} + +#[test] +fn parse_multiple_set_actions() { + let actions = parse("SET a = :v1, b = :v2, c = :v3").unwrap(); + assert_eq!(actions.len(), 3); + assert!( + actions + .iter() + .all(|a| matches!(a, UpdateAction::Set { .. })) + ); +} + +#[test] +fn parse_set_with_arithmetic_add() { + let actions = parse("SET price = price + :inc").unwrap(); + assert_eq!(actions.len(), 1); + if let UpdateAction::Set { value, .. } = &actions[0] { + assert!(matches!( + value, + Expr::Arithmetic { + op: ArithOp::Add, + .. + } + )); + } else { + panic!("Expected Set action"); + } +} + +#[test] +fn parse_set_with_arithmetic_sub() { + let actions = parse("SET stock = stock - :dec").unwrap(); + if let UpdateAction::Set { value, .. } = &actions[0] { + assert!(matches!( + value, + Expr::Arithmetic { + op: ArithOp::Sub, + .. + } + )); + } else { + panic!("Expected Set action"); + } +} + +#[test] +fn parse_set_with_if_not_exists() { + let actions = parse("SET price = if_not_exists(price, :default)").unwrap(); + if let UpdateAction::Set { value, .. } = &actions[0] { + if let Expr::Function { name, args } = value { + assert_eq!(name, "if_not_exists"); + assert_eq!(args.len(), 2); + } else { + panic!("Expected Function expression"); + } + } else { + panic!("Expected Set action"); + } +} + +#[test] +fn parse_set_with_list_append() { + let actions = parse("SET tags = list_append(tags, :newtags)").unwrap(); + if let UpdateAction::Set { value, .. } = &actions[0] { + assert!(matches!(value, Expr::Function { name, .. } if name == "list_append")); + } else { + panic!("Expected Set action"); + } +} + +#[test] +fn parse_simple_remove() { + let actions = parse("REMOVE oldattr").unwrap(); + assert_eq!(actions.len(), 1); + assert!(matches!(&actions[0], UpdateAction::Remove { .. })); +} + +#[test] +fn parse_multiple_remove() { + let actions = parse("REMOVE a, b, c").unwrap(); + assert_eq!(actions.len(), 3); +} + +#[test] +fn parse_set_and_remove_combined() { + let actions = parse("SET a = :v REMOVE b").unwrap(); + assert_eq!(actions.len(), 2); + assert!(matches!(&actions[0], UpdateAction::Set { .. })); + assert!(matches!(&actions[1], UpdateAction::Remove { .. })); +} + +#[test] +fn parse_add_action() { + let actions = parse("ADD counter :inc").unwrap(); + assert_eq!(actions.len(), 1); + assert!(matches!(&actions[0], UpdateAction::Add { .. })); +} + +#[test] +fn parse_delete_action() { + let actions = parse("DELETE colors :remove_colors").unwrap(); + assert_eq!(actions.len(), 1); + assert!(matches!(&actions[0], UpdateAction::Delete { .. })); +} + +#[test] +fn parse_all_four_action_types() { + let actions = parse("SET a = :v ADD b :inc DELETE c :rm REMOVE d").unwrap(); + assert_eq!(actions.len(), 4); +} + +#[test] +fn parse_nested_path_in_set() { + let actions = parse("SET address.city = :city").unwrap(); + if let UpdateAction::Set { path, .. } = &actions[0] { + assert_eq!(path.len(), 2); + assert_eq!(path[0], PathElement::Attribute("address".into())); + assert_eq!(path[1], PathElement::Attribute("city".into())); + } else { + panic!("Expected Set action"); + } +} + +#[test] +fn parse_list_index_in_set() { + let actions = parse("SET items[0] = :v").unwrap(); + if let UpdateAction::Set { path, .. } = &actions[0] { + assert_eq!(path.len(), 2); + assert_eq!(path[1], PathElement::Index(0)); + } else { + panic!("Expected Set action"); + } +} + +#[test] +fn parse_empty_expression_rejected() { + let tokens = tokenize("").unwrap(); + // Empty tokens, so parse_update must fail. + let err = parse_update(&tokens).unwrap_err(); + assert!( + matches!(err, DynamoDbError::ValidationException(msg) if msg.contains("at least one action")) + ); +} + +#[test] +fn parse_invalid_action_keyword_rejected() { + let err = parse("price = :v").unwrap_err(); + assert!(matches!(err, DynamoDbError::ValidationException(msg) if msg.contains("expected SET"))); +} + +#[test] +fn parse_name_ref_in_path() { + let actions = parse("SET #n = :v").unwrap(); + if let UpdateAction::Set { path, .. } = &actions[0] { + assert_eq!(path[0], PathElement::Attribute("#n".into())); + } else { + panic!("Expected Set action"); + } +} + +#[test] +fn redundant_parens_rejected() { + let err = parse("SET c = ((c - :v))").unwrap_err(); + assert!( + matches!(&err, DynamoDbError::ValidationException(msg) if msg.contains("redundant parentheses")), + "Expected redundant parens error, got: {err:?}" + ); +} + +#[test] +fn single_parens_accepted() { + let actions = parse("SET c = (c - :v)").unwrap(); + assert!(matches!(&actions[0], UpdateAction::Set { .. })); +} + +#[test] +fn unmatched_paren_rejected() { + // Opening paren with no matching close must be a ValidationException, + // not a panic or a silently-accepted expression. + let err = parse("SET c = (c - :v").unwrap_err(); + assert!( + matches!(&err, DynamoDbError::ValidationException(msg) if msg.contains("closing")), + "Expected unmatched-paren error, got: {err:?}" + ); +} + +#[test] +fn unmatched_close_paren_rejected() { + // A stray closing paren with no matching open must also be rejected. + let err = parse("SET c = c - :v)").unwrap_err(); + assert!( + matches!(&err, DynamoDbError::ValidationException(_)), + "Expected validation error for stray ')', got: {err:?}" + ); +} From 4f4fc2126f2ba0e0949455aff09f0a7e3700a9dc Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 19:14:16 +0000 Subject: [PATCH 10/10] style: remove two em dashes from comments this branch added The update-parser ADD/DELETE doc comment and the scan Segment check comment carried an em dash. Both are now two clauses. --- crates/core/src/expression/update_parser/mod.rs | 2 +- crates/engine/src/scan.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/core/src/expression/update_parser/mod.rs b/crates/core/src/expression/update_parser/mod.rs index 7e2fd3f0a..24764535f 100644 --- a/crates/core/src/expression/update_parser/mod.rs +++ b/crates/core/src/expression/update_parser/mod.rs @@ -390,7 +390,7 @@ fn parse_delete_actions( Ok(()) } -/// Parse a value for ADD/DELETE — only placeholders are valid. +/// Parse a value for ADD/DELETE. Only placeholders are valid. fn parse_add_delete_value(tokens: &[Token], pos: &mut usize) -> Result { if *pos >= tokens.len() { return Err(validation_err( diff --git a/crates/engine/src/scan.rs b/crates/engine/src/scan.rs index 7fdd19799..d69a26ec8 100755 --- a/crates/engine/src/scan.rs +++ b/crates/engine/src/scan.rs @@ -85,7 +85,7 @@ pub async fn handle_scan( // runs ahead of every expression check (measured 2026-10-14: a Scan with // an undefined filter value and Limit 0 reports the limit; with Segment // and no TotalSegments it reports TotalSegments). - // Validate Segment/TotalSegments — DynamoDB returns different messages per direction + // Validate Segment/TotalSegments. DynamoDB returns a different message per direction match (input.segment, input.total_segments) { (Some(_), None) => { return Err(DynamoDbError::ValidationException(