Skip to content

Add Support for Vector Data Type in SQL for GraphQL#3714

Open
RubenCerna2079 wants to merge 44 commits into
mainfrom
dev/rubencerna/vector-support-graphql
Open

Add Support for Vector Data Type in SQL for GraphQL#3714
RubenCerna2079 wants to merge 44 commits into
mainfrom
dev/rubencerna/vector-support-graphql

Conversation

@RubenCerna2079

@RubenCerna2079 RubenCerna2079 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Why make this change?

What is this change?

  • In order to allow graphql to read array types that are not [string] we moved the section that parses the scalar values to types hot chocolate can use in order output them to the user. This new CoerceJsonLeafValueToRuntimeType method is then used in a loop to allow support for arrays inside the ExecutionHelper.cs file.
  • In multiple places throughout the code, DAB looks at the SyntaxKind of the values it is working with in order to know if the value can be directly used or if it has a relationship with another entity. In order to allow graphql to write to the data base, we now add a new function TryGetUnderlyingFieldKind in ISqlMetadataProvider.cs that checks if the value is a list and it comes from a row that has the IsArrayType. If that condition is met, then it means the list value we have should be processed as a regular ScalarType and not as a List/Object that is a relationship to a different entity. This method is used in the following files: MultipleMutationInputValidator.cs, MultipleCreateOrderHelper.cs, SqlMutationEnginge.cs.
  • Lastly, as part to allow graphql to write to the data base, we add a new GetStringifiedValue function that checks if the value that we are going to parse to the data base is a list or a regular value and transforms it into a string accordingly. Since we set all values as strings, this is something that cannot be directly done with a list as the .ToString() function will return a useless value. This value is then parsed before being used as a parameter in the data base query.

How was this tested?

  • Integration Tests
  • Unit Tests
    Added tests that query (read) from the data base as well as mutation (change) to the data base such as create, update, and delete.

Sample Request(s)

Query

query {
  dbo_normalvectors {
    items {
      Embedding
      ProductID
    }
  }
}
image

Create

mutation {
  createdbo_normalvector(
    item: { ProductID: 10000, Embedding: [125, 1.22222, 0.75] }
  ) {
    ProductID
    Embedding
  }
}
image

Update

mutation {
  updatedbo_normalvector(
    ProductID: 10000,
    item: { Embedding: [0.532, 200, 100.152] }
  ) {
    ProductID
    Embedding
  }
}
image

Delete

mutation {
  deletedbo_normalvector(
    ProductID: 10000
  ) {
    ProductID
    Embedding
  }
}
image

souvikghosh04 and others added 30 commits June 10, 2026 10:51
Joint runtime + driver bump that prerequisites issue #2768 (MSSQL JSON
data type support). Behavior-preserving — no production logic changes.

WHY BOTH BUMPS ARE REQUIRED TOGETHER
====================================

SqlDbType.Json (numeric value 35) is a BCL enum value in System.Data,
added in .NET 9 and present in .NET 10. It is NOT a Microsoft.Data.SqlClient
symbol — SqlClient does not get to add values to a BCL enum it does not
own. On .NET 8, the SqlDbType enum stops at DateTimeOffset = 34, so

    Enum.TryParse<SqlDbType>("json", ignoreCase: true, out _)

in TypeHelper.GetSystemTypeFromSqlDbType returns false regardless of
which SqlClient version is installed. The companion feature PR for #2768
plans to add a single dictionary entry

    [SqlDbType.Json] = typeof(string)

which simply will not compile on net8.0. Both upgrades must therefore
land together as a single prerequisite.

WHY .NET 10 AND NOT .NET 9
==========================

- .NET 10 is the current Long-Term Support release (Nov 2025+).
- .NET 9 is Standard-Term Support, EOL May 2026.
- DAB's current .NET 8 line reaches EOL November 2026.

Going straight to LTS avoids a second forced upgrade inside 12 months.

SCOPE OF CHANGES (NO BEHAVIOR CHANGE INTENDED)
==============================================

Runtime / SDK pin:
  - global.json: 8.0.420 -> 10.0.301

Target framework (all 11 src/**/*.csproj):
  - net8.0 -> net10.0

NuGet packages (Directory.Packages.props):
  - Microsoft.Data.SqlClient                    5.2.3  -> 6.0.2
  - Microsoft.Extensions.Caching.Memory         8.0.1  -> 10.0.0
  - Microsoft.Extensions.Caching.Abstractions   9.0.0  -> 10.0.0
  - Microsoft.Extensions.Primitives             9.0.0  -> 10.0.0
  - Microsoft.Extensions.Configuration.Binder   9.0.0  -> 10.0.0
  - Microsoft.Extensions.Configuration.Json     9.0.0  -> 10.0.0
  - Microsoft.Extensions.Caching.StackExchangeRedis 9.0.3 -> 10.0.0

The Microsoft.Extensions.* bumps are required because OpenTelemetry's
transitive chain (Microsoft.Extensions.Logging.Configuration 10.0.0)
demands Configuration.Binder >= 10.0.0 — staying on 9.0.0 produces
NU1605 package-downgrade errors (warning-as-error).

Service.csproj (Microsoft.NET.Sdk.Web) — drop now-redundant
PackageReferences (NU1510, warning-as-error):
  - Microsoft.Extensions.Configuration.Binder  (in shared framework)
  - Microsoft.Extensions.Configuration.Json    (in shared framework)

Azure DevOps pipelines (9 occurrences across 8 files):
  - UseDotNet@2 version: 8.0.x -> 10.0.x

Container base images (Dockerfile):
  - mcr.microsoft.com/dotnet/sdk:8.0-cbl-mariner2.0   -> 10.0-azurelinux3.0
  - mcr.microsoft.com/dotnet/aspnet:8.0-cbl-mariner2.0 -> 10.0-azurelinux3.0
  (cbl-mariner2.0 tag does not exist for .NET 9+; Azure Linux 3.0 is
   Microsoft's documented successor.)

Aspire AppHost (src/Aspire.AppHost/AppHost.cs):
  - WithArgs("-f", "net8.0") -> WithArgs("-f", "net10.0")  (mssql + pg)

Build/publish scripts:
  - scripts/publish.ps1:               $dotnetTargetFrameworks net8.0 -> net10.0
  - scripts/create-manifest-file.ps1:  $dotnetTargetFrameworks + hashtable
                                       keys net8.0_{rid} -> net10.0_{rid}
                                       (TODO marker: release-engineering to
                                       confirm download URLs/hashes resolve)

License file rename:
  - external_licenses/Microsoft.Data.SqlClient.SNI.5.2.0.License.txt
    -> external_licenses/Microsoft.Data.SqlClient.SNI.6.0.0.License.txt
  - scripts/notice-generation.ps1: path updated to match
  - TODO marker at top of license file: contents still 5.2.0 text;
    release-engineering to refresh from upstream before merge.

SUPPRESSED ASPDEPR008 (warning-as-error) — DELIBERATE
=====================================================

ASP.NET Core 10 deprecates IWebHostBuilder / IWebHost in favor of
WebApplicationBuilder / IHost. Two locations in this repo still consume
the obsolete types and would block compilation under the repo's
TreatWarningsAsErrors=true setting:

  - src/Service/Program.cs (2 sites): test-only helpers
    CreateWebHostBuilder + CreateWebHostFromInMemoryUpdatableConfBuilder
    are consumed by the existing TestServer fixture which takes
    IWebHostBuilder.
  - src/Service.Tests/Configuration/ConfigurationTests.cs (3 sites):
    the consumer of those helpers.

Migrating from WebHost to HostBuilder/WebApplicationBuilder is a
behavior-affecting refactor and explicitly OUT OF SCOPE for this
no-behavior-change prerequisite PR. Suppression is added at the
project level (NoWarn=ASPDEPR008 on Service.csproj + Service.Tests.csproj)
with an inline TODO comment pointing at this branch and a follow-up
issue link placeholder.

VALIDATION
==========

dotnet --version          : 10.0.301
dotnet restore --nologo   : 11/11 projects restored (8.05s avg)
dotnet build  -c Debug    : 0 warnings, 0 errors, 10.32s

Multi-engine test categories (MsSql, PostgreSql, MySql, CosmosDb_NoSql,
DwSql) were NOT run locally; they MUST run green on the Azure DevOps
multi-engine matrix before this PR merges. That matrix is the gate.

NEXT STEPS BEFORE MERGE
=======================

1. Release-engineering: refresh
   external_licenses/Microsoft.Data.SqlClient.SNI.6.0.0.License.txt
   from upstream for the SNI 6.0.0 version (currently still 5.2.0 text
   with a TODO marker at top).
2. Release-engineering: confirm
   scripts/create-manifest-file.ps1 net10.0_{linux,win,osx}-x64 download
   URLs and SHA hashes resolve once the .NET 10 publish cycle runs.
3. Multi-engine CI matrix runs green (MsSql, PostgreSql, MySql,
   CosmosDb_NoSql, DwSql) — that is the merge gate.
4. File follow-up issue for ASPDEPR008 migration (WebHost ->
   WebApplicationBuilder / IHost) so the NoWarn suppressions can be
   removed in a future behavior-changing PR.
- Fix UseDotNet displayName v8.0.x -> v10.0.x across pipeline files

- Replace internal branch-path TODOs with durable follow-up notes

- Rename SNI license to 6.0.2 (actual runtime version) and fix notice-generation reference

- Bump Microsoft.AspNetCore.{TestHost,Authorization,Authentication.JwtBearer,Mvc.Testing} to 10.0.0
Aspire.Hosting pulls MessagePack 2.5.192 transitively (via KubernetesClient), which has high-severity advisory GHSA-hv8m-jj95-wg3x (CVE-2026-48109). The repo treats NU1903 as error, failing restore. Pin to patched 2.5.301 via CPM and add a direct reference in Aspire.AppHost to force the transitive upgrade.
…or change

SqlClient 6.x changed the internal SqlError 9-parameter constructor: the
win32ErrorCode parameter is now Int32 (was UInt32 in 5.x), and there are now
multiple 9-parameter overloads. Select the constructor by exact parameter
types and pass (int)0 for win32ErrorCode so CreateSqlException works across
SqlClient versions.
Starting with .NET 10, setting an environment variable to an empty string
preserves it as an empty value instead of deleting it (which previously
surfaced here as null). This caused ValidateAspNetCoreUrls to treat an empty
ASPNETCORE_URLS as invalid and exit with error, a regression from .NET 8 where
an empty value was equivalent to unset. Explicitly treat an empty value as
valid (Kestrel falls back to default URLs), preserving the prior behavior.
Whitespace-only values remain invalid.
These three files were modified by an earlier merge in a way unrelated to the
.NET 10 / SqlClient 6.x upgrade, reverting recent main formatting/cleanup:

- CLRtoJsonValueTypeUnitTests.cs: restore the DBTYPE_RESOLUTION_ERROR constant
  that was inadvertently removed.
- SqlUpdateQueryStructure.cs: restore main's formatting of the update-fields
  block.
- MutationEngineFactory.cs: restore main's version.

Restoring them to match main keeps this PR scoped to the upgrade and addresses
reviewer feedback (constant removal and formatting).
@RubenCerna2079 RubenCerna2079 changed the base branch from main to dev/rubencerna/first-changes-vector-type July 10, 2026 18:20
@RubenCerna2079 RubenCerna2079 changed the title Dev/rubencerna/vector support graphql Add Support for Vector Data Type in SQL for GraphQL Jul 10, 2026
@RubenCerna2079 RubenCerna2079 added this to the July 2026 milestone Jul 11, 2026
@RubenCerna2079 RubenCerna2079 linked an issue Jul 11, 2026 that may be closed by this pull request
Base automatically changed from dev/rubencerna/first-changes-vector-type to main July 13, 2026 21:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds end-to-end GraphQL support for SQL Server vector/array-backed columns by (1) coercing leaf-list results into HotChocolate runtime scalar types, and (2) ensuring list-valued mutation inputs for array columns are treated as scalars (not relationships) and correctly parameterized for SQL.

Changes:

  • Add leaf-list result coercion (CoerceJsonLeafValueToRuntimeType) so GraphQL can return list-of-scalar fields (e.g., vector exposed as [Single]) without HotChocolate leaf coercion errors.
  • Add ISqlMetadataProvider.TryGetUnderlyingFieldKind and plumb it through mutation validation/parsing so array columns aren’t misclassified as relationship inputs.
  • Add MSSQL integration tests + enable GraphQL for the VectorType test entity, plus stringify list inputs for SQL parameters.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/Service.Tests/SqlTests/GraphQLQueryTests/MsSqlGraphQLVectorTypesTests.cs Adds GraphQL integration coverage for vector read/query + create/update/delete mutations.
src/Service.Tests/dab-config.MsSql.json Enables GraphQL for the VectorType entity in MSSQL test config.
src/Core/Services/TypeHelper.cs Adds SystemType→SyntaxKind mapping helper used for array column kind detection.
src/Core/Services/MultipleMutationInputValidator.cs Uses underlying kind detection to avoid treating array scalar lists as relationship fields.
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs Implements TryGetUnderlyingFieldKind for SQL-backed metadata providers.
src/Core/Services/MetadataProviders/ISqlMetadataProvider.cs Adds the TryGetUnderlyingFieldKind API to the provider contract.
src/Core/Services/MetadataProviders/CosmosSqlMetadataProvider.cs Implements the new API as “not supported” for Cosmos.
src/Core/Services/ExecutionHelper.cs Refactors scalar coercion and adds list-of-leaf coercion to runtime types for HotChocolate.
src/Core/Resolvers/SqlMutationEngine.cs Plumbs metadata provider into mutation input parsing to distinguish array columns from relationships.
src/Core/Resolvers/Sql Query Structures/SqlUpdateQueryStructure.cs Uses GetStringifiedValue so list values are correctly converted before parameter parsing.
src/Core/Resolvers/Sql Query Structures/SqlInsertQueryStructure.cs Uses GetStringifiedValue for insert parameter values.
src/Core/Resolvers/Sql Query Structures/BaseSqlQueryStructure.cs Adds GetStringifiedValue to serialize list inputs into JSON array strings for parsing.
src/Core/Resolvers/MultipleCreateOrderHelper.cs Uses underlying kind detection to treat array scalar lists as scalars.
config-generators/mssql-commands.txt Updates generator command to enable GraphQL for VectorType.

// Licensed under the MIT License.

using System.Net;
using System.Text.Json;
using Azure.DataApiBuilder.Core.Services;
using Azure.DataApiBuilder.Service.Exceptions;
using Azure.DataApiBuilder.Service.GraphQLBuilder.Mutations;
using HotChocolate.Language;
Comment on lines +400 to +414
if (TryGetBackingColumn(entityName, fieldName, out string? columnName))
{
SourceDefinition sourceDefinition = GetSourceDefinition(entityName);
ColumnDefinition column = sourceDefinition.Columns[columnName];

// If the column is an array type, we need to get the syntax kind from the element system type.
if (column.IsArrayType && TypeHelper.TryGetSyntaxKindFromSystemType(column.ElementSystemType!, out fieldKind))
{
return true;
}
}

fieldKind = 0;
return false;
}
Comment on lines +255 to +265
/// <summary>
///
/// </summary>
/// <param name="entityName"></param>
/// <param name="fieldName"></param>
/// <param name="fieldKind"></param>
/// <returns></returns>
public bool TryGetUnderlyingFieldKind(
string entityName,
string fieldName,
[NotNullWhen(true)] out SyntaxKind fieldKind);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

Add Vector Support for GraphQL

5 participants