Repository navigation
feat(sdk): run experiments on datasets and send scores - #121
Merged
Merged
Conversation
runExperiment gets a dataset version page by page and makes the experiment. It runs the task on each item, with concurrency and runs for each item. It sends the outputs and the local scores in batches of at most 100. Then it completes the experiment, waits for the server evaluators, and returns the summary. Each item run has its own trace. Each span of the run has td.experiment.id and td.dataset.item_id. In CI mode, runExperiment records the GitHub Actions run, and it rejects when the check is not successful. score sends a score on a trace, a span, or a session to /v1/scores. It never rejects. It sends its errors to onError. getPrompt and the new calls now use one function for the key, the base URL, and the Retry-After header.
minScoresFromEnv now makes its object from pairs. Before, the name __proto__ set the prototype of the object, and its mark went away with no error. A number score now prints as Math.round(value * 100) / 100, a rule that the Python SDK can copy. The tests now also cover the experiment attributes on log records and the request order that the code holds.
run_experiment and run_experiment_async are the Python twins of runExperiment, with the same options, defaults, errors, and requests. The sync call runs the items in threads, each in a copy of the context of the caller. The async call runs them as asyncio tasks. The two read the dataset page by page and post the outputs in batches of at most 100. score and score_async send a score to /v1/scores. They never raise and send their errors to on_error. The new _api module holds the key and base URL, one JSON request with its retry rules, the redirect guard, and the Retry-After parser. The prompt code now uses it too. The checks count lengths in UTF-16 units, as the API does.
actions/experiment is a composite action. It runs the eval script of the user, a .ts, .js, or .py file or a command, with the dataset and the min-scores inputs in the variables that the SDKs read. Then it writes the result to the job summary and to one comment on the pull request, and it fails the job when the script exits with an error. The inputs go to the scripts through env, never into a run line. The results file has mode 0600, and the comment escapes the names. A token that cannot write the comment gives a warning. CI now runs shellcheck on the action script.
The READMEs of the two core SDKs get the Experiments and the Scores sections. They tell which data goes to telemetry.dev when output capture is off: the outputs, the task errors, the scorer reasons, and the metadata. The conformance document gets the experiment attributes and the wire contract of the two SDKs: the requests, the retry rules, the limits, the score rules, CI mode, and the variables of the action.
send() called fetch as a method of the Api object. Workers and browsers reject a fetch call that has a different `this` object. Thus score and runExperiment did not operate there without a `fetch` option. send() calls fetch as a function, not as a method. When a run stops after runExperiment makes the experiment, it sends the complete request before it rejects. Examples are an item post or a page fetch that is not successful, and more than 10,000 items. The server then completes the experiment with its item runs, or sets the status to failed when it has none. An error of that request goes to onError, and runExperiment rejects with the first error. The create request tries again only after a 429. The server sends a 503 when its database work is longer than its time limit. That 503 can come after the commit, and a retry then makes a second experiment. Each run of an item gets its own copy of the item. Thus a task that changes its input does not change the next run. The SDK cuts a pull request title without half of a surrogate pair, because Postgres rejects a lone surrogate in jsonb. A score field that can be null can also be missing, as in the Python SDK. A result without checkStatus stays an invalid_response. score() without options reports invalid_score and does not reject. New tests do checks of the 4 MiB batch limit, the 10,000-item limit, the 30-minute wait, and the retry after a timeout or a network error.
An output that JSON cannot hold stopped the full run outside the guard of its item. An example is a dict that holds itself. An exception with a __str__ that raises did the same. The SDK sends None as the output, as the TypeScript SDK does, and the name of the type as the error. A lone surrogate in an output or an error becomes U+FFFD. The request body is the JSON of JSON.stringify, which escapes a lone surrogate in a label. When a run stops after run_experiment makes the experiment, it sends the complete request before it raises. This also occurs after Ctrl-C and after a cancellation. An error of that request goes to on_error. An error of an item run stops all workers. The create request tries again only after a 429, as in the TypeScript SDK. Each run of an item gets a deep copy of the item. The requests for experiments, scores, and prompts send the User-Agent telemetry-dev-python/<version>, because Cloudflare rejects Python-urllib. New tests do checks of the 4 MiB batch limit, the 10,000-item limit, the 30-minute wait, and the retry after a timeout or a dropped connection.
The action ran a .ts script with npx --yes tsx. npx then downloads the newest tsx from npm, in a step that has the API key. The action uses the tsx in node_modules/.bin of the working directory or of a directory above it. If it finds no tsx, it stops with a clear error. report.mjs used each comment that started with its marker, thus it tried to change a pasted copy. It changes only a comment of its token. For a user token, that is a comment of its login. The GITHUB_TOKEN and app tokens cannot read /user, and they write as a bot. The README tells that the key must be for all environments or for experiment. It tells that the capture settings do not apply to item outputs, and that a pull request from a fork gets no API key. The example does not run the job for those pull requests.
The conformance document gives the complete request after a run stops, and the create retry only after a 429. It also gives the copy of the item for each run and the outputs that JSON cannot hold. It gives the Python User-Agent, the fetch call as a function, the fields that a result must have, and the cut of the title.
The base-url input of the experiment action and the baseUrl comment in api.ts gave the old default, https://ingest.telemetry.dev. The default base URL of the SDKs changed to https://api.telemetry.dev in #109.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Apps can run experiments on a telemetry.dev dataset from code and from CI, and send scores, with the TypeScript and Python SDKs.
runExperiment/run_experimentreads the dataset 100 items at a time, makes the experiment, runs the task of the app on each item, and sends the outputs and the local scores in batches of 100. Each item run is a trace of its own withtd.experiment.idandtd.dataset.item_idon each span. Then it waits for the evaluators that telemetry.dev runs, prints the scores, and returns the result with the URL of the experiment page.score/score()sends one score of a trace or a span.The action in
actions/experimentruns the experiment script of a pull request and writes the result as a comment on it. A score under its pass mark fails the step.The routes that the SDK calls are in telemetry-dev/telemetry.dev#526. Release the SDK only after that PR deploys, so that
runExperimentandscoredo not get 404.Tests:
packages/sdk/tests/experiments.test.ts(29 tests, for example "runs each item of each page, posts batches of 100 on one version, then completes" and "a failed post of items stops new items, completes the experiment, and rejects with the API error"),packages/sdk/tests/scores.test.ts(8),packages/sdk/tests/experiment-action.test.ts(11, therun.shandreport.mjsof the action),sdks/python/tests/test_experiments.py(33, the twins of the TypeScript tests), andsdks/python/tests/test_scores.py(6), with the full TypeScript and Python suites.Related: telemetry-dev/telemetry.dev#526