Skip to content

add per usecase example notebooks for quickstarter #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adithyasamavedhi-ttd wants to merge 1 commit into
base: main
Choose a base branch
from
Open

add per usecase example notebooks for quickstarter #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
535 changes: 368 additions & 167 deletions README.md
View file
Open in desktop

Large diffs are not rendered by default.

200 changes: 200 additions & 0 deletions example_notebook/Deletion and Opt-Out (DSR) Example Notebook.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": "# Deletion and Opt-Out (DSR) Example Notebook\n\nHonour a user's data subject request \u2014 delete their data, or opt them out of future\ntargeting \u2014 across every segment tied to your account.\n\n## Is this the right notebook for you?\n\n| | |\n|---|---|\n| **Who Should Use This?** | Anyone processing data subject requests: advertisers, data providers, merchants |\n| **What Does the Notebook Do?** | Deletes a user's data, or suppresses them from future targeting |\n| **What Data Does It Send?** | One row per identity \u2014 `id_type` and `id_value` only |\n| **Destination Trade Desk Endpoints** | The three `/data/deletion-optout/*` endpoints (via the `DeletionOptOutAdvertiserContext`, `DeletionOptOutThirdPartyContext` and `DeletionOptOutMerchantContext` classes of the ttd-databricks SDK) |\n\n### Deletion or opt-out?\n\nBoth actions use the same endpoints and the same input schema. Only `request_type` differs:\n\n| `request_type` | Effect |\n|---|---|\n| `PartnerDsrRequestType.OPT_OUT` | The user is suppressed from future targeting |\n| `PartnerDsrRequestType.DELETION` | The user's data is removed |\n\n### Which endpoint?\n\nPick the one matching the data you are acting on. Step 3 has a cell for each.\n\n| Your data | Context | Relevant OpenTTD API Documentation |\n|---|---|---|\n| First-party audience data | `DeletionOptOutAdvertiserContext` | [Advertiser](https://open.thetradedesk.com/advertiser/docsApp/GuidesAdvertiser/data/doc/post-data-deletion-optout-advertiser) \u00b7 [External provider](https://open.thetradedesk.com/provider/docsApp/GuidesProvider/audience/doc/post-data-deletion-optout-advertiser-external) |\n| Third-party audience data | `DeletionOptOutThirdPartyContext` | [Third party](https://open.thetradedesk.com/provider/docsApp/GuidesProvider/audience/doc/post-data-deletion-optout-thirdparty) |\n| Merchant data | `DeletionOptOutMerchantContext` | [Merchant](https://open.thetradedesk.com/provider/docsApp/GuidesProvider/retail/doc/post-data-deletion-optout-merchant) |\n\n## What you need\n\n- [ ] A TTD Platform API token \u2014 see [Create an API token](https://open.thetradedesk.com/advertiser/docsApp/GuidesAdvertiser/data/doc/DataApiCallsAdvertiser#ui-method-create)\n- [ ] The ID for your endpoint: advertiser ID, data provider ID, or merchant ID"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Step 1: Install the SDK"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "%pip install ttd-databricks\n\ndbutils.library.restartPython()"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Step 2: Configure credentials\n\nFor a first run you can paste values inline. In production, read them from Databricks Secrets:\n\n```python\nAPI_TOKEN = dbutils.secrets.get(scope=\"ttd\", key=\"api-token\")\nADVERTISER_ID = dbutils.secrets.get(scope=\"ttd\", key=\"advertiser-id\")\n```\n\n> **Note:** Authenticate with a Platform API token, sent as the `TTD-Auth` header. Generate one\n> in the OpenTTD Access Management app \u2014 see [Create an API token](https://open.thetradedesk.com/advertiser/docsApp/GuidesAdvertiser/data/doc/DataApiCallsAdvertiser#ui-method-create). Secret keys and\n> `TtdSignature` headers are a legacy method and are not supported by this SDK."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "API_TOKEN = \"<your-ttd-auth-token>\"\n\n# Set whichever applies to the endpoint you are using in Step 3.\nADVERTISER_ID = \"<your-advertiser-id>\"\nDATA_PROVIDER_ID = \"<your-data-provider-id>\"\nMERCHANT_ID = 0 # integer, not a string"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Step 3: Create the client and context\n\nRun **one** of the three cells below \u2014 whichever matches your data. Each sets `context`\nand `ENDPOINT`; every later step works the same way regardless of which you chose.\n\n`PartnerDsrRequestType` comes from `ttd_data.models`, not from the `ttd_databricks` package."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "from ttd_data.models import PartnerDsrRequestType\n\nfrom ttd_databricks_python.ttd_databricks import (\n TTDEndpoint,\n TtdDatabricksClient,\n get_ttd_input_schema,\n)\n\n# Two ways to create the client \u2014 this notebook uses (ii) everywhere below.\n# i) Dependency injection \u2014 you build the DataClient and pass it in:\n# from ttd_data import DataClient\n# client = TtdDatabricksClient(data_api_client=DataClient(), api_token=API_TOKEN)\n# ii) Factory \u2014 from_params builds the DataClient for you:\nclient = TtdDatabricksClient.from_params(api_token=API_TOKEN)\n\n# Switch to PartnerDsrRequestType.DELETION to delete instead of opt out.\nREQUEST_TYPE = PartnerDsrRequestType.OPT_OUT"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "### Option A \u2014 first-party (advertiser) data"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "from ttd_databricks_python.ttd_databricks import TTDEndpoint, DeletionOptOutAdvertiserContext\n\nENDPOINT = TTDEndpoint.DELETION_OPTOUT_ADVERTISER\ncontext = DeletionOptOutAdvertiserContext(\n advertiser_id=ADVERTISER_ID,\n request_type=REQUEST_TYPE,\n data_provider_id=None, # optional; set only if you are an external provider\n)\n\nprint(f\"Context: {context}\")"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "### Option B \u2014 third-party (data provider) data"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "from ttd_databricks_python.ttd_databricks import TTDEndpoint, DeletionOptOutThirdPartyContext\n\nENDPOINT = TTDEndpoint.DELETION_OPTOUT_THIRDPARTY\ncontext = DeletionOptOutThirdPartyContext(\n data_provider_id=DATA_PROVIDER_ID,\n request_type=REQUEST_TYPE,\n brand_id=None, # optional\n)\n\nprint(f\"Context: {context}\")"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "### Option C \u2014 merchant data"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "from ttd_databricks_python.ttd_databricks import TTDEndpoint, DeletionOptOutMerchantContext\n\nENDPOINT = TTDEndpoint.DELETION_OPTOUT_MERCHANT\ncontext = DeletionOptOutMerchantContext(\n merchant_id=MERCHANT_ID, # integer\n request_type=REQUEST_TYPE,\n)\n\nprint(f\"Context: {context}\")"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Step 4: Inspect the required input schema\n\nAll three endpoints take the same two mandatory columns and nothing else:\n`id_type` and `id_value`. There is no segment or timestamp \u2014 the request applies across\nevery segment tied to your account."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "from ttd_databricks_python.ttd_databricks import get_ttd_input_schema\nfrom ttd_databricks_python.ttd_databricks.schemas import get_required_column_names\n\ninput_schema = get_ttd_input_schema(ENDPOINT)\n\nprint(\"Mandatory columns:\", get_required_column_names(ENDPOINT))\nprint(\"\\nFull input schema:\")\nfor field in input_schema.fields:\n print(f\" {field.name}: {field.dataType.simpleString()} (nullable={field.nullable})\")"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Step 5: Prepare your input DataFrame\n\nOne row per identity to delete or opt out.\n\n> **Tip:** Start with a handful of rows. These requests are not reversible, so confirming\n> the flow on a small sample matters more here than anywhere else."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "rows = [\n {\"id_type\": \"TDID\", \"id_value\": \"123e4567-e89b-12d3-a456-426652340000\"},\n {\"id_type\": \"DAID\", \"id_value\": \"a9342d1f-69f1-4bf8-bc2b-1f20eb451f21\"},\n {\"id_type\": \"UID2\", \"id_value\": \"48MjlfIUZpOKNAm9nod7/jCLAXUYsnE1tpVHQSDS0uo=\"},\n {\"id_type\": \"RampID\", \"id_value\": \"XY1005wXyWPB1SgpMUKIpzA0I3UaLEz-2lg0wFAr1PWK7FMhs\"},\n]\n\ninput_df = spark.createDataFrame(rows, schema=input_schema)\ndisplay(input_df)"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Step 6: Sending Data\n\nThere are two ways to send your data. Pick one \u2014 you do not need both.\n\n| | Ad hoc | Batch processing |\n|---|---|---|\n| **State management** | None. Every call sends every row you give it. | Provided. A metadata table records progress, so each run sends only rows added since the last one. |\n| **Input** | A DataFrame you build in the notebook | A Delta input table |\n| **Output** | Returned inline as a DataFrame | Written to a Delta output table |\n| **Best for** | One-off loads and first tests | Recurring pipelines |"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "### Step 6a: Ad Hoc Usage (No State Management Provided)\n\n`push_data` sends your DataFrame straight to the Data API and returns the input columns\nenriched with per-row status. The SDK keeps no record of what it has already sent, so\nre-running this cell sends every row again.\n\nIt does not raise on API or row-level failures \u2014 every outcome is reported inline."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "result_df = client.push_data(df=input_df, context=context, batch_size=1600)\n\ndisplay(result_df)"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "`push_data` adds these columns to your input:\n\n| Column | Meaning |\n|---|---|\n| `success` | `True` if the row was accepted |\n| `error_code` | Failure category, `null` on success |\n| `error_message` | Human-readable reason, `null` on success |\n| `processed_timestamp` | When the row was submitted |\n| `uid2_resolutions` | Raw identifier \u2192 UID2 mapping, empty unless `uid2_config` was set |"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "from pyspark.sql.functions import col\n\ntotal = result_df.count()\nsucceeded = result_df.filter(col(\"success\")).count()\n\nprint(f\"Total: {total} | Succeeded: {succeeded} | Failed: {total - succeeded}\")\n\nfailed_df = result_df.filter(~col(\"success\"))\nif failed_df.count():\n display(failed_df.select(\"error_code\", \"error_message\"))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "### Step 6b: Batch Processing (State Management Provided)\n\n`batch_process` reads from a Delta input table, writes results to an output table, and\nrecords how far it got in a metadata table. With `process_new_records_only=True`, each run\npicks up only the rows added since the last successful run, so you can schedule it without\nre-sending history.\n\n**One time steps:** Create the three Delta tables. Every future run reuses these same tables \u2014\nthe metadata table is what remembers your progress, so do not drop or recreate it between\nruns. The `setup_*` methods are safe to re-run: they return the existing table if it is\nalready there."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "input_table = client.setup_input_table(endpoint=ENDPOINT)\noutput_table = client.setup_output_table(endpoint=ENDPOINT)\nmetadata_table = client.setup_metadata_table()\n\nprint(f\"Input table: {input_table}\")\nprint(f\"Output table: {output_table}\")\nprint(f\"Metadata table: {metadata_table}\")"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "**Every run:**\n\n**1. Append new rows to the input table.** In production this is your upstream pipeline's job."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "from pyspark.sql import functions as F\n\n(\n spark.createDataFrame(rows, schema=input_schema)\n .withColumn(\"updated_at\", F.current_timestamp())\n .write.format(\"delta\").mode(\"append\").saveAsTable(input_table)\n)\n\ndisplay(spark.table(input_table))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "**2. Call `batch_process`.** Re-running it picks up only rows appended since the last run."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "client.batch_process(\n context=context,\n input_table=input_table,\n output_table=output_table,\n metadata_table=metadata_table,\n process_new_records_only=True, # incremental; set False to reprocess every row\n batch_size=1600, # rows per API request\n)\n\ndisplay(spark.table(output_table))\ndisplay(spark.table(metadata_table))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## (Optional) Sending email addresses or phone numbers (UID2)\n\nSkip this section if you are sending device IDs or UID2s you have already resolved.\n\nPass a `uid2_config` when you create the client. It is the same call as Step 3 with one\nextra argument. Then set `id_type` to `Email`, `Phone`,\n`HashedEmail`, or `HashedPhone` on your rows. From there `push_data` and `batch_process` take them exactly like any other\nidentifier type, and nothing else about Step 6a or Step 6b changes.\n\nThe SDK resolves each identifier to a UID2 (or EUID) using your operator before the\nrequest leaves Databricks, so The Trade Desk never receives the raw email or phone\nnumber. The mapping comes back in the `uid2_resolutions` column."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": "from ttd_data.uid2 import IdentityScope, UID2Config\n\nfrom ttd_databricks_python.ttd_databricks import TtdDatabricksClient\n\nuid2_client = TtdDatabricksClient.from_params(\n api_token=API_TOKEN,\n uid2_config=UID2Config(\n base_url=\"<your-uid2-operator-url>\",\n api_key=\"<your-uid2-api-key>\",\n client_secret=\"<your-uid2-client-secret>\",\n identity_scope=IdentityScope.UID2, # use IdentityScope.EUID for European identities\n ),\n)\n\nuid2_data = [\n {\"id_type\": \"Email\", \"id_value\": \"user@example.com\"},\n {\"id_type\": \"HashedEmail\", \"id_value\": \"tMmiiTI7IaAcPpQPFQ65uMVCWH8av9jw4cwf/F5HVRQ=\"},\n]\n\nuid2_result_df = uid2_client.push_data(\n df=spark.createDataFrame(uid2_data, schema=input_schema),\n context=context,\n)\n\ndisplay(uid2_result_df.select(\"id_type\", \"success\", \"error_message\", \"uid2_resolutions\"))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Next steps\n\n- **Other use cases** \u2014 one notebook per use case:\n [First Party Data (1PD) Example Notebook](https://github.com/thetradedesk/ttd-databricks-python/blob/add-easy-start-examples-per-usecase-to-sdk-docs/example_notebook/First%20Party%20Data%20%281PD%29%20Example%20Notebook.ipynb),\n [Third Party Data (3PD) Example Notebook](https://github.com/thetradedesk/ttd-databricks-python/blob/add-easy-start-examples-per-usecase-to-sdk-docs/example_notebook/Third%20Party%20Data%20%283PD%29%20Example%20Notebook.ipynb),\n [Offline Conversion Data (CAPI) Example Notebook](https://github.com/thetradedesk/ttd-databricks-python/blob/add-easy-start-examples-per-usecase-to-sdk-docs/example_notebook/Offline%20Conversion%20Data%20%28CAPI%29%20Example%20Notebook.ipynb),\n [Deletion and Opt-Out (DSR) Example Notebook](https://github.com/thetradedesk/ttd-databricks-python/blob/add-easy-start-examples-per-usecase-to-sdk-docs/example_notebook/Deletion%20and%20Opt-Out%20%28DSR%29%20Example%20Notebook.ipynb).\n- **Full reference** \u2014 [README](https://github.com/thetradedesk/ttd-databricks-python/blob/add-easy-start-examples-per-usecase-to-sdk-docs/README.md) covers authentication, error handling,\n UID2 support, custom HTTP clients, and server URL overrides.\n- **Server URLs** \u2014 you do not need to configure one. Each endpoint already targets its\n own default server, and it can be overridden with a preferred server if you need one."
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Loading
Loading