Large documents cause NER task to fail due to Trigger.dev payload/output size limits

[dpancic]

Description

The NER plugin works correctly for small and medium-sized documents, but fails consistently when processing large documents.

The issue was observed after the workshop while testing the plugin with large spreadsheet-based sources, including the NMS dataset.

The underlying NLP processing appears to complete successfully, but the Trigger.dev task fails at the end of the workflow when Trigger.dev attempts to store or transfer the task output.

Observed behaviour

For large documents, the Trigger.dev run fails with an error similar to:

TriggerApiError: Failed to upload large payload to object store
    at Function.generate (...)
    at _doZodFetchWithRetries (...)

Small and medium-sized documents are processed successfully.

The issue was reproduced while testing on:

• Project: test (Project Baselayer)
• Documents: german-geo.txt and test.xml
• Large real-world dataset: NMS dataset

Suspected cause

The NER task appears to return the complete NLP result as the Trigger.dev task output.

For small documents this works without problems. For large documents, however, the resulting object becomes sufficiently large that Trigger.dev attempts to offload the payload/output to object storage.

The task then fails with:

Failed to upload large payload to object store

This appears to be related to payload/offloading size limitations in the self-hosted Trigger.dev installation.

There is a related upstream Trigger.dev issue:

triggerdotdev/trigger.dev#1343

The relevant limits appear to be hardcoded in the current Trigger.dev version used by our self-hosted installation, so increasing the limits through configuration does not currently appear to be an option.

Important observation

The failure does not appear to be caused by the NER processing itself.

The NLP task can process the document, but the workflow fails when the resulting data is returned through Trigger.dev.

In other words, the problematic flow appears to be:

Large document
    ↓
NER processing
    ↓
Large NLP result
    ↓
Trigger.dev task output
    ↓
Trigger.dev payload/offloading
    ↓
❌ Failed to upload large payload to object store

Suggested direction

The NER plugin should avoid returning very large NLP result objects through Trigger.dev.

Possible approaches include:

1. Persist the NLP result directly rather than returning the complete result as the Trigger.dev task output.
2. Process the document in chunks and persist/process the results incrementally.
3. Return only a small status/result object from the Trigger.dev task while storing the actual NLP output elsewhere.
4. If appropriate, redesign the workflow so that large intermediate results never have to pass through Trigger.dev's task output mechanism.

The preferred solution from my side would be to modify the NER plugin/workflow rather than modify the self-hosted Trigger.dev installation.

Alternative: modify Trigger.dev

Another possible solution would be to fork Trigger.dev and modify the relevant hardcoded payload/offloading limits in our self-hosted installation.

However, this would introduce a maintenance burden because we would then have to maintain a custom Trigger.dev fork and potentially reapply the changes when upgrading Trigger.dev.

For this reason, I would prefer to address the problem at the NER plugin level if possible.

Alternative: move away from self-hosted Trigger.dev

Another option would be to move away from the current self-hosted Trigger.dev setup and use a configuration/service where the relevant limits can be adjusted.

However, this would likely be the more expensive and operationally complex solution and does not address the underlying issue of passing very large NLP results through the task output mechanism.

Expected behaviour

The NER plugin should be able to process large documents without requiring the complete NLP result to be returned as a Trigger.dev task payload.

For large documents, the workflow should instead:

• process the document successfully;
• persist or otherwise handle the NLP result without exceeding Trigger.dev payload limits;
• return only a small task result/status through Trigger.dev;
• complete successfully regardless of the size of the NLP result.

Additional context

This was discovered during post-workshop testing.

The NMS dataset is currently too large for the existing workflow because of the Trigger.dev payload/offloading limitation.

The issue is reproducible with large inputs and does not occur with small documents, which strongly suggests that the size of the task output is the relevant factor.