Skip to content

fix: add resource limits - #418

Open
infernus01 wants to merge 1 commit into
tektoncd:mainfrom
infernus01:fix/limits
Open

fix: add resource limits#418
infernus01 wants to merge 1 commit into
tektoncd:mainfrom
infernus01:fix/limits

Conversation

@infernus01

@infernus01 infernus01 commented Sep 9, 2026

Copy link
Copy Markdown
Member

Relates to: #103

Summary

Updates the default resource limits in pruner manifests to align with actual measured usage from benchmark testing. The previous limits were too
conservative for the controller (512Mi) and could cause OOMKills under moderate load.

Benchmark Data

Conducted load testing on a cluster with varying numbers of completed PipelineRuns:

Scale Resident PRs Controller Heap Memory
Baseline 0 ~23 MB
Low 1,000 ~65 MB
Medium 6,000 ~148 MB
High 16,000 ~301 MB
Production 26,000-38,000 846-1126 MB

Key findings:

  • Memory usage is linear with the number of resident PipelineRuns in the informer cache
  • Growth coefficient: ~13.6 MB per 1,000 PipelineRuns
  • Webhook is stateless with minimal memory footprint (~60-70 MB regardless of scale)

Operator Integration

These upstream manifest values serve as the baseline. The tektoncd/operator applies these limits when deploying pruner, with override capability
via:

  apiVersion: operator.tekton.dev/v1alpha1
  kind: TektonConfig
  spec:
    tektonpruner:
      options:
        deployments:
        - name: tekton-pruner-controller
          spec:
            template:
              spec:
                containers:
                - name: controller
                  resources:
                    limits:
                      memory: 4Gi  # Can override if needed

A companion PR in tektoncd/operator would add a transformer to ensure these limits are applied

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
  • Has Tests included if any functionality added or changed
  • pre-commit Passed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

NONE

@tekton-robot
tekton-robot requested a review from jkhelil September 9, 2026 15:12
@tekton-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from infernus01 after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Sep 9, 2026
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.40%. Comparing base (6574e88) to head (b48ba7b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #418   +/-   ##
=======================================
  Coverage   50.40%   50.40%           
=======================================
  Files          19       19           
  Lines        2113     2113           
=======================================
  Hits         1065     1065           
  Misses        916      916           
  Partials      132      132           
Flag Coverage Δ
unit-tests 50.40% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anithapriyanatarajan anithapriyanatarajan added the kind/design Categorizes issue or PR as related to design. label Sep 10, 2026
Comment thread config/webhook.yaml Outdated
@anithapriyanatarajan

Copy link
Copy Markdown
Contributor

Thanks for the benchmark numbers @infernus01 🙇‍♀️

But I am not convinced the limits directly set in the deployment manifests. As per the table above, Memory scales linearly with resident PipelineRuns (~13.6MB/1000), so the "correct" limit is a function of the user's cluster, not something we can know upstream. Whatever number we pick will OOMKill somebody. Also, if the target namespace has a LimitRange whose max is below what we hardcode, the pods are rejected at admission. If we ship without limits, the admin's LimitRange defaults apply.

we could move the benchmark table into the docs as sizing guidance, so users know what to set for their own scale

@tektoncd/operator-maintainers - Request your guidance on this. Are we planning to enforce this config pattern across all tekton components individually or Shall we manage this with operator as default config values that could be overwritten.

@anithapriyanatarajan

Copy link
Copy Markdown
Contributor

/kind bug

@tekton-robot tekton-robot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 10, 2026
@infernus01 infernus01 changed the title fix: add resource limits and pin builder image fix: add resource limits Sep 10, 2026
@infernus01
infernus01 force-pushed the fix/limits branch 2 times, most recently from 0b91fac to fcfc274 Compare September 10, 2026 11:44
Signed-off-by: Shubham Bhardwaj <shubbhar@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug. kind/design Categorizes issue or PR as related to design. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants