crowdstrike: fix FDR deduplication for CSPM findings

[navnit-elastic]

Proposed commit message

crowdstrike: fix FDR deduplication for CSPM findings

Classify aidmaster/userinfo/data from aws.s3.object.key when log.file.path
is absent, run CSPM pipelines before fingerprinting, and include rule.id
plus resource id so distinct Cloud Security findings stay unique.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes CrowdStrike FDR: enable_deduplication fingerprint drops distinct events and fails to dedup re-ingested data (CSPM/IOA) #20064
• Closes CrowdStrike FDR: classify aidmaster/userinfo/data from aws.s3.object.key, not just log.file.path #20113

Screenshots

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[ShourieG]

@vera-review-bot review

[vera-review-bot]

Severity: 🟡 Medium confidence: high path: packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml:626

Adding bare rule.id to the fingerprint also re-keys EPP, FIM and Data Protection events, not just CSPM findings as the README states; scope it with a CSPM-only _temp.cspm_rule_id instead.

Details

The fingerprint processor has no CSPM gate, so rule.id is hashed for every document that happens to carry it. Three non-CSPM sub-pipelines populate rule.id and are all invoked at default.yml:355-371, well before this fingerprint at line 616:

• epp_detection_summary.yml:230 renames crowdstrike.DetectId to rule.id
• fim_rule_matched.yml:208 sets rule.id from crowdstrike.PolicyRuleID
• data_protection_detection_summary.yml:225 sets rule.id from crowdstrike.Policy.ID

So the _id computed for EPP detection summary, FIM rule matched, and Data Protection detection summary events changes in 4.4.2. Any of those documents already indexed under 4.4.1 and re-delivered by FDR after the upgrade will be written again under a new _id rather than overwriting, which is the exact duplication the Enable Data Deduplication option exists to prevent.

This also contradicts the documentation added in the same PR. Both _dev/build/docs/README.md and docs/README.md state the addition applies "For Cloud Security (CSPM) findings", and the changelog entry is scoped to "Cloud Security findings". The code applies it unconditionally.

Note the companion script at line 600 has the same mis-scoped guard: if: ctx.rule?.id != null matches EPP/FIM/Data Protection events too. It happens to be harmless today only because none of those pipelines set crowdstrike.ResourceId, crowdstrike.resource.resourceId, crowdstrike.crn, or event.id before the fingerprint — so _temp.cspm_resource_id stays unset for them. The guard is still relying on that coincidence rather than on the event actually being CSPM.

The rest of the reorder checks out: categorize.yml keys only off event_simpleName and contains no CloudSecurity entries, its script is a no-op when the key is absent, and none of the CSPM fixtures carry a mapped event_simpleName, so moving the CSPM pipelines ahead of categorize does not clobber the CSPM event.kind/category/type. The hit_count: 138 arithmetic also holds (135 + 5 + 1 = 141 lines, minus the pre-existing pair plus the IOM and IOA re-ingest pairs).

Recommendation:

Mark CSPM events explicitly and fingerprint a CSPM-scoped rule id, so non-CSPM _id values are unchanged across the upgrade.

Add the marker just before the two CSPM pipeline calls (all three discriminator fields survive those pipelines):

  - set:
      tag: set__temp_is_cspm_4f7ab210
      if: >-
        (ctx.crowdstrike?.disposition != null && ctx.crowdstrike.disposition.equalsIgnoreCase('Failed')) ||
        (ctx.crowdstrike?.event_simpleName != null && ctx.crowdstrike.event_simpleName.equalsIgnoreCase('CloudSecurityIOMEvaluation')) ||
        (ctx.crowdstrike?.vertex_type != null && ctx.crowdstrike.vertex_type.equalsIgnoreCase('ioa'))
      field: _temp.is_cspm
      value: true

Gate the resolver on it and capture the rule id there:

  - script:
      tag: script_cspm_resource_id_for_fingerprint_7c3e9a12
      if: ctx._temp?.is_cspm == true && ctx.rule?.id != null
      source: |-
        ctx._temp = ctx._temp ?: [:];
        ctx._temp.cspm_rule_id = ctx.rule.id.toString();
        if (ctx.crowdstrike?.ResourceId != null && ctx.crowdstrike.ResourceId != '') {
          ctx._temp.cspm_resource_id = ctx.crowdstrike.ResourceId.toString();
        } else if (ctx.crowdstrike?.resource?.resourceId != null && ctx.crowdstrike.resource.resourceId != '') {
          ctx._temp.cspm_resource_id = ctx.crowdstrike.resource.resourceId.toString();
        } else if (ctx.crowdstrike?.crn != null && ctx.crowdstrike.crn != '') {
          ctx._temp.cspm_resource_id = ctx.crowdstrike.crn.toString();
        } else if (ctx.event?.id != null && ctx.event.id != '') {
          ctx._temp.cspm_resource_id = ctx.event.id.toString();
        }

Then swap the fingerprint field:

      fields:
        - '@​timestamp'
        - crowdstrike.id
        - crowdstrike.aid
        - crowdstrike.cid
        - _temp.type
        - _temp.cspm_rule_id
        - _temp.cspm_resource_id

If re-keying the non-CSPM streams is intentional, keep rule.id but drop the "For Cloud Security (CSPM) findings" qualifier from both READMEs and call out the one-time _id change in the changelog.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[navnit-elastic]

Updated README and changelog to call out one-time _id change.

[vera-review-bot]

Review summary

Issues found across the latest commits c9f2ec7…7221dfe (26 commits) — 1 medium

• 🟡 Adding bare rule.id to the fingerprint also re-keys EPP, FIM and Data Protection events, not just CSPM findings as the README states (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 68bc3a2

History

• 💚 Build #46657 succeeded 7221dfe
• 💚 Build #46579 succeeded b8c8ee0
• 💚 Build #46537 succeeded 9e2f133

cc @navnit-elastic