Pulls data from cloud and SaaS APIs, databases and files into a DFE deployment.
- A REST provider is a YAML profile, with no Rust: the base URL, auth modes, endpoints, decoder and pager, checked at load with the line of any unknown key.
- Database sources over ODBC, the ClickHouse HTTP interface, MongoDB and SAP HANA, dumped whole or tailed, by key tuple on SQL and by change stream or
_idon MongoDB. - Any tool run in a Docker or podman container as a source, its stdout JSON lines or HTTP posts collected, and Vector's native gRPC protocol received.
- A cursor or checkpoint written once the transport has taken the batch, never before, so a failed tick fetches again rather than losing records. A dump travels as one snapshot, framed
begin, rows andend. - A per-source CEL filter, re-read every tick, and a memory brake that pauses polling a provider when the service is under memory pressure, for at most
self_regulation.max_hold_secsat a time. - Secrets from a secrets manager, an env var or a literal, config reload on SIGHUP or file watch, and a deployment contract that generates its Dockerfile, its Compose file and the released chart.
Every record goes to the DFE pipeline over Kafka or gRPC, the same way dfe-receiver's do.
The source framework polls APIs, databases and files on a schedule. The extractors take what external tools hand over. Any combination runs at once, and every record leaves through one emitter.
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart TB
accTitle: L3 Components -- dfe-fetcher
accDescr: dfe-engine writes the fetcher's config. APIs, databases and files feed the source framework, and external tools feed the extractors. Inside dfe-fetcher both hand records to the emitter, which sends them to Kafka topics or a DFE service over gRPC. A record the transport refuses for good goes to the dead letters.
subgraph CTRL["Control"]
direction LR
engine("<b>dfe-engine</b><br/><span>writes the config</span>")
end
subgraph IN["Inputs"]
direction LR
providers("<b>APIs, databases, files</b><br/><span>polled by the framework</span>")
tools("<b>External tools</b><br/><span>run or heard by extractors</span>")
end
subgraph FETCH["dfe-fetcher"]
direction LR
framework("<b>Source framework</b><br/><span>REST, DB and file readers</span>")
extractors("<b>Extractors</b><br/><span>container, ingest, Vector</span>")
emitter("<b>Emitter</b><br/><span>send, retry, dead-letter</span>")
framework --> emitter
extractors --> emitter
end
subgraph OUT["Destinations"]
direction LR
kafka[("<b>Kafka</b><br/><span>*_land topics</span>")]
grpc("<b>DFE service</b><br/><span>over gRPC</span>")
end
subgraph DL["Dead letters"]
direction LR
dlq[("<b>DLQ</b><br/><span>Kafka topic or file spool</span>")]
end
CTRL -.->|config| FETCH
IN --> FETCH
FETCH --> OUT
FETCH -.->|refused for good| DL
class providers,tools ext
class engine control
class framework,extractors,emitter part
class kafka,dlq bus
class grpc load
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
style CTRL fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style IN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style FETCH fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style OUT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style DL fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
Figure 1. L3 Components -- dfe-fetcher. Dashed boxes are outside DFE, and dotted arrows are config and dead letters. Up: top of dfe-fetcher. Down: dfe-fetcher architecture.
What each part does, what the fetcher depends on and why it is built this way are in docs/architecture.md.
On Linux the build needs protoc, a C toolchain, pkg-config and librdkafka. scalo compiles its gRPC .proto files, aws-lc-sys compiles C, and rdkafka-sys links librdkafka through pkg-config. It rejects a librdkafka older than the version after the + in its Cargo.lock entry, and a Debian or Ubuntu package can be older than that, so there build librdkafka from source first.
sudo apt-get install -y protobuf-compiler build-essential pkg-config
cargo build --release
./target/release/dfe-fetcher --config config.example.yaml config-check
./target/release/dfe-fetcher --config config.example.yaml--config goes before the subcommand. config.example.yaml enables no source, so the fetcher starts idle and logs no work configured until a source, an extractor or the ingest listener is switched on. Metrics and health answer on port 9090 (/metrics, /livez, /readyz), and /readyz answers {"status":"ready"} when idle. The ingest HTTP endpoint on 8080 and the Vector gRPC receiver on 6000 are off until a deployment turns them on.
A default build has the REST sources and the file dump. The database engines and the file tailer are cargo features of crates/fetcher/Cargo.toml (db-odbc, db-clickhouse, db-mongodb, db-hana, file-tail), and a config naming an engine the binary lacks is refused at load with the feature to build. The release image is built with all five and jemalloc (build.rust.features in .hyperi-ci.yaml).
config.example.yaml is the annotated reference, docs/config-schema.yaml the generated list of every key, and docs/reference/profile-grammar.md the profile grammar field by field. Settings load through a cascade: CLI flags, then DFE_FETCHER_* environment variables, then the config file, then defaults. The file is the one --config names, and there is no DFE_FETCHER_CONFIG variable. Without --config, scalo's search for settings.yaml and defaults.yaml stands in, with .env read from the working directory, as docs/reference/configuration.md sets out. Each provider has its typed block (sources.aws, sources.okta, and so on), and sources.rest.<id> takes a shipped or inline profile. A second account of a provider, or an API with no typed block, is a sources.rest.<id> instance. The id is the connection id: the cursor key, the metric and log label, and the prefix of _source_fetcher on every record.
Credential and identity fields take a spec: vault:<mount>/<path>:<key> (or its bao: and openbao: spellings), env:NAME, file:<path>, or a literal. Identity fields (tenant, account, client id and the like) resolve at startup, and one that cannot be resolved stops the process with a configuration error. A reload that changes a source resolves its specs again. A provider secret resolves on first use and is held until its source is rebuilt or the process restarts, so a secret rotated behind an unchanged spec takes a restart. An aws: spec is refused at load, because the fetcher is not built with the AWS Secrets Manager resolver. Credentials in Kubernetes shows how a deployment supplies them.
Each REST source is a shipped profile under crates/fetcher/profiles/, and databases and files are separate source kinds (sources.db, sources.file). A profile's maturity is repeated in docs/capability-catalog.yaml, and the fetcher warns at startup for any enabled source that is not stable. The database and file sources are alpha, code-complete but not production-validated, so every sources.db and sources.file instance logs that warning. Promoting them is tracked in #323. docs/cloud-setup/README.md indexes the setup guides.
| Source | Block | What it pulls |
|---|---|---|
| AWS | sources.aws |
CloudTrail, GuardDuty, Security Hub, Config, CloudWatch Logs and Metrics, Inspector v2, Health |
| Azure | sources.azure |
Activity Log, Defender alerts, Sentinel incidents, the Entra ID audit feeds, Log Analytics queries |
| Microsoft 365 | sources.m365 |
Management Activity feeds (audit, DLP, Exchange), Graph security alerts |
| Google Cloud | sources.gcp |
Cloud Audit Logs, VPC flow and DNS query logs, any Cloud Logging filter, Security Command Center findings |
| Google Cloud Pub/Sub, Google Workspace | sources.gcp_pubsub, sources.google_workspace |
Pull subscriptions, Reports API activity |
| Okta, Cisco Duo, CrowdStrike, Cloudflare | sources.okta, sources.duo, sources.crowdstrike, sources.cloudflare |
System Log, authentication logs, Falcon alerts, account audit logs |
| Datadog, runZero, SAP S/4HANA | sources.rest with the shipped profile |
Audit trail and security signals, asset inventory snapshots, an OData V2 entity set read by timestamp |
| Bitwarden, 1Password, Slack, GitHub, Salesforce | sources.bitwarden, sources.onepassword, sources.slack, sources.github, sources.salesforce |
Each product's audit and event logs |
| Object store | sources.object_store |
S3 and S3-compatible bucket prefixes, each object read at least once |
| PyPI, crates.io, Go modules | sources.pypi, sources.crates_io, sources.go_modules |
Package metadata for supply-chain monitoring |
| Databases (alpha) | sources.db |
Any ODBC-reachable database, ClickHouse, MongoDB or SAP HANA, dumped whole or tailed by key |
| Files (alpha) | sources.file |
Directories of NDJSON, JSON or CSV dumps, or log files tailed |
An API starts as a profile, shipped under crates/fetcher/profiles/ or inline under sources.rest.<id>.profile. Rust is added for what the grammar cannot express and nothing else, as a variant on the matching axis in crates/rest.
- docs/architecture.md: the parts, how data flows through them and why it is built this way.
- docs/internals.md: the source framework, its readers and the tick.
- docs/delivery.md and docs/extractors.md: the emitter, and the container and Vector extractors.
- docs/reference/: the profile grammar, the snapshot envelope, the database drivers and the configuration cascade.
- docs/cloud-setup/: one setup guide per provider.
- docs/testing.md: the test tiers and the live-API suites.
The suite's published docs are at https://dfe.docs.hyperi.io, and this repository's pages at https://dfe.docs.hyperi.io/dfe-fetcher/.
How to work in this repo, the commands that prove a change and what tends to bite are in CONTRIBUTING.md. A question that is not a bug goes in an issue.
Open bugs: this repo | whole DFE suite
Licensed BUSL-1.1, see LICENSE.