Skip to content

Repository files navigation

listbot

listbot logo

listbot builds compact Logstash-style YAML translation maps for two common security enrichment jobs:

  • cve.yaml: Emerging Threats SID to CVE/CAN lookup data
  • iprep.yaml: IPv4 reputation data from public OSINT feeds

Each map is also written as a bzip2-compressed copy, so the generated map artifact set is:

  • cve.yaml
  • cve.yaml.bz2
  • iprep.yaml
  • iprep.yaml.bz2
  • NOTICE

When logging is enabled, run also writes run.log; failed logged runs are also appended to error.log.

The project used to be shell-heavy. It now runs as a Python CLI managed by Astral uv, with native parsers for text, CSV-like files, gzip, zip, CIDR ranges, and DShield-style start/end ranges.

Usage

Create the Python 3.14 virtual environment and install dependencies:

uv sync --python 3.14

Option 1: Run through uv

Use this when you do not want to activate the virtual environment manually. uv run starts the command inside the project environment.

Generate both maps in the current directory:

uv run listbot run --output-dir .

Generate one map at a time:

uv run listbot gen-cve --config config.toml --output-dir .
uv run listbot gen-iprep --config config.toml --output-dir .

Run any command with a TOML config:

uv run listbot run --config config.toml --output-dir .

Option 2: Activate .venv

Use this when you prefer the shorter listbot ... command during an interactive session or in a shell script that already activates the environment.

Activate the virtual environment:

source .venv/bin/activate

Generate both maps in the current directory:

listbot run --output-dir .

Generate one map at a time:

listbot gen-cve --config config.toml --output-dir .
listbot gen-iprep --config config.toml --output-dir .

Run any command with a TOML config:

listbot run --config config.toml --output-dir .

You can leave the environment with:

deactivate

Option 3: Docker / Docker Compose

Build a local image:

docker build -t listbot:local .

Run the default combined build. The container writes artifacts to ./output and keeps the download cache in ./.cache/listbot:

mkdir -p output .cache/listbot
docker run --rm \
  -v "$PWD/config.toml:/config/listbot.toml:ro" \
  -v "$PWD/output:/data" \
  -v "$PWD/.cache/listbot:/cache/listbot" \
  listbot:local

Run a single command through the same image:

docker run --rm \
  -v "$PWD/config.toml:/config/listbot.toml:ro" \
  -v "$PWD/output:/data" \
  -v "$PWD/.cache/listbot:/cache/listbot" \
  listbot:local gen-iprep --config /config/listbot.toml --output-dir /data --cache-dir /cache/listbot

With Docker Compose:

mkdir -p output .cache/listbot
docker compose run --rm listbot

To remove the one-off container and then clean up the Compose project network and any named Compose volumes after a successful run:

docker compose run --rm listbot && docker compose down -v

In the provided docker-compose.yml, output and .cache/listbot are bind-mounted host directories, so down -v does not delete those directories or their generated files. It only removes Docker-managed resources created by Compose.

The image runs as the numeric user 1000:1000 by default so bind-mounted output files are usable on typical Linux developer machines. Build with explicit IDs if your host user differs:

LISTBOT_UID="$(id -u)" LISTBOT_GID="$(id -g)" docker compose build

/config, /data, and /cache are also owned by group 0 with group write access, so the image works when a platform assigns an arbitrary UID in group 0 (for example OpenShift). At runtime listbot only writes to the output, cache, and optional bz2_dir directories, so a read-only root filesystem works when those paths are volumes.

Option 4: Kubernetes

The image sets a numeric user, so runAsNonRoot: true can be verified. Create a ConfigMap from the complete config.toml (a loaded config must list every IPREP feed):

kubectl create configmap listbot-config --from-file=listbot.toml=config.toml

Example CronJob using the image's default command (run with /data and /cache/listbot):

apiVersion: batch/v1
kind: CronJob
metadata:
  name: listbot
spec:
  schedule: "0 */6 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
            runAsGroup: 1000
            fsGroup: 1000
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: listbot
              image: ghcr.io/telekom-security/listbot:latest
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
              volumeMounts:
                - name: config
                  mountPath: /config
                  readOnly: true
                - name: data
                  mountPath: /data
                - name: cache
                  mountPath: /cache
          volumes:
            - name: config
              configMap:
                name: listbot-config
            - name: data
              persistentVolumeClaim:
                claimName: listbot-data
            - name: cache
              emptyDir: {}

Notes:

  • Another runAsUser works as well; fsGroup makes the mounted volumes writable for it.
  • Use a PersistentVolumeClaim instead of emptyDir for /cache if the download cache should survive between runs.
  • To publish the .bz2 files to a second volume, mount it and pass the full command via args, for example ["run", "--config", "/config/listbot.toml", "--output-dir", "/data", "--cache-dir", "/cache/listbot", "--bz2-dir", "/public"].

Commands

run

Builds cve.yaml, iprep.yaml, their compressed .bz2 copies, and NOTICE in one command. With the shipped defaults it also performs threshold checks and writes logs. This is the normal command for local use and scheduled generation.

listbot run --output-dir /var/lib/listbot

Useful options:

  • --config config.toml: load command defaults from an explicit TOML config file
  • --bz2-dir PATH: also copy the generated .bz2 files to this directory
  • --no-bz2-dir: disable bz2_dir configured in --config
  • --file-mode 0644: octal permissions for generated YAML, .bz2, and NOTICE files
  • --workers 8: number of concurrent IP feed downloads
  • --timeout 30: HTTP timeout in seconds
  • --suricata-version 8.0.0: Emerging Threats Open ruleset version
  • --cve-url URL: explicit sid-msg.map override
  • --cache / --no-cache: enable or disable the local raw-download cache
  • --cache-dir PATH: cache directory for feed payloads and metadata
  • --cache-max-age 6h: refresh cache entries after this age; supports m, h, d
  • --feed-usage-profile all|commercial|non-commercial: filter IPREP feeds by usage profile
  • --check-thresholds: enable minimum count checks
  • --min-cve 5000: minimum CVE mappings for a checked run
  • --min-iprep 500000: minimum IP reputation mappings for a checked run
  • --log-dir PATH: write run.log or error.log and enable checks
  • --no-log: disable logging configured in --config

Scheduling with cron

cron starts jobs with a minimal environment (PATH=/usr/bin:/bin), and a script run by cron does not read shell profiles such as .bashrc or .profile. If uv was installed with the official installer, it lives in ~/.local/bin and is not found. cron discards the resulting uv: command not found unless a mail server is configured, so the run fails silently and no listbot logs are written.

Use a small wrapper script that sets up the environment:

#!/bin/bash
set -euo pipefail
source "$HOME/.local/bin/env"
cd /opt/listbot
uv run listbot run >> /var/log/listbot-cron.log 2>&1
  • $HOME/.local/bin/env is created by the uv installer and adds uv to PATH. cron sets HOME for the job user.
  • If uv was installed differently (package manager, pipx), use its absolute path from command -v uv instead of the source line.
  • set -euo pipefail stops the script if cd or source fails.
  • The redirect keeps errors that happen before listbot writes its own logs.

Make the script executable and schedule it, for example in /etc/crontab:

40 10 * * *  root  /opt/listbot/listbot-cron.sh

To test the script with an environment similar to cron's:

env -i HOME=/root PATH=/usr/bin:/bin /opt/listbot/listbot-cron.sh; echo "exit=$?"

gen-cve

Builds only the CVE map:

listbot gen-cve --config config.toml --output-dir .

The command uses the same default cache as run. Use --no-cache for a live download or override the cache location with --cache-dir. If ./config.toml exists, it is loaded automatically; use --config to select another file. --feed-usage-profile is accepted for config consistency, but CVE generation does not use IP reputation feeds.

The parser keeps the legacy scalar YAML format. If one SID contains multiple CVE/CAN references, the first reference in source order wins, so values stay as single strings.

gen-iprep

Builds only the IP reputation map:

listbot gen-iprep --config config.toml --output-dir .

The command uses the same default cache as run. Use --no-cache for live feed downloads or override the cache age with --cache-max-age. If ./config.toml exists, it is loaded automatically; use --config to select another file. Use --feed-usage-profile to restrict active feeds to a commercial, non-commercial, or all-feeds profile.

The parser validates IPv4 addresses with Python's ipaddress module and only writes globally routable IPv4 addresses. This avoids false matches from comments or version-like strings in feed metadata.

By default, run uses threshold checks and logging because both the built-in defaults and the shipped config.toml enable them. A checked run passes when each generated count reaches at least its configured minimum. Default thresholds:

  • CVE mappings: at least 5000
  • IP reputation mappings: at least 500000

Override them with --min-cve and --min-iprep. For artifact-only runs, set [thresholds].enabled = false and [logging].enabled = false in a config file. --no-log disables configured logging, but threshold checks can still be active.

listbot does not publish, push, or notify by itself. External jobs can watch or copy --output-dir when they need to move artifacts into another system.

Config

All commands load ./config.toml automatically when it exists. You can point to a different TOML file with --config. No legacy config names are searched. CLI options override config values.

Excerpt:

[run]
output_dir = "."
# bz2_dir = "/srv/listbot/public"
file_mode = "0644"
workers = 8
timeout = 30.0
suricata_version = "8.0.0"
# cve_url = "https://example.test/sid-msg.map"

[cache]
enabled = true
cache_dir = ".cache/listbot"
cache_max_age = "6h"

[feeds]
# all, commercial, or non-commercial.
# all is an operational choice, not a legal clearance.
usage_profile = "all"

[feeds.iprep]
# The real config.toml lists every IPREP feed explicitly.
# A loaded config must contain every known feed ID with a boolean value.
feodotracker = true
firehol_blocklist_de_ssh = true
firehol_stopforumspam_30d = true
firehol_anonymous = true

[thresholds]
enabled = true
min_cve = 5000
min_iprep = 500000

[logging]
enabled = true
# dir = "/var/log/listbot"

Caching is enabled by default for run, gen-cve, and gen-iprep. A valid cache entry avoids a network request and reuses the raw upstream payload. When a cache entry is older than cache_max_age, listbot deletes it before attempting a fresh download. There is no stale fallback: if the refresh fails, the feed is handled like any other failed download. Cache durations must be positive values with m, h, or d suffixes, for example 30m, 6h, or 2d.

bz2_dir is an optional second target for the compressed maps. When it is set, run, gen-cve, and gen-iprep copy their .bz2 files there in addition to output_dir. YAML, NOTICE, and logs stay in output_dir. Each file is written to a temporary name first and then renamed, so readers never see a partial file. If run fails its threshold checks, bz2_dir is not updated and keeps the last good files; the run status and logs say so. --bz2-dir PATH enables or overrides the directory, and --no-bz2-dir disables a configured one.

file_mode sets the permissions of the generated YAML, .bz2, and NOTICE files in output_dir and bz2_dir (default 0644), independent of the process umask. Use 3 or 4 octal digits, for example 0640; setuid, setgid, and sticky bits are not accepted. Logs and cache files keep their default permissions. If the permissions cannot be set, for example on a network mount that does not allow chmod, the file is still written and listbot prints a warning (also recorded in the logs) without failing the run.

Logging also enables threshold checks. If logging.dir is omitted, logs are written to output_dir. run.log is an additive history of all logged runs; failed runs are also appended to error.log. Threshold failures are shown as a dedicated Rich panel and logged with actual count, configured minimum, and missing count.

The repository includes a commented config.toml with all configurable settings, so local scheduled runs can be adjusted without keeping the README open next to the terminal.

The [feeds.iprep] section is intentionally strict. When a config file is loaded, every known IP reputation feed must be present as true or false. Unknown feed IDs and missing feed IDs are rejected so feed changes are explicit. Without a config file, listbot uses the built-in default feed set.

The global [feeds].usage_profile filter is applied after the individual feed booleans:

  • commercial: only feeds classified as explicitly commercial/permissive
  • non-commercial: commercial/permissive feeds plus explicitly non-commercial feeds
  • all: every enabled feed, including unclear or restricted upstream terms

This profile is a conservative technical policy, not legal advice. The generated NOTICE remains the source record for upstream terms, retrieval timestamps, and derived-data attribution.

Terminal Output

listbot uses Rich for colored terminal output:

  • progress bars for download, parse, and write phases
  • colored success/error summaries
  • output file locations
  • feed success/failure counts
  • a complete feed contribution table sorted by added indicators, including failed feeds

The generated YAML and .bz2 files are unchanged by the terminal UI.

Output Format

The YAML files intentionally use the same scalar mapping style as the legacy generator:

"2000016": "CAN-2004-0120"
"1.2.3.4": "attack source"

The files are sorted by key and deduplicated. For duplicate IPs across feeds, the first matching feed in src/listbot/feeds.py wins.

Every map build also writes a NOTICE file next to the generated maps. It records the feed source, configured and final URL, retrieval timestamp, license/terms note, extracted/added row counts, and states that the YAML maps are derived data from third-party OSINT feeds.

Feed Policy

The default feed set is public OSINT only:

  • no API keys
  • no paid feeds
  • no default feeds that require authentication
  • dead or deprecated historical feeds removed

The IP reputation feeds are configured in src/listbot/feeds.py. A feed entry contains the feed name, URL, output tag, parser style, and expansion limits for CIDR/range data. Feed order matters: the first feed that contributes an IP wins the tag for that IP.

config.toml controls which default IP reputation feeds are active. The [feeds.iprep] section is explicit by design: every feed ID must be present, so operators can review new sources before running scheduled builds.

[feeds].usage_profile adds a second, global filter for environments with different usage requirements. The filter is intentionally defensive: feeds without explicit licensing or terms are treated as unknown and are only used with all.

Tags are short analyst-facing labels intended for Kibana dashboards. Specific tags are preferred when a feed states a concrete behavior such as C2, scanning, open proxies, service abuse, or form spam. attack source is used for generic attack-focused feeds where the upstream data does not support a narrower claim. anonymizer is enrichment context, not a malware or attack assertion.

Feed data is not relicensed by this project. The license/terms column records what upstream currently publishes or what is visible in the feed itself. If no explicit feed license was found, the table says so instead of guessing.

Usage classes used by the profile filter:

  • unrestricted: used by commercial, non-commercial, and all
  • non_commercial: used by non-commercial and all
  • restricted: used only by all
  • unknown: used only by all

Initial conservative classification:

  • unrestricted: feodotracker, maltrail_mass_scanner, ipsum_level3
  • non_commercial: firehol_dshield, dshield, bitwire_outbound, turris
  • restricted: binary_defense_banlist
  • unknown: all other IP reputation feeds
Feed ID Source Tag Purpose and notes Upstream license / terms
feodotracker abuse.ch Feodo Tracker botnet C2 Botnet C2 IP blocklist. abuse.ch Terms of Use; Feodo documents vendor use for commercial and non-commercial purposes.
threatfox_ip_port_recent abuse.ch ThreatFox malware infra Recent ip:port malware infrastructure and botnet C2 IOCs. abuse.ch Terms of Use; no separate SPDX-style data license stated.
neo23x0_c2 Neo23x0 signature-base C2 server C2 indicators from the public signature-base IOC set. Detection Rule License 1.1.
cybercrime_tracker CyberCrime Tracker malware infra Cybercrime and malware infrastructure indicators. No explicit feed license found.
firehol_webclient FireHOL webclient malware infra Malware infrastructure destinations a web client should not contact. FireHOL aggregate/mirror; upstream list licenses may vary. FireHOL asks users to check each source site.
et_compromised Emerging Threats compromised IPs compromised Compromised hosts from Emerging Threats. Emerging Threats/Proofpoint terms; the blockrules feed has no explicit license header. ET Open rules are GPLv2/BSD by SID range.
maltrail_mass_scanner Maltrail mass scanners mass scanner Static mass-scanner indicators. MIT.
tor_bulk_exit Tor bulk exit list tor exit Current Tor exits. Tor Project exit-list service; no explicit data license found on the feed endpoint.
tor_exit_addresses Tor exit addresses tor exit TorDNSEL-style exit address data. Tor Project exit-list service; no explicit data license found on the feed endpoint.
firehol_dm_tor FireHOL dm_tor tor exit FireHOL-hosted Tor exit snapshot. FireHOL mirror; upstream license/terms depend on the original Tor-related source.
firehol_proxylists FireHOL proxylists open proxy Open proxy list snapshot. FireHOL mirror; upstream proxy-list licenses may vary or be unstated.
firehol_proxyrss FireHOL proxyrss open proxy Open proxy RSS snapshot. FireHOL mirror; upstream proxy-list licenses may vary or be unstated.
firehol_proxyspy FireHOL proxyspy open proxy ProxySpy-derived open proxy list. FireHOL mirror; upstream proxy-list licenses may vary or be unstated.
firehol_web_proxies FireHOL RI web proxies open proxy Open web proxy indicators. FireHOL mirror; upstream proxy-list licenses may vary or be unstated.
firehol_socks_proxy FireHOL socks proxy open proxy Open SOCKS proxy indicators. FireHOL mirror; upstream proxy-list licenses may vary or be unstated.
firehol_sslproxies FireHOL SSL proxies open proxy Open SSL proxy indicators. FireHOL mirror; upstream proxy-list licenses may vary or be unstated.
spys spys.me proxy list open proxy Open proxy list. No explicit feed license found.
firehol_botscout FireHOL BotScout form spammer BotScout spammer snapshot. FireHOL mirror; upstream BotScout terms apply, explicit feed license not stated in the mirrored file.
sblam Sblam blacklist form spammer HTTP/form spam sources. No explicit feed license found; feed labels the data as HTTP/form spam sources.
firehol_bitcoin FireHOL bitcoin nodes bitcoin node Bitcoin node enrichment data, not necessarily malicious. FireHOL mirror; upstream source license/terms may vary.
firehol_cruzit FireHOL CruzIT web attacker Web attack sources. FireHOL mirror; upstream source license/terms may vary or be unstated.
firehol_mwdomainlist FireHOL malwaredomainlist malware host Malware host IP indicators. FireHOL mirror; upstream source license/terms may vary or be unstated.
firehol_dshield FireHOL DShield 30d scan source DShield scan-source netset snapshot. FireHOL mirror of DShield-derived data; DShield publishes CC BY-NC-SA 2.5 terms on its direct feed.
firehol_darklist FireHOL darklist.de attack source Generic blacklisted attack sources from darklist.de. FireHOL mirror; no explicit darklist.de feed license found in the mirrored file.
binary_defense_banlist Binary Defense ATIF bad reputation Public bad-reputation banlist. Public use only; commercial resale or fee-based services are prohibited in the feed header.
firehol_blocklist_de_apache FireHOL blocklist.de apache web attacker blocklist.de Apache and web-service abuse sources. FireHOL mirror of blocklist.de apache feed; blocklist.de export terms apply.
firehol_blocklist_de_bots FireHOL blocklist.de bots bot activity blocklist.de bot activity sources. FireHOL mirror of blocklist.de bots feed; blocklist.de export terms apply.
firehol_blocklist_de_ftp FireHOL blocklist.de ftp ftp abuse blocklist.de FTP abuse sources. FireHOL mirror of blocklist.de FTP feed; blocklist.de export terms apply.
firehol_blocklist_de_imap FireHOL blocklist.de imap mail abuse blocklist.de IMAP abuse sources. FireHOL mirror of blocklist.de IMAP feed; blocklist.de export terms apply.
firehol_blocklist_de_mail FireHOL blocklist.de mail mail abuse blocklist.de mail abuse sources. FireHOL mirror of blocklist.de mail feed; blocklist.de export terms apply.
firehol_blocklist_de_sip FireHOL blocklist.de sip sip abuse blocklist.de SIP/VoIP abuse sources. FireHOL mirror of blocklist.de SIP feed; blocklist.de export terms apply.
firehol_blocklist_de_ssh FireHOL blocklist.de ssh ssh abuse blocklist.de SSH abuse sources. FireHOL mirror of blocklist.de SSH feed; blocklist.de export terms apply.
firehol_blocklist_de FireHOL blocklist.de all service abuse Reported service abuse sources. FireHOL mirror of blocklist.de all feed; blocklist.de export terms apply.
cinsscore CINS Score bad reputation Bad reputation badguys list. Public CINS Army list; no explicit feed license found.
greensnow GreenSnow attack source Generic attack-source blacklist. Public blacklist; no explicit feed license found.
rutgers Rutgers DROP attackers attack source Generic DROP attack sources. No explicit feed license found.
dshield DShield block list scan source Top scan-source subnets, parsed from start/end ranges. Creative Commons BY-NC-SA 2.5 per feed header.
ipsum_level3 IPsum level 3 bad reputation IPs seen on at least three source lists. Unlicense/public-domain dedication.
firehol_php_commenters_30d FireHOL Project Honey Pot commenters 30d comment spam Project Honey Pot comment spam sources seen in the last 30 days. FireHOL mirror; Project Honey Pot terms may apply; explicit feed license not stated in the mirrored file.
firehol_php_dictionary_30d FireHOL Project Honey Pot dictionary 30d web scanner Project Honey Pot directory and dictionary attack sources seen in the last 30 days. FireHOL mirror; Project Honey Pot terms may apply; explicit feed license not stated in the mirrored file.
firehol_php_harvesters_30d FireHOL Project Honey Pot harvesters 30d web harvester Project Honey Pot web harvester sources seen in the last 30 days. FireHOL mirror; Project Honey Pot terms may apply; explicit feed license not stated in the mirrored file.
firehol_php_spammers_30d FireHOL Project Honey Pot spammers 30d form spammer Project Honey Pot form spammer sources seen in the last 30 days. FireHOL mirror; Project Honey Pot terms may apply; explicit feed license not stated in the mirrored file.
firehol_stopforumspam_30d FireHOL StopForumSpam 30d forum spammer Forum spammer IPs seen in the last 30 days. FireHOL mirror; StopForumSpam terms may apply; explicit feed license not stated in the mirrored file.
firehol_stopforumspam_toxic FireHOL StopForumSpam toxic spam network StopForumSpam networks with large amounts of spambots. FireHOL mirror; StopForumSpam terms may apply; explicit feed license not stated in the mirrored file.
firehol_abusers_30d FireHOL abusers 30d web abuse Recent web abuse aggregate. FireHOL aggregate/mirror; upstream licenses vary.
firehol_cleantalk FireHOL CleanTalk form spammer CleanTalk form spammer snapshot. FireHOL mirror; upstream CleanTalk terms apply, explicit feed license not stated in the mirrored file.
myip MyIP blacklist web blacklist Website firewall blacklist archive. MyIP.ms publishes the download for website firewall use; no explicit data license found.
bitwire_outbound Bitwire IP blocklist bad reputation Aggregated outbound blocklist. Aggregated data is CC BY-NC-SA 4.0 and subject to original provider licenses; Bitwire code is MIT.
botvrij Botvrij IOC list threat IOC Threat IOC destination IPs. Botvrij describes the data as open-source IOCs; no explicit SPDX-style data license found.
alienvault_reputation AlienVault reputation bad reputation AlienVault IP reputation database. AlienVault/LevelBlue OTX terms may apply; no explicit license in the feed header.
turris Turris Sentinel Greylist service abuse Turris Sentinel service-abuse greylist data. CC BY-NC-SA 4.0; commercial offerings require contacting CZ.NIC.
firehol_anonymous FireHOL anonymous anonymizer Large anonymizer aggregate including Tor and open proxy sources; enrichment context, not a malware assertion. FireHOL aggregate/mirror; upstream anonymizer-list licenses may vary.

The CVE map uses Emerging Threats Open sid-msg.map. The ET Open license file states that older SID ranges are GPLv2 and Emerging Threats SID ranges are BSD; the generated cve.yaml only contains SID-to-CVE/CAN lookup values extracted from that map.

Development

Run tests:

uv run pytest

Confirm the Python runtime:

uv run python --version

Check the CLI:

uv run listbot --help
uv run listbot run --help

Run a live generation into a temporary directory. With an activated .venv, you can drop the uv run prefix here as shown in the usage section above.

tmpdir="$(mktemp -d)"
uv run listbot run --output-dir "$tmpdir"
ls -lh "$tmpdir"

Notes

  • The tool currently supports IPv4 only, matching the original generator.
  • Generated YAML and .bz2 files are build artifacts, not source files.

License

listbot code is licensed under GPL-3.0-only. See LICENSE.

The generated maps are derived from third-party OSINT feeds. Those feed terms remain governed by their upstream providers.

About

🤖 Daily updated translation maps for T-Pot

Resources

Stars

22 stars

Watchers

8 watching

Forks

Releases

Packages

Used by

Contributors

Languages