Docker MCRIT

This repository is the Docker deployment of the MinHash-based Code Recognition and Investigation Toolkit. docker-compose.yml runs five services: MongoDB, the MCRIT server and worker (one image, built here from the upstream tag), the MCRITweb frontend behind gunicorn, and NGINX in front of it. The versions are pinned in .env - MCRIT_TAG/MCRIT_BRANCH, MCRITWEB_TAG/MCRITWEB_BRANCH, MONGO_TAG and NGINX_TAG - and moving them is how an upgrade is done. MCRIT itself was released at Botconf 2023 (paper, slides, video); for using it, see the documentation in the two upstream repositories.

Requirements

Docker Engine with the Compose v2 plugin, so that docker compose works. The images build on amd64 and arm64. Disk is the sizing constraint: ./storage/mongodb grows with the indexed corpus, and MongoDB's cache defaults to about half of the host's RAM.

First deployment

git clone https://github.com/familiary/docker-mcrit.git
cd docker-mcrit
chown -R 10001:10001 storage/mcritweb
docker compose up -d

The containers run as uid 10001, so ./storage/mcritweb - a host bind mount holding MCRITweb's SQLite database and cached reports - has to be handed to that uid once; MCRITweb refuses to start with that command in the message otherwise. The first up builds both images, which takes a while.

NGINX publishes ports 80 and 443. MongoDB, the MCRIT API on 8000 and MCRITweb on 5000 are published on the loopback interface only. Register the first account through the web interface; it becomes the administrator.

Configuration

.env holds the version pins and is read by both compose files. MCRIT_AUTH_TOKEN is read from the environment or from .env and passed to the server and the worker.

config/ holds MCRIT's configuration modules as this repository ships them. The deviations from MCRIT's stock configuration are STORAGE_SERVER, QUEUE_SERVER and BAND_MATCHES_REQUIRED, each carrying a comment saying why. docs/TUNING.md mirrors the upstream tuning guide.

A deployment's own settings go into config.local/, which is gitignored: at startup each MCRIT container copies config/ into the installed package's config directory and then copies config.local/ on top. A file there replaces the shipped module of the same name, so copy the whole module, edit it, and keep the settings you are not changing. Everything you do not override keeps tracking the repository and a git pull cannot conflict with your settings.

If the MCRIT being deployed defines settings the assembled configuration does not carry, every MCRIT container says so at startup and keeps running:

WARNING: StorageConfig.py is missing settings the installed MCRIT defines: STORAGE_MONGODB_FLAGS

The setting is absent rather than at its upstream default, so add it to config/, or to the config.local/ file that replaced it.

MongoDB reads mongodb/mongod.conf, mounted read-only into the container, so its settings are a file to edit rather than flags appended to a command: list. The WiredTiger cache size is the setting worth revisiting: it defaults to about half of the host's RAM minus 1 GB, which is too much when MongoDB shares the host with the workers. The file sets no log path on purpose, so mongod keeps logging to stdout and docker compose logs mongodb shows it.

MCRITweb reads storage/mcritweb/config.py, its Flask instance configuration, which is not tracked here. TRUSTED_PROXY_COUNT belongs in it. From 1.5.0 MCRITweb meters failed logins per source address, and because everything here is served through the included NGINX it sees NGINX's address rather than the caller's unless told how many proxies to look past. Left at the default, ten failed logins from anywhere lock out every user for fifteen minutes:

TRUSTED_PROXY_COUNT = 1

1 is the count for this deployment as shipped - the NGINX in docker-compose.yml and nothing else. Add one for each further proxy in front of it. Counting matters in both directions: too low meters every user together, too high lets a caller pick their own throttle key by sending the header themselves. Development mode starts no NGINX, so an instance reached directly on port 5000 wants the default of 0 and needs no file.

NGINX serves plain HTTP by default, which is meant for a first look. For TLS, copy nginx/ssl/fullchain.pem.example and nginx/ssl/privkey.pem.example to the same names without the .example suffix, fill them with the certificate and key, and in the nginx service of docker-compose.yml swap the mcritweb_plain.conf mount for the two commented-out mcritweb_ssl lines. The .pem files are gitignored. Both site configurations listen for server_name localhost and reject other hosts, so set the deployment's name in whichever one is in use.

Running in production

Serve over TLS as described above. Set MCRIT_AUTH_TOKEN, which makes the MCRIT server reject unauthenticated requests; MCRITweb has no matching variable, because it keeps the token per server in its own database, so after the first start put the same value under Server in the administration menu, at Change Backend Server; the form that creates the first account asks for it as well. Until both sides carry it, MCRITweb's own calls are rejected too. Passing the token through the compose environment puts it where docker inspect and docker compose config show it, so anyone with access to the Docker host has the token: treat host access as equivalent to API access.

Log rotation needs nothing: every service logs to the json-file driver capped at five files of 50 MB. Back up ./storage/mongodb. A file-level copy is only valid with everything stopped; for a running instance, stream a dump out:

docker compose exec -T mongodb mongodump --archive --gzip > mcrit-dump.archive.gz

Upgrading

An MCRIT or MCRITweb bump is a change to the pins in .env together with a CHANGELOG.md entry; RELEASING.md describes the procedure, including regenerating config/ and what the changelog entry has to say. CHANGELOG.md is where a release states whether it needs a migration - the disassembly split that ./migrate.sh completes is one such step.

An upgrade that crosses several MCRIT releases takes each release's Upgrading steps in order. From 1.9.x to 1.13 that means rebuilding the images, letting the first start build its indexes, and running ./repair.sh once before serving matching - see the 2026-09-30 entry in CHANGELOG.md for the measured durations.

MONGO_TAG is 8.0. A fresh instance needs nothing. An existing ./storage/mongodb written by 5.0 cannot be opened by 8.0 directly, because MongoDB refuses to skip a major version: step it 5.0 -> 6.0 -> 7.0 -> 8.0, raising featureCompatibilityVersion before each next step. Back up first; raising the FCV is not reversible. For each of 6.0, 7.0 and 8.0 in turn:

docker compose down -t 60                         # give mongod time to shut down cleanly
sed -i 's/^MONGO_TAG=.*/MONGO_TAG=6.0/' .env     # then 7.0, then 8.0
docker compose up -d --wait mongodb
docker compose exec mongodb mongosh --quiet --eval 'db.adminCommand({setFeatureCompatibilityVersion: "6.0"})'

From 7.0 onwards the command also requires confirm: true, so those two steps read {setFeatureCompatibilityVersion: "7.0", confirm: true}. If the corpus is disposable, ./reset.sh discards it instead.

Check the SMDA escaper fingerprint after any rebuild

MCRIT does not pin SMDA, so rebuilding the images can pull a newer SMDA whose instruction escaping differs. MinHashes are derived from the escaped representation, so when the escaping changes, previously indexed signatures stop being comparable to newly computed ones. The failure is silent: the old signatures still look valid and still match each other, while identical code submitted afterwards no longer finds them. SMDA 4.4.5 is the worked example - it corrected how segment-qualified memory operands are escaped, and altered about a fifth of the stored MinHashes on a real corpus.

/status reports what the running build produces, so note it after a build and compare it after the next one:

curl -s http://127.0.0.1:8000/status | python3 -m json.tool | grep -E "smda_version|escaper_fingerprint"

An unchanged fingerprint means stored MinHashes stay valid across the rebuild and nothing is owed. A changed one means the index should be re-minhashed to stay internally consistent - the stored disassembly makes that a local recomputation, with no need to re-submit any samples.

Development mode

docker-compose-dev.yml runs MCRIT and MCRITweb from host checkouts under ./repositories, so edits take effect without rebuilding. Clone them first, then start the stack:

./clone_repositories.sh
docker compose -f docker-compose-dev.yml up

It starts no NGINX: MCRITweb is on port 5000 and the MCRIT API on port 8000, both on loopback.

Version history

CHANGELOG.md records which MCRIT and MCRITweb a pull now gives you, what changed in the deployment itself, and what an upgrade requires.

This repository is licensed under the GNU General Public License v3, see LICENSE.