Geocruncher

Geocruncher is a standalone application for Geological Computations. It is a dependency of the VisualKarsys webservice developped by ISSKA/SISKA.

It implements certain types of computations in Python and C++, and also acts as a link to libraries such as gmlib, develop by the BRGM.

It has the following architecture, handled by docker compose:

• A Flask HTTP API that accepts JSON parameters and Protobuf geological models, queues computation requests, reports their status, and returns results
• A Worker system using Celery, to handle computations in different work queues
• A message broker & temporary file storage & result backend using redis

See the HTTP API reference for the complete contract and API documentation.

To run computations locally, start the Docker stack and call its HTTP API. See the local setup and migration guide. For in-process Python workflows, import the functions in geocruncher.computations.

Geocruncher possesses an optional integrated profiler and monitoring stack, used in production by the VisualKarsys team for observability, to estimate computation times and identify key areas where performance should be improved.

Installation

We recommand using the latest version of Docker. Tested on Docker 24 to 27. No other steps are requiered after cloning the repository. We recommand using the master branch.

git clone https://github.com/ISSKA/geocruncher
cd geocruncher
git checkout master
git submodule update --init

Afterwards you can start the application with the provided run script:

./scripts/run.sh

Development

To run Geocruncher in local development mode in Docker container with Hot Reloading, run the script ./scripts/run.sh

To create a local Python venv that mirrors the base and local stages of docker/Dockerfile, run:

./scripts/setup-local-venv.sh
source .venv/bin/activate

The script uses uv, builds the native Draco and PyGeoAlgo pieces, and stores downloaded native dependencies under .cache/local-dev. On Linux, the local venv requires glibc 2.39 or newer because the published forgeo-gmlib wheel targets manylinux_2_39_x86_64. Older Linux distributions should use the Docker setup. BRGM will not publish a version of forgeo-gmlib for older glibc versions for safety reasons.

Recommanded extensions : ms-python.python, charliermarsh.ruff and astral-sh.ty.

We use Ruff for both linting and formatting. It is included in the dev dependency group and configured in pyproject.toml. Run it manually with:

uv run ruff check --fix
uv run ruff format

We use ty for type checking. It is included in the dev dependency group and configured in pyproject.toml. Run it manually with:

uv run ty check .

We use pytest for tests. It is included in the dev dependency group and configured in pyproject.toml. The test suite is split into:

• tests/unit: focused tests for pure helpers and small module behaviours
• tests/contract: API, task and computation boundary tests using fakes for Redis, Celery and native computations
• tests/integration: real dummy-project computations and mesh codec checks using the native geometry dependencies

Run the fast unit and contract suites with:

uv run pytest tests/unit tests/contract

Run the full suite with:

uv run pytest

Generate a coverage report for the fast suite with:

uv run pytest tests/unit tests/contract --cov=geocruncher --cov=api --cov-report=term-missing

Integration tests are marked with integration, tests requiring native geometry dependencies are marked with native, and longer-running dummy-project computations are marked with slow. To run only the integration suite:

uv run pytest -m integration

Pre-commit hook

Ruff and ty are wired up via pre-commit so they run automatically before each commit. After cloning, install the hook once per checkout:

uv run pre-commit install

To run the hooks against the whole repository: uv run pre-commit run --all-files.

Protobuf

We use Buf to format, lint and validate the Protobuf schema in proto/. The published module and generated SDKs are available in the Buf Schema Registry.

Run the checks from the module directory:

cd proto
buf format --diff
buf lint
buf build

Monitoring

Geocruncher includes an optional monitoring stack for observing Celery worker metrics, computation performance and logs. The monitoring setup uses:

• Prometheus - Time-series database for metrics collection and storage
• Grafana - Visualization interface with pre-configured dashboards and datasources
• Celery Exporter - Exports Celery worker metrics to Prometheus
• Loki - Log aggregation backend for container logs collected from the monitoring environment
• Alloy - Collects Docker container logs and forwards them to Loki

To start the monitoring stack:

cd monitoring
sudo docker-compose -f monitoring.docker-compose.yaml up -d

Once running, access Grafana at http://localhost:3000 (default credentials: admin/admin). The pre-provisioned dashboard of the celery-exporter displays Celery task metrics, queue lengths, worker status, and processing times. The other pre-provisioned dashboard visualizes the profiling data collected by the profiler of Geocruncher.

Prometheus is available at http://localhost:9090 for direct metric queries.

Loki is available at http://localhost:3100 for direct log queries. In Grafana, the Loki datasource can be used to explore logs collected by Alloy from Docker containers, alongside the Prometheus and Redis datasources already provisioned.

The monitoring stack connects to your existing Geocruncher instance via Redis on the host machine. Alloy also needs access to the Docker socket to discover containers and ship their logs to Loki. Ensure your main Geocruncher services are running before starting monitoring.

Deployment

Inside the Docker folder you will find the deployment files for the VisualKarsys webservice's infrastructure. We have a dev and prod environment, and use systemd services and journalctl for logging.

Please adapt to your needs.

The API exposes an HTTP server with no compression. We strongly recommand putting a proxy in front that handles HTTPS and GZIP compression to greatly improve security and performance.

Depending on your needs, we also recommand implementing basic authentification to prevent unwanted users from starting computations on your instance, if it is exposed to the internet.