title	ICSAC Editorial System
emoji	📄
colorFrom	indigo
colorTo	gray
sdk	static
pinned	false
license	mit
short_description	Open-source AI editorial pipeline for ICSAC submissions

ICSAC Editorial System

This is the open-source AI editorial pipeline that accepts and reviews every submission to the Institute for Complexity Science and Advanced Computing and its journal Persistence (/submit).

If you sent a paper to ICSAC — this repository is exactly what received and read it. The submission handler, the rubrics, the prompts, the citation checks, the redaction layer, the audit pass that scores the panel itself. With one defensive exception (the prompt-injection signal in the RQC audit is not published, to avoid teaching adversarial submissions what to avoid — see rubrics/review_quality_control.md for the rationale), nothing about how your work is evaluated is hidden.

Why this is public

Independent and heterodox researchers have legitimate cause to be skeptical of black-box editorial AI. The reasonable response is not to hide the system behind a "trust us." The reasonable response is to publish it.

You can read the rubrics ICSAC reviewers apply (rubrics/). You can read the templates that draft the acceptance and revise-and-resubmit letters (templates/, intake/templates/). You can read the redacted reviews of every accepted paper at icsacinstitute.org/accepted — produced by the exact redaction layer in this repo. You can run the pipeline yourself.

What this repo does NOT include

This repo is the AI reviewer half of the institute. Intentionally not public: the manuscript archive, author records, member directory, internal moderation tooling, the website that fronts all of this, and the operational infrastructure that runs it. Those remain private to protect the privacy and security of authors, members, the team, affiliates, and site visitors. What this repository exposes is how your manuscript was evaluated and what was recorded — not the people, payment, identity, or infrastructure layers around that evaluation.

What it does

The pipeline has two halves: a long-running submission front-end that accepts new manuscripts, and a batch review pipeline that runs on a timer and drains the queue. Both live in this repo.

For each submission:

1. Intake (intake/intake_server.py) — the FastAPI handler behind icsacinstitute.org/submit accepts a DOI / arXiv reference OR a direct PDF upload. The upstream form host verifies the submitter's ORCID via OAuth and signs each request with an HMAC; this handler re-validates and persists the submission record.

2. Manuscript fetch (submission_intake.py) — the worker either downloads the manuscript from the resolver (DOI route) or uses the uploaded PDF as the archive of record. Text and references are extracted.

3. Citation verification (citation_verify.py, citation_misattribution.py) — every cited work is checked against arXiv, Crossref, and Semantic Scholar. Fabricated and misattributed citations are flagged.

4. Code and data link (code_availability.py) — ICSAC links to an author's code and data and never hosts them. A paper that says its code or data is available must say where: the link given on the submission form, a supplement related identifier, or a repository link in the paper's body. A paper that makes the claim with no link anywhere is held before the panel runs and recommended for revise and resubmit; the curator signs that like every other decision, and the email quotes the paper's own sentence and asks for the link. The check reads availability statements, not the words "code" and "data", skips the reference list, and treats "not applicable" as nothing to link. A wrong flag is cleared with intake/code-check-override.sh <id>, which sends the paper to the panel. The published paper's page says: "Code and data: [link], hosted by the author. ICSAC links to it and doesn't host, run or maintain it."

5. Five-reviewer panel (review.py) — a panel of independent model instances reviews the manuscript against ICSAC's rubrics (scope, methodology, calibration, tone, AI provenance signal). Each reviewer scores blind to the others. The panel tolerates one slot failure per pass via self-heal (MIN_REVIEWERS=4), so a published review may reflect four valid outputs if a slot errored. Each slot is a chain of providers (direct free-tier providers with the institute's own keys, the Hugging Face router, OpenRouter). A paper the panel cannot staff is re-queued on every batch tick and the curation team is alerted loudly once it has waited a day for reviewers.

6. Review quality control (review_quality_control.py) — a separate auditor reviews the panel itself. Low-confidence dimensions, missing injection indicators, or systemic drift trigger curator alerts.

7. Redaction (redaction.py) — internal reasoning, vendor names, and operational metadata are stripped before any review is shared with the author or published.

8. Decision (action.py) — the panel recommends one of three outcomes:

   • Accept — a DOI under the institute's own Crossref prefix is registered and the paper is published at icsacinstitute.org/publications/<slug> with a redacted copy of the panel's review; an archival copy goes to Zenodo under that same DOI.
   • Revise and resubmit — author receives the panel's feedback and may resubmit a revised version.
   • Scope reject — reserved for submissions that fall outside the institute's remit (pseudoscience, non-engageable). Not a standard editorial outcome.

   Every panel run produces a recommendation, not a verdict. The recommendation, the full panel report, and the RQC audit are sent to the curator via the operator's configured notification channel. The curator confirms, modifies, or overrides before any author email is drafted, any DOI is registered, or any deposit is staged. The panel never auto-delivers a decision.

After acceptance

Acceptance is where most of the human-in-the-loop machinery lives. Nothing in this part of the pipeline sends an email or registers anything on its own.

1. The curator's accept (intake/decide.sh <id> accept) stages a Crossref deposit draft (crossref_deposit.py): a DOI string is reserved under the institute's prefix, the deposit XML is built and validated against the Crossref schema, and nothing is sent. A Zenodo archive draft is staged carrying that same DOI as an external identifier, so Zenodo mints nothing. The acceptance email is written to the operator's mailbox as a draft; the code never sends it.
2. The author's response page (intake/author_approval.py) — the acceptance email carries a personal link and a date. On the page the author approves, holds, or withdraws; opts out of any promotion category (social media, print or broadcast, newsletters, website features) or of the annual volume, each all-or-nothing; leaves notes for the curation team; and ticks whether the institute may quote those notes. Every response is timestamped and audited. The window is seven days by default; if it closes in silence the operator is told once. Nothing on the page publishes anything.
3. Registration (intake/register-doi.sh <id> --live, run by the operator after a typed confirmation) refuses a hold, a withdrawal, or an open window without an answer, then registers the DOI at Crossref, pushes the landing page, publishes the Zenodo archive under the same DOI, and drafts the author's "published" notice. Each step is checkpointed so a failed run resumes where it stopped instead of repeating anything.
4. One follow-up (intake/post_publication.py), seven days after registration: a drafted invitation to join the institute's reviewer pool. It is skipped when the author opted out of newsletters, held, withdrew, or already holds a credential, and it never repeats.
5. No automated email asks for a reply. Every choice an author has is a link or a button on the response page; support starts with an email to help@icsacinstitute.org carrying the submission ID.

What it is not

This is the system one institute uses to evaluate its own submissions. It is not a general-purpose academic peer review platform. It is not a service you can submit to without going through icsacinstitute.org/submit. It is not a replacement for human editorial judgment — every panel outcome routes to a human curator for the final verdict, and any author who disagrees with that verdict can appeal to the curation team.

It is also opinionated. The rubrics reflect the institute's editorial scope: complexity science, information theory, persistence dynamics, and adjacent methodology. A submission outside that scope will be flagged as out-of-scope regardless of its merit on its own terms.

Repository layout

The editorial-side modules live at the top level. The submission front-end is a Python subpackage at intake/.

Editorial pipeline (top level)

Path	Purpose
editorial_workflow.py	Top-level workflow orchestration
submission_intake.py	Manuscript fetching and text extraction
citation_verify.py	Cross-reference citation validation
citation_misattribution.py	Catches "real DOI, wrong paper" errors
code_availability.py	Holds a paper that says it has code or data but gives no link
review.py	The five-reviewer panel
review_quality_control.py	Auditor that scores the panel
redaction.py	Redaction layer for public review output
action.py	Decision dispatch (accept / R&R / scope reject)
repository_deposit.py	Stages accepted papers to a repository backend (Zenodo)
email_send.py, email_render.py	Author correspondence primitives
publications.py, publish_watcher.py	Post-acceptance publication
watch.py	Polls Zenodo for curator state transitions and posts branded comments on accept/decline
notify.py	Curator-channel notification helpers (channel-agnostic dispatcher; forkable)
directory.py	Community directory data layer
stats.py	Aggregate statistics over historical reviews
rubrics/	The editorial rubrics applied to every submission
templates/	Editorial-side correspondence templates (Zenodo-comment posts, etc.)
editorial-batch.{service,timer}, editorial-review.{service,timer}, submission-watcher.{service,timer}	systemd units for the batch workflow, community-request poller, and intake/decision watcher
crossref_deposit.py	Crossref registration: stages a schema-validated deposit draft at accept; register --live is the operator's irreversible, checkpointed step

Submission front-end (intake/)

Path	Purpose
intake/intake_server.py	FastAPI app: POST /api/submit, GET /api/submission/{id}/state, GET /healthz
intake/submission_worker.py	Drains the queue, resolves deferred DOIs, dispatches into the review pipeline
intake/apply_decision.py	Applies the curator's verdict: publications registration, deposit staging, author email
intake/code-check-override.sh	Clears a wrong code-and-data flag and queues the paper for the panel
intake/author_approval.py	The author's response page: approve / hold / withdraw, exclusions, quote permission, the objection window
intake/post_publication.py	The one post-publication follow-up (reviewer invitation), drafted seven days after registration
intake/notify_author.py	Author email rendering; attaches redacted panel report + RQC as PDFs
intake/rehydrate.py	Refetch a stubbed DOI submission's bytes from the resolver, verify SHA
intake/templates/	Author-facing email templates
intake/scripts/	Smoke tests for the test-mode pipeline
intake/README.md	Subpackage-scoped documentation

Running it yourself

The pipeline is designed to run on a single host. The submission server runs as a long-lived FastAPI service; the editorial workflow runs from systemd timers (twice daily by default) and drains the queue when it ticks.

Dependencies:

• A submission repository (we use Zenodo — needs an API token with read + deposit scopes)
• An LLM provider (OpenRouter, Anthropic, or compatible)
• An SMTP account for author correspondence
• A registry destination (the institute's website repository, in our case) for post-acceptance publication of accepted papers and redacted reviews

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

cp config.example.py config.py     # edit for SMTP, Zenodo, LLM provider
export INTAKE_HMAC_SECRET=$(openssl rand -hex 32)   # for the intake HMAC gate

# Submission server (long-running):
uvicorn intake.intake_server:app --host 127.0.0.1 --port 8443 --log-level warning

# Submission worker (drains the queue when markers appear):
python -m intake.submission_worker

# Editorial batch tick (run from systemd timer in production):
python editorial_workflow.py batch-tick

Set secrets in /etc/icsac/editorial.env (loaded by systemd EnvironmentFile=). For manual runs, source the same file. See intake/README.md for the full list of intake-side environment variables.

License

MIT. See LICENSE.

The rubrics and review templates are MIT-licensed code artifacts — feel free to fork, adapt, and use as the basis for your own institute's review system.

A note to authors

If your paper was reviewed by this system and you disagree with the panel's recommendation: write to help@icsacinstitute.org with your submission ID in the subject line. The panel's recommendation is a first pass: the curation team makes every decision and reads every appeal.

If your paper was accepted: the redacted review is published at icsacinstitute.org/publications/ alongside the work itself, and you decide on your own response page how the institute may promote it.

If you want to know exactly which prompts the panel saw, which rubrics it applied, which citations it verified, and which form fields the submission handler collected before forming its recommendation: read the source. That is why this repository exists.

Support this work

ICSAC operates on bootstrap funding and private donations. Open review for independent and heterodox researchers is the kind of infrastructure no one is paid to build. If you want to help keep it running:

icsacinstitute.org/community/donate