[Epic] Macro & Scripting Support (Lua-based macro system)

[AgreeDK]

Background

A macro/scripting language is one of the most requested features from users
moving from GSAK. This epic tracks the work towards a modern successor to the
GSAK Macro Language, as described in the attached concept document
(OpenSAK Lua Macro System Concept v4).

In short: an embedded Lua runtime with a small, documented OpenSAK macro API.
Lua provides the language; OpenSAK provides the geocaching vocabulary. GSAK
macros are ported to Lua rather than run unchanged.

This does not depend on Geocaching.com API access. Macros can work with
everything OpenSAK already stores; only the extra fields that come from the
API (roadmap item 11) will be missing until that access exists. The macro API
must handle missing fields gracefully by returning nil rather than raising an
error (concept §9a).

This is a tracking issue. It isn't meant to be implemented in one go. Each
step below is split out into its own issue when it's picked up (created right
before the code, not all upfront), in the same way as #821.

Community use cases (the design is driven by these)

• Macro description: refile caches into correct database #808 — Refile caches into the correct database (loops, create/switch database, move caches)
• Macro description: filter database and dump gpx #809 — Filter database and dump GPX (raw SQL WHERE filter + per-database export)
• Macro use case: dump sqlite table #810 — Dump SQLite table (arbitrary read-only query → table/HTML report)
• Macro Description: FindStatGen #813 — FindStatGen

More real-world GSAK macros are welcome as comments or new use-case issues.
A goal for the migration phase is that all of the above can be ported to
OpenSAK as reference examples.

Plan

Sequencing update (Oct 2026): following feedback from @GeePa67, UI functions and per-macro saved settings (#982, #983) now come before the Macro Manager and migration support. Most GSAK macros depend on forms, so there's little to manage or port until those exist.

Step 1 — Headless command-line import (#937)

Import GPX/PQ files from a folder and/or a GSAK database without the GUI, so a
refresh can be scheduled with the OS's own tools. It covers part of the
automation demand immediately, and it moves the import logic out of the GUI
dialogs into a shared import service. The macro API's
opensak.import_gpx() will call that same service later.

Step 2 — Lua packaging spike (go/no-go)

The biggest technical risk isn't the API design. It's whether a Lua runtime
can be shipped reliably in every build artifact. Before any API work, verify
with a minimal build that runs a .lua file calling show_message() and
log():

• Python–Lua bridge chosen (Lupa is the main candidate): licence, Python
  3.12 wheels for Windows x64, Linux x86_64, macOS arm64 and x86_64
• Which Lua version/runtime to embed (e.g. Lua 5.4 vs. LuaJIT)
• Bundled correctly by PyInstaller in all four artifacts
• Works inside the MSIX package
• macOS: native extension is signed and the .dmg still notarizes
• Smoke test in CI on all platforms

Outcome: a documented go/no-go and the chosen bridge. If it fails, the design
is revisited before continuing.

Step 3 — SQL Query tool (#810)

A native, read-only tool (e.g. under Tools) to run an arbitrary SELECT
against the active database and view the result as a table. It's useful on
its own right away, lets users test queries interactively, and later becomes
opensak.sql_query().

• Read-only enforced at the database level (e.g. a read-only SQLite
  connection and/or PRAGMA query_only), not only by checking the SQL text
• A schema reference / "show tables and columns" helper
• Export of the result (CSV/HTML) to cover the report part of Macro use case: dump sqlite table #810
• Related: Distance and heading sql extensions #917 (distance and heading SQL extensions)

Step 4 — Macro API v1: design decisions

A short design issue to settle the open questions before implementation:

• Sandboxing: which Lua standard libraries are removed or restricted (os,
  io, load/dofile, require, …); no access to Python internals
• Cancellation / execution limits for runaway macros (instruction-count
  hook, a Cancel button)
• Transactions: does a macro that modifies many caches run in one
  transaction, with rollback on error? Should it prompt for a backup first?
• Threading: how a long macro runs without freezing the GUI
• Error reporting with macro file name and line number
• API versioning (Requires-API metadata, concept §8)
• Macro folder location per platform (alongside the existing data paths)
• Mapping concept terms to OpenSAK's actual data model. For example, the
  concept's cache:add_tag(): what is OpenSAK's equivalent (user flag,
  user note, something new)?
• Confirm the known limitation in concept §9b (raw WHERE text is not
  parameterized) is accepted for v1

Step 5 — Phase 1: proof of concept

Macros menu → Run Macro…; runs a .lua file with opensak.show_message(),
opensak.log() and read-only access to a small set of cache fields.

Step 6 — Phase 2: useful macro API

• Cache access: current cache, cache by code, all / filtered caches
• Filtering and selection, reusing the Filter dialog's Where tab engine
  (opensak.filter(where_sql)); sorting
• User notes and write operations on caches (per the Step 4 decisions)
• Import/export wrappers (GPX, CSV) on top of the shared import service from Step 1
• Database management for Macro description: refile caches into correct database #808: database_exists, create_database,
  switch_database, move caches — thin wrappers around DatabaseManager
• Regex functions bridged to Python re (regex_match / regex_replace /
  regex_find), since Lua patterns aren't enough for GSAK-style description parsing
• Utilities: sleep(ms), sql_query() (from Step 3), a simple table/HTML
  report helper
• Nil-safe field access from the start (concept §9a)

Step 7 — UI functions and saved settings

• Lua macros: UI functions — simple dialogs and declarative forms #982 — UI functions: simple dialogs (message, input, choose, file/folder picker) first, then declarative forms built from a Lua table (no direct Qt access)
• Lua macros: per-macro saved settings (load/save user options) #983 — Per-macro saved settings: a private JSON store per macro, so user options persist between runs; integrates with forms to prefill last-used values

Step 8 — Phase 3: Macro Manager

Discovery of local macro files, metadata display, error display,
Edit / Open Macro Folder / Reload, and optional keyboard shortcuts. Candidate:
running a macro headless via the command line from #937 (e.g. --run-macro),
so macros can also be scheduled.

Step 9 — Phase 4: migration support

A GSAK → OpenSAK porting guide with common patterns, and the community use
cases (#808, #809, #810, #813) ported as reference macros. Optional
compatibility helpers for frequently used GSAK concepts.

Step 10 — Phase 5: extended API

More database, waypoint, coordinate and export functions, plus further form
field types, driven by real macros people send in, not a wishlist.

Out of scope for v1

• A sharing/marketplace platform for macros. Macros are exchanged externally
  (forum, Facebook group, GitHub), as GSAK macros are today. The runtime must
  still be safe to run macros of unknown origin (concept §9).
• Running GSAK macro source unchanged / full GSAK interpreter compatibility.
• A built-in scheduler. Scheduling uses the OS's own tools, via the CLI from Headless command-line import mode (GPX folder + GSAK database) for scheduled/unattended runs #937.

Checklist

• Step 1 — Headless command-line import (Headless command-line import mode (GPX folder + GSAK database) for scheduled/unattended runs #937)
• Step 2 — Lua packaging spike (go/no-go)
• Step 3 — SQL Query tool (Macro use case: dump sqlite table #810)
• Step 4 — Macro API v1 design decisions
• Step 5 — Phase 1: proof of concept
• Step 6 — Phase 2: useful macro API
• Step 7 — UI functions (Lua macros: UI functions — simple dialogs and declarative forms #982) and saved settings (Lua macros: per-macro saved settings (load/save user options) #983)
• Step 8 — Phase 3: Macro Manager
• Step 9 — Phase 4: migration support (use cases ported)
• Step 10 — Phase 5: extended API

OpenSAK_Lua_Macro_System_Concept_v4.md

[AgreeDK]

Sharing macros (planned, not part of v1 runtime work)

Consistent with the concept's scope (no in-app sharing platform in v1), sharing
happens on GitHub, in two tiers:

• Official examples in the main repo (e.g. examples/macros/): a small set
  maintained by the project, including the ported use cases (Macro description: refile caches into correct database #808, Macro description: filter database and dump gpx #809, Macro use case: dump sqlite table #810,
  Macro Description: FindStatGen #813). Run in CI against a test database, doubling as regression tests for
  the macro API.
• Community repository OpenSAK-Org/OpenSAK-Macros (same model as
  OpenSAK-Data): submissions via pull request, tested and approved by
  maintainers.
  • One folder per macro (.lua + short README); the metadata header from
    concept §8 is mandatory and validated in CI
  • CI generates an overview table and an index.json, so a future in-app
    browser could read it without restructuring
  • Once --run-macro exists (Step 7), CI runs each macro against a test
    database
  • One licence for the whole repository (e.g. MIT); ported GSAK macros need
    the original author's permission (stated in the PR template)
  • "Approved" means reviewed and tested against a given OpenSAK version, not
    a guarantee. The sandbox is what provides safety
  • A "Submit a macro" issue form for users who don't use git, plus a
    Discussions category for drafts and questions

To be set up around Phase 3–4, once there are macros to share.

[AgreeDK]

Editor support (from community feedback)

Brian Hart asked for OpenSAK's Lua API to be supported in VS Code, not just plain Lua. He currently writes GSAK macros in Notepad++, which handles syntax checking but makes debugging cumbersome.

Proposed approach:

Ship an API definition file for the Lua Language Server (the standard VS Code Lua extension), so opensak.* functions get autocomplete, signature help and type checking. Ideally generated from the same source as the API docs, so the two can't drift apart.
Runtime errors always report the macro's file name and line number (already in the concept, §9).
Step-through debugging is a possible later step, not v1. It's harder with an embedded runtime.

Source: Facebook comment on the Macro Support update, Sept 2026.

[GeePa67]

I feel that UI functions should come before a macro manager or migration support. I think you will find that the majority of GSAK macros have some UI component so you won't have a lot for manage or migrate until UI features exist.

[GeePa67]

Another thing to consider is that GSAK allows for saving user options for macros into an xml file (could easily use json or some other format though) that can easily be written/loaded by the macro with standard macro functions. This feature is very nice so users don't have to change a bunch of settings every time they run the macro.

[AgreeDK]

@GeePa67

Thanks, both points are well taken.

UI before manager/migration: Agreed. Right now the API has only opensak.confirm() on the UI side, and you're right that most real GSAK macros depend on forms, so there'd be little to migrate without them. We'll reorder so UI comes before the macro manager and migration support. Our current thinking is two steps: first a set of simple dialogs (message, text input, pick from list, yes/no, file/folder picker), then custom forms. For forms, rather than recreating GSAK's Form Designer or exposing Qt directly, we're leaning towards a declarative approach: the macro describes its fields in a Lua table and OpenSAK builds the dialog. That keeps the sandbox intact and should cover most of what GSAK forms are used for. If you have examples of macros with forms you consider typical (or tricky), they'd be very useful for sizing this.

Saved macro options: Good idea, and a natural fit. We'd give each macro its own JSON settings store that it can load/save through the API, scoped so a macro can only access its own settings. Combined with the forms above, a dialog could prefill itself with the last-used values automatically, so users don't have to re-enter their settings every run.

[AgreeDK]

Thanks again @GeePa67, split out as #982 (UI functions and declarative forms) and #983 (per-macro saved settings), and the plan above is updated so they come before the Macro Manager and migration support.