Skip to content
LouLouLibsPublic

About

Create symlinks across projects from TOML config

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

21 Commits

Folders and files

Repository files navigation

pipelink

Create symbolic links across project directories from a TOML configuration file. Designed for monorepos and data pipelines where one project's output is another project's input β€” pipelink wires them together with symlinks, avoiding data duplication.

Installation

Download a release

Download the latest binary for your platform from the releases page.

# macOS (Apple Silicon)
curl -L https://github.com/louloulibs/pipelink/releases/latest/download/pipelink_darwin_arm64.tar.gz | tar xz

# macOS (Intel)
curl -L https://github.com/louloulibs/pipelink/releases/latest/download/pipelink_darwin_amd64.tar.gz | tar xz

# Linux (x86_64)
curl -L https://github.com/louloulibs/pipelink/releases/latest/download/pipelink_linux_amd64.tar.gz | tar xz

Build from source

Requires Go 1.22+.

git clone https://github.com/louloulibs/pipelink.git
cd pipelink
go build -ldflags="-s -w" -o pipelink .

The -ldflags="-s -w" flag strips debug symbols for a smaller binary (~4 MB β†’ ~3 MB).

Usage

pipelink link

Read a TOML config and create all symlinks:

pipelink link input.toml

Preview what would happen without creating anything:

pipelink link --dry-run input.toml

pipelink validate

Check that all source files and directories exist, without creating symlinks:

pipelink validate input.toml

Exits with code 0 if all sources are present, 1 if any are missing.

pipelink pull

Download every type = "remote" entry into the local cache and verify it against its pinned sha256 (see Remote sources). Files already cached with a matching hash are skipped. Run link afterwards.

pipelink pull input.toml            # fetch what is missing or changed
pipelink pull --dry-run input.toml  # list remote entries and whether they are cached
pipelink pull --verify input.toml   # re-hash cached files instead of trusting their stamps

pipelink push

Upload a local file to an rclone remote and print its sha256 with ready-to-paste TOML and Nickel snippets. Config files are not modified.

pipelink push holdings.parquet s3-msi:bucket/project/13f/holdings.parquet

Flags

Flag Short Description
--dry-run -d Print actions without executing (link only)
--verbose -v Show additional output
--help -h Help for any command

Configuration

Pipelink reads a TOML file where each top-level table defines one link. Each entry has three sections: metadata, source, and target.

Link a single file

[STATION_TEMPS.metadata]
type = "file"
description = "Weather station temperature readings"

[STATION_TEMPS.source]
directory = "/data/NOAA"
file = "station_temps_2025.csv"

[STATION_TEMPS.target]
directory = "./input/Weather"
file = "station_temps_2025.csv"

Link multiple files

[POPULATION.metadata]
type = "files"
description = "County population estimates"

[POPULATION.source]
directory = "/data/Census/Population"
file = [
    "county_population.parquet",
    "county_demographics.parquet",
]

[POPULATION.target]
directory = "./input/Census"
file = [
    "county_population.parquet",
    "county_demographics.parquet",
]

Source and target file arrays must have the same length. Each file at index i in source is linked to the file at index i in target.

Link a directory

[CENSUS_MAPS.metadata]
type = "directory"
description = "TIGER/Line shapefiles"

[CENSUS_MAPS.source]
directory = "/data/Census/ShapeFiles"

[CENSUS_MAPS.target]
directory = "input/ShapeFiles/Census"

Remote sources

Some inputs are not produced by a sibling project: large shared datasets built once (e.g. a holdings panel used by several projects) or public replication packages. Keep them out of git, on remote storage, and declare them with type = "remote". pipelink pull downloads them into a cache shared by all projects on the machine, and pipelink link symlinks the cached file into place like any local source.

[HOLDINGS.metadata]
type        = "remote"
description = "Holdings panel, 1980-2024"
license     = "Licensed data: institution use only"   # optional, informational

[HOLDINGS.source]
uri    = "s3-msi:bucket/project/13f/holdings.parquet"  # rclone remote:path, or https:// URL
sha256 = "9db34bb6f836d8e7..."                          # pins the version; pull fails on mismatch
# cache = "~/data/cache"                                # optional

[HOLDINGS.target]
directory = "input/13F"
file      = "holdings.parquet"   # optional; defaults to the basename of uri
  • Backends. https:// and http:// URLs are fetched directly. Anything else is an rclone path (remote:path, including on-the-fly remotes such as :local:/abs/path), so every rclone backend works: S3, SFTP, Google Drive, Dropbox… rclone must be on PATH (or set PIPELINK_RCLONE), and credentials come from your rclone config.
  • Cache. source.cache if set, else $PIPELINK_CACHE, else ~/.cache/pipelink. The layout mirrors the URI (<cache>/<remote>/<path> or <cache>/<host>/<path>), and URIs containing .. are rejected. Two projects that pin the same file share one cached copy.
  • Integrity. Downloads go to a temporary file and are renamed into place only after the sha256 matches, so a failed or wrong download never replaces a good cached file. After verification, pipelink writes a stamp (.<name>.pipelink: hash, size, mtime) so later pulls of multi-GB files skip re-hashing while the file is untouched. --verify ignores stamps.
  • Unpinned entries. If sha256 is omitted, pull still downloads, then prints the hash with a warning so you can pin it.
  • link / validate. A remote entry counts as present only once it is cached. Neither command touches the network, and link reports uncached entries as missing with a hint to run pull.
  • Single files only. Archive a directory (tar) before pushing it.

Config reference

metadata (required)

Field Type Description
type string "file", "files", "directory", or "remote"
description string Optional human-readable description
generated_by string[] Optional list of scripts that produce the source
license string Optional; informational, e.g. for licensed remote data

source (required)

Field Type Description
directory string Absolute path to the source directory
file string or string[] Filename(s) within the directory. Omit for type = "directory"
task string Optional path prefix prepended to directory
uri string remote only: rclone remote:path or https:// URL
sha256 string remote only: expected hash of the file (strongly recommended)
cache string remote only: cache root override (default $PIPELINK_CACHE or ~/.cache/pipelink)

target (required)

Field Type Description
directory string Path to the target directory (relative to working directory, or absolute)
file string or string[] Target filename(s). Defaults to source filenames (for remote, the basename of uri) if omitted

Output

Pipelink prints colored output showing each link with Unicode arrows:

πŸ”—    Processing ... input.toml ... for linking    πŸ”—

      4 files to process

Linking  POPULATION     (multiple files)
         County population estimates
Target:  β”Œβ”€β–Ά input/Census/county_population.parquet
Source:  └── data/Census/Population/county_population.parquet
Target:  β”Œβ”€β–Ά input/Census/county_demographics.parquet
Source:  └── data/Census/Population/county_demographics.parquet

βœ“ 4 links created

Missing source files are filtered out with a warning and the remaining links are still created.

Snakemake integration

Use pipelink as a rule in your Snakemake pipeline to establish data dependencies before running analysis:

rule link_inputs:
    input:
        config="input.toml",
    output:
        touch(".links_created"),
    shell:
        "pipelink link {input.config} && touch {output}"

Nickel configuration

Pipelink pairs well with Nickel for type-safe, validated configuration. Instead of writing TOML by hand, define links in a .ncl file with contracts that enforce correct structure, then export to TOML.

A typical input.ncl in a project directory:

let link_contracts = import "../utilities/config/nickel/link_contracts.ncl" in
let
    Link = link_contracts.link,
    serialize_records = link_contracts.serialize_records
in

{
  POPULATION | Link = 'files {
    source = {
      file = ["county_population.parquet",
              "county_demographics.parquet"],
      directory = "/data/Census/Population",
    },
    target = { directory = "./input/Census" },
    metadata = {
      generated_by = ["import_census.R"],
      description = "County population estimates",
    },
  },

  STATION_TEMPS | Link = 'file {
    source = {
      file = "station_temps_2025.csv",
      directory = "/data/NOAA",
    },
    target = { directory = "./input/Weather" },
  },

  TIGER_LINES | Link = 'files {
    source = {
      file = ["tl_counties.shp", "tl_states.shp"],
      directory = "/data/Census/ShapeFiles",
    },
    target = { directory = "./input/ShapeFiles" },
  },
}
|> serialize_records

The Link contract validates each entry as one of three enum variants ('file, 'files, 'dir), and serialize_records flattens the structure into the TOML schema pipelink expects. Target filenames default to source filenames when omitted.

Export to TOML and link in one step:

nickel export input.ncl --format toml > tmp/input.toml
pipelink link tmp/input.toml

Or as a Snakemake rule:

rule link_inputs:
    input:
        config="input.ncl",
    output:
        toml="tmp/input.toml",
        stamp=touch(".links_created"),
    shell:
        """
        nickel export {input.config} --format toml > {output.toml}
        pipelink link {output.toml}
        """

About

Create symlinks across projects from TOML config

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages