Create symbolic links across project directories from a TOML configuration file. Designed for monorepos and data pipelines where one project's output is another project's input β pipelink wires them together with symlinks, avoiding data duplication.
Download the latest binary for your platform from the releases page.
# macOS (Apple Silicon)
curl -L https://github.com/louloulibs/pipelink/releases/latest/download/pipelink_darwin_arm64.tar.gz | tar xz
# macOS (Intel)
curl -L https://github.com/louloulibs/pipelink/releases/latest/download/pipelink_darwin_amd64.tar.gz | tar xz
# Linux (x86_64)
curl -L https://github.com/louloulibs/pipelink/releases/latest/download/pipelink_linux_amd64.tar.gz | tar xzRequires Go 1.22+.
git clone https://github.com/louloulibs/pipelink.git
cd pipelink
go build -ldflags="-s -w" -o pipelink .The -ldflags="-s -w" flag strips debug symbols for a smaller binary (~4 MB β ~3 MB).
Read a TOML config and create all symlinks:
pipelink link input.tomlPreview what would happen without creating anything:
pipelink link --dry-run input.tomlCheck that all source files and directories exist, without creating symlinks:
pipelink validate input.tomlExits with code 0 if all sources are present, 1 if any are missing.
Download every type = "remote" entry into the local cache and verify it against its pinned sha256 (see Remote sources). Files already cached with a matching hash are skipped. Run link afterwards.
pipelink pull input.toml # fetch what is missing or changed
pipelink pull --dry-run input.toml # list remote entries and whether they are cached
pipelink pull --verify input.toml # re-hash cached files instead of trusting their stampsUpload a local file to an rclone remote and print its sha256 with ready-to-paste TOML and Nickel snippets. Config files are not modified.
pipelink push holdings.parquet s3-msi:bucket/project/13f/holdings.parquet| Flag | Short | Description |
|---|---|---|
--dry-run |
-d |
Print actions without executing (link only) |
--verbose |
-v |
Show additional output |
--help |
-h |
Help for any command |
Pipelink reads a TOML file where each top-level table defines one link. Each entry has three sections: metadata, source, and target.
[STATION_TEMPS.metadata]
type = "file"
description = "Weather station temperature readings"
[STATION_TEMPS.source]
directory = "/data/NOAA"
file = "station_temps_2025.csv"
[STATION_TEMPS.target]
directory = "./input/Weather"
file = "station_temps_2025.csv"[POPULATION.metadata]
type = "files"
description = "County population estimates"
[POPULATION.source]
directory = "/data/Census/Population"
file = [
"county_population.parquet",
"county_demographics.parquet",
]
[POPULATION.target]
directory = "./input/Census"
file = [
"county_population.parquet",
"county_demographics.parquet",
]Source and target file arrays must have the same length. Each file at index i in source is linked to the file at index i in target.
[CENSUS_MAPS.metadata]
type = "directory"
description = "TIGER/Line shapefiles"
[CENSUS_MAPS.source]
directory = "/data/Census/ShapeFiles"
[CENSUS_MAPS.target]
directory = "input/ShapeFiles/Census"Some inputs are not produced by a sibling project: large shared datasets built once (e.g. a holdings panel used by several projects) or public replication packages. Keep them out of git, on remote storage, and declare them with type = "remote". pipelink pull downloads them into a cache shared by all projects on the machine, and pipelink link symlinks the cached file into place like any local source.
[HOLDINGS.metadata]
type = "remote"
description = "Holdings panel, 1980-2024"
license = "Licensed data: institution use only" # optional, informational
[HOLDINGS.source]
uri = "s3-msi:bucket/project/13f/holdings.parquet" # rclone remote:path, or https:// URL
sha256 = "9db34bb6f836d8e7..." # pins the version; pull fails on mismatch
# cache = "~/data/cache" # optional
[HOLDINGS.target]
directory = "input/13F"
file = "holdings.parquet" # optional; defaults to the basename of uri- Backends.
https://andhttp://URLs are fetched directly. Anything else is an rclone path (remote:path, including on-the-fly remotes such as:local:/abs/path), so every rclone backend works: S3, SFTP, Google Drive, Dropbox⦠rclone must be onPATH(or setPIPELINK_RCLONE), and credentials come from your rclone config. - Cache.
source.cacheif set, else$PIPELINK_CACHE, else~/.cache/pipelink. The layout mirrors the URI (<cache>/<remote>/<path>or<cache>/<host>/<path>), and URIs containing..are rejected. Two projects that pin the same file share one cached copy. - Integrity. Downloads go to a temporary file and are renamed into place only after the sha256 matches, so a failed or wrong download never replaces a good cached file. After verification, pipelink writes a stamp (
.<name>.pipelink: hash, size, mtime) so later pulls of multi-GB files skip re-hashing while the file is untouched.--verifyignores stamps. - Unpinned entries. If
sha256is omitted,pullstill downloads, then prints the hash with a warning so you can pin it. - link / validate. A remote entry counts as present only once it is cached. Neither command touches the network, and
linkreports uncached entries as missing with a hint to runpull. - Single files only. Archive a directory (
tar) before pushing it.
metadata (required)
| Field | Type | Description |
|---|---|---|
type |
string | "file", "files", "directory", or "remote" |
description |
string | Optional human-readable description |
generated_by |
string[] | Optional list of scripts that produce the source |
license |
string | Optional; informational, e.g. for licensed remote data |
source (required)
| Field | Type | Description |
|---|---|---|
directory |
string | Absolute path to the source directory |
file |
string or string[] | Filename(s) within the directory. Omit for type = "directory" |
task |
string | Optional path prefix prepended to directory |
uri |
string | remote only: rclone remote:path or https:// URL |
sha256 |
string | remote only: expected hash of the file (strongly recommended) |
cache |
string | remote only: cache root override (default $PIPELINK_CACHE or ~/.cache/pipelink) |
target (required)
| Field | Type | Description |
|---|---|---|
directory |
string | Path to the target directory (relative to working directory, or absolute) |
file |
string or string[] | Target filename(s). Defaults to source filenames (for remote, the basename of uri) if omitted |
Pipelink prints colored output showing each link with Unicode arrows:
π Processing ... input.toml ... for linking π
4 files to process
Linking POPULATION (multiple files)
County population estimates
Target: βββΆ input/Census/county_population.parquet
Source: βββ data/Census/Population/county_population.parquet
Target: βββΆ input/Census/county_demographics.parquet
Source: βββ data/Census/Population/county_demographics.parquet
β 4 links created
Missing source files are filtered out with a warning and the remaining links are still created.
Use pipelink as a rule in your Snakemake pipeline to establish data dependencies before running analysis:
rule link_inputs:
input:
config="input.toml",
output:
touch(".links_created"),
shell:
"pipelink link {input.config} && touch {output}"Pipelink pairs well with Nickel for type-safe, validated configuration. Instead of writing TOML by hand, define links in a .ncl file with contracts that enforce correct structure, then export to TOML.
A typical input.ncl in a project directory:
let link_contracts = import "../utilities/config/nickel/link_contracts.ncl" in
let
Link = link_contracts.link,
serialize_records = link_contracts.serialize_records
in
{
POPULATION | Link = 'files {
source = {
file = ["county_population.parquet",
"county_demographics.parquet"],
directory = "/data/Census/Population",
},
target = { directory = "./input/Census" },
metadata = {
generated_by = ["import_census.R"],
description = "County population estimates",
},
},
STATION_TEMPS | Link = 'file {
source = {
file = "station_temps_2025.csv",
directory = "/data/NOAA",
},
target = { directory = "./input/Weather" },
},
TIGER_LINES | Link = 'files {
source = {
file = ["tl_counties.shp", "tl_states.shp"],
directory = "/data/Census/ShapeFiles",
},
target = { directory = "./input/ShapeFiles" },
},
}
|> serialize_recordsThe Link contract validates each entry as one of three enum variants ('file, 'files, 'dir), and serialize_records flattens the structure into the TOML schema pipelink expects. Target filenames default to source filenames when omitted.
Export to TOML and link in one step:
nickel export input.ncl --format toml > tmp/input.toml
pipelink link tmp/input.tomlOr as a Snakemake rule:
rule link_inputs:
input:
config="input.ncl",
output:
toml="tmp/input.toml",
stamp=touch(".links_created"),
shell:
"""
nickel export {input.config} --format toml > {output.toml}
pipelink link {output.toml}
"""