zawk: AWK + stdlib + Rust

zawk is a modern AWK+ language for writing short programs processing textual data. To a first approximation, it is an implementation of the AWK language; many common Awk programs produce equivalent output when passed to zawk.

You might be interested in zawk if you want your scripts to handle escaped CSV/TSV like standard Awk fields, or if you want your scripts to execute faster, or if you want a standard AWK library to make life easy.

Features:

• CSV/TSV support
• JSON Lines (-i jsonl), Apache Parquet (-i parquet) and Markdown table (-i markdown) input
• Read input files from S3 directly: zawk -i csv '{print $1}' s3://bucket1/demo.csv
• High performance
• gawk mostly compatible
• A standard library: text, math, datetime, crypto, parser, encode/decode, ID, KV, SQLite/MySQL, Redis/NATS etc.
• i18n support: length("你好Hello") # 7, substr("你好Hello", 1, 2) # 你好
• Load awk script from URL
• Field list shorthand: a command-line program that is only a comma-separated list of two or more fields prints them, e.g. zawk '$1,$3' is zawk '{ print $1,$3 }'. A single field such as zawk '$2' keeps the standard awk meaning (print records whose $2 is non-empty/non-zero).
• awk file help support
• gawk's switch statement: switch ($1) { case 1: ...; break; case /^a/: ...; break; default: ... }
• String interpolation with f-strings: f"SELECT * FROM {table} WHERE id = {$1}" substitutes variables and fields ({name}, {NR}, {$1}, {$NF}); {{ and }} are literal braces. Expressions and format specs are not supported.

The info subdirectory has more in-depth information on zawk:

• Overview: what zawk is all about, how it differs from Awk.
• Types: A quick gloss on zawk's approach to types and type inference.
• Parallelism: An overview of zawk's parallelism support.
• Benchmarks: A sense of the relative performance of zawk and other tools when processing large CSV or TSV files.
• Standard Library: A standard library by zawk, including exciting functions that are new when compared with Awk.
• FAQ: FAQ about zawk.

zawk/frawk is dual-licensed under MIT or Apache 2.0.

Installation

Mac with Homebrew:

$ brew install --no-quarantine linux-china/tap/zawk
$ sudo xattr -r -d com.apple.quarantine $(readlink -f $(brew --prefix zawk))

or install by cargo-binstall:

$ cargo binstall zawk

You will need to install Rust. If you have not updated rust in a while, run rustup update nightly (or rustup update if building using stable).

Building a Binary

With those prerequisites, cloning this repository and a cargo build --release or cargo install --path <zawk repo path> will produce a binary that you can add to your PATH if you so choose:

$ cd <zawk repo path>
$ cargo install --path .

zawk is now on crates.io, so running RUSTFLAGS="-C target-feature=+aes,+sse2" cargo install zawk with the desired features should also work.

SIMD fast paths (SSE2/AVX2) are selected at runtime, so the default build runs on any CPU of the target architecture. To additionally optimize the rest of the code for the CPU you are building on, use:

$ RUSTFLAGS="-C target-cpu=native" cargo install zawk

The resulting binary may crash with SIGILL on other machines, so do not distribute it.

Bugs and Feature Requests

zawk has bugs, and many rough edges. If you notice a bug in zawk, filing an issue with an explanation of how to reproduce the error would be very helpful. There are no guarantees on response time or latency for a fix. No one works on zawk full-time. The same policy holds for feature requests.

Credits

Thanks to Eli Rosenthal's frawk. zawk is based on frawk. Without frawk, there would be no zawk.