A fast file deduplicator
-
Updated
Oct 10, 2025 - Rust
#
file-deduplication
Here are 35 public repositories matching this topic...
Yet Another Dupes Finder
-
Updated
Oct 8, 2026 - Rust
Smash through to find duplicate files super fast by slicing files intelligently!
go windows macos linux cli freebsd duplicate-files deduplication cli-tool file-deduplication file-deduplication-tool
-
Updated
Jan 30, 2026 - Go
.NET 10 API for document file format identification, text/metadata/attachment/embedded object/sensitive item (PII/PHI/FERPA)/entity extraction.
metadata sdk csharp dotnet email text extraction text-extraction archive pst embedded-objects dlp file-format-detection entity-extraction phi file-deduplication pii microsoft-office pii-detection file-identification
-
Updated
Aug 7, 2026 - C#
Case study using dotfurther's Open Discover Platform with the RavenDB document store to rapidly create a full-text search/eDiscovery/information governance capable demonstration application.
metadata text-extraction full-text full-text-search ravendb ediscovery indexing-engine file-format-detection data-breach file-deduplication pii information-governance-catalog personally-identifiable-information archive-extractor pii-detection file-identification full-text-extraction document-ingestion information-governance
-
Updated
May 28, 2024
fast, parallel duplicate file detection
fast command-line file-deduplication statically-linked duplicate-file-finder duplicate-file-detection
-
Updated
Nov 18, 2025 - C
A safe, fast, and controllable duplicate file manager for the command line. Not just a duplicate finder. It is a complete workflow for locating, reviewing, and cleaning duplicate files — with byte-level accuracy, reversible actions, multi-stage detection, and full user control at every step.
rust cli duplicate-files file-management duplicate-detection file-deduplication blake3 smartlegionlab
-
Updated
Sep 30, 2026 - Rust
File deduplicator
-
Updated
Apr 27, 2022 - Rust
a small C++ CLI for finding duplicate files (namely, a deduplicator..) using size, hashing, and byte-by-byte comparison!! this really helped me find, like, 23 files (turned out to be the same powerpoint presentation that i spammed because it wasn't showing in the downloads..)
cli c-plus-plus cplusplus cpp command-line filesystem file-management systems-programming file-deduplication cpp23 duplicate-file-finder
-
Updated
Sep 18, 2026 - C++
Duplicate file finder and de-duplicator. A tool that detects duplicate files and replaces them with symlinks to a shared file in a special shared files folder.
-
Updated
Mar 28, 2026 - C#
A simple command line utility that identifies groups of identical files and displays them to the console.
-
Updated
Mar 5, 2025 - Rust
CloneZapper is a Python script that hunts down identical files within a directory and its subdirectories, ruthlessly eliminating them!
-
Updated
Jun 1, 2024 - Python
Этот проект представляет собой мощный инструмент для поиска и анализа дублирующихся файлов в указанной директории. Программа позволяет эффективно выявлять одинаковые файлы на основе их содержимого, используя алгоритм хеширования SHA-256. Она поддерживает настройку параметров, таких как минимальный размер файла для проверки и игнорирование определен
python hashing productivity multithreading data-deduplication file-system sha256 file-management system-utility cli-tool dev-tools file-deduplication file-comparison disk-cleanup command-line-utility duplicate-file-finder
-
Updated
Feb 14, 2025 - Python
Production-oriented file deduplication and data-protection platform with tenant isolation, SHA-256 exact deduplication, MinHash/LSH and dHash near-duplicate detection, DLP, AES-256-GCM encryption, RBAC, and audit logging.
python mongodb data-deduplication minhash rbac aes-gcm sha256 locality-sensitive-hashing perceptual-hashing multi-tenancy dlp data-protection file-storage file-deduplication dhash data-loss-prevention fastapi near-duplicate-detection duplicate-file-detection
-
Updated
Sep 13, 2026 - JavaScript
A Node.js CLI tool that recursively scans directories, detects duplicate files by content using hashing, and safely removes redundant copies with dry-run support.
nodejs hashing automation filesystem developer-tools cli-tool file-deduplication duplicate-file-finder
-
Updated
Sep 17, 2026 - JavaScript
A corpus-hygiene utility for RAG data pipelines that identifies duplicate content risk, quantifies duplication with actionable statistics, and supports controlled remediation before indexing. It enables staged audit-then-cull workflows that improve retrieval quality, reduce embedding/indexing cost, and strengthen governance in knowledge curation.
bash dotnet knowledge-management data-quality file-deduplication data-curation data-governance rag document-deduplication retrieval-augmented-generation corpus-hygiene
-
Updated
May 5, 2026 - Shell
Golang Deduplication Utility Library
-
Updated
Dec 29, 2017 - Go
Find duplicate files in multiple folder(s) scanning .txt or/and .torrent files and depending on the selected mode (readonly: true | false) get information about duplicated files /+ extract them into new folders
nodejs cli typescript functional-programming file-scanner inquirer fp-ts file-deduplication eslint-plugin-functional inquirer-fuzzy-path
-
Updated
Oct 8, 2026 - TypeScript
Add this topic to your repo
To associate your repository with the file-deduplication topic, visit your repo's landing page and select "manage topics."