Command-line duplicate file finder and remover with advanced filtering and deduplication options
fclones is a command-line utility for identifying and removing duplicate files with a focus on performance and flexibility. It provides separate commands for finding duplicates (group) and removing them (remove, link, move, dedupe), allowing users to inspect results before making filesystem changes. The tool supports various selection criteria including file size filters, pattern matching with globs and regex, and can process files from multiple directory roots or standard input.
The tool implements several performance optimizations including parallel processing, automatic SSD/HDD detection for access strategy tuning, and prefix compression for reduced memory usage. On Linux, it orders file operations by physical placement on HDDs and uses page-cache-friendly access patterns. It supports multiple hash functions from fast non-cryptographic to 512-bit cryptographic hashes, with optional persistent caching.
fclones offers multiple deduplication strategies beyond simple removal, including soft/hard link replacement and copy-on-write reflinks on supported filesystems. It outputs results in various formats (text, CSV, JSON) and integrates well with Unix utilities through stdin processing. The tool includes safety features like dry-run mode and detailed progress reporting, making it suitable for managing large file collections where data integrity is critical.
# via Snap
snap install fclones
# via Homebrew
brew install fclones
# via Cargo
cargo install fclones