fclones

fclones is a command-line utility for identifying and removing duplicate files across directory trees. It operates in two phases: the group command finds identical files and outputs them in various formats (text, JSON, CSV), while separate commands like remove, link, and dedupe handle cleanup operations. The tool supports multiple deduplication strategies including file removal, hard/soft linking, and copy-on-write reflinks on supported filesystems.

The tool includes extensive filtering capabilities for file selection, supporting glob patterns, regular expressions, size constraints, and recursion depth limits. It can work with piped input from other utilities or process files directly from stdin. Priority options allow users to specify which files to preserve based on modification time, creation time, or directory nesting level.

Built in Rust with performance optimization as a core focus, fclones employs parallel processing for I/O operations, adapts access patterns based on storage type (SSD vs HDD), and uses path compression to handle millions of files efficiently. It includes features like progress reporting, dry-run mode for preview operations, and hash caching for improved performance on subsequent runs. The tool integrates well with Unix utilities and provides machine-readable output formats for automation workflows.

Installation

# via Homebrew
brew install fclones

# via Snap
snap install fclones

# via Cargo
cargo install fclones