Benchmark shell commands with hyperfine

Tools used

• hyperfine

  Command-line benchmarking tool that runs statistical analysis across multiple timing runs of shell commands

  29.2kRust
• 

  fd

  Fast filesystem search tool that replaces find with regex patterns and smart defaults

  44.9kRust
• 

  jq

  Command-line JSON processor for slicing, filtering, mapping and transforming structured data

  36.0kC

Before you start

• -Linux, macOS, BSD or Windows
• -The commands you want to measure, working from the current directory

Steps

1. 1Step 1: Install hyperfine

   hyperfine is packaged for most systems. Distribution packages can be a few releases behind; the GitHub releases page has current binaries and .deb packages.

   sh

   # macOS or Linux with Homebrew
   $ brew install hyperfine
   
   # Debian / Ubuntu
   $ sudo apt install hyperfine
   
   # Fedora
   $ sudo dnf install hyperfine
   
   # Arch Linux
   $ sudo pacman -S hyperfine
   
   # Windows
   $ winget install hyperfine
   
   # From source
   $ cargo install --locked hyperfine

   sh

   $ hyperfine --version

2. 2Step 2: Run a basic benchmark

   Quote each command. hyperfine does at least 10 runs and picks a run count that takes about 3 seconds in total. --runs sets an exact count; --min-runs and --max-runs set bounds.

   The result shows the mean and standard deviation, user and system time, and the min and max range.

   sh

   $ hyperfine 'sleep 0.3'

   sh

   $ hyperfine --runs 20 'make -s test'

3. 3Step 3: Control caches with --warmup and --prepare

   Commands that read files run faster once the OS has cached them. --warmup N runs the command N times before measuring, so every timed run sees a warm cache. --prepare runs a command before each timed run, which you can use to drop the cache and measure cold runs instead.

   On Linux, dropping the page cache needs root. Run sudo -v first so the prepare command does not stop at a password prompt.

   sh

   $ hyperfine --warmup 3 'grep -R TODO .'

   sh

   $ sudo -v
   $ hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'grep -R TODO .'

4. 4Step 4: Compare commands

   Pass several commands and hyperfine benchmarks each in turn, then prints a summary of how many times faster the fastest one is. -n gives each command a short name for the output. fd skips hidden and .gitignored files by default, so -HI makes it walk the same files as find.

   hyperfine 1.19+ has --reference CMD to compare everything against a baseline you choose instead of the fastest command.

   sh

   $ hyperfine --warmup 3 \
     -n find "find . -name '*.md'" \
     -n fd 'fd -HI -e md'

   sh

   $ hyperfine --reference 'git status' 'git status --untracked-files=no'

5. 5Step 5: Scan a parameter with -P or -L

   -P NAME MIN MAX runs the command once per integer in the range and replaces {NAME} in the command. -D sets a different step size, including decimals. -L NAME a,b,c takes a list of values instead. -L can be given several times to benchmark every combination.

   sh

   $ hyperfine --prepare 'make clean' -P threads 1 8 'make -j {threads}'

   sh

   $ hyperfine -P delay 0.3 0.7 -D 0.2 'sleep {delay}'

   sh

   $ hyperfine -L compiler gcc,clang -L opt O0,O2 '{compiler} -{opt} main.c -o /dev/null'

6. 6Step 6: Export the results

   --export-markdown writes a table you can paste into a README or pull request. --export-json writes every result, including the time of each individual run, for further analysis. --export-csv, --export-asciidoc and --export-orgmode are also available, and several can be used at once. Times in the JSON file are in seconds.

   sh

   $ hyperfine --warmup 3 --export-markdown bench.md --export-json bench.json \
     "find . -name '*.md'" 'fd -HI -e md'

   sh

   $ jq -r '.results[] | "\(.command)\t\(.mean)\t\(.stddev)"' bench.json

7. 7Step 7: Measure very fast commands with -N

   By default hyperfine runs each command through a shell (sh on Unix, cmd.exe on Windows), measures the shell's startup time, and subtracts it. For commands that finish in under about 5 ms, that correction is noisy compared with the command itself. -N (hyperfine 1.13+, same as --shell=none) runs the command directly with no shell. Without a shell, globs, ~, pipes, ; and variables do not work.

   --shell (-S) picks a different shell when the command needs its syntax.

   sh

   $ hyperfine -N --warmup 10 'git rev-parse HEAD'

   sh

   $ hyperfine --shell zsh 'for i in {1..10000}; do echo test; done'