s5cmd

Command-line tool for S3 and local filesystem operations, running transfers in parallel with wildcard support and batch command files.

install

s5cmd is a command-line tool written in Go for working with S3 and S3-compatible object storage, as well as the local filesystem. It supports listing buckets and objects, uploading, downloading, deleting, copying, moving and renaming objects, creating and removing buckets, printing object contents to stdout, and summarizing object sizes grouped by storage class. It can also select JSON records from objects using SQL expressions, and it works with Google Cloud Storage and other S3 API compatible services.

Its worker pool runs transfers concurrently. Wildcards work across S3 operations: s5cmd lists objects by the prefix up to the first wildcard, filters the results in memory, and runs the operation on each match in parallel (for example, s5cmd cp 's3://bucket/logs/2020/03/*' .). Deletes use the S3 batch delete API, so up to 1000 matching objects go in a single request. A command file mode (s5cmd run) executes many commands in batch.

Other documented features include --include and --exclude filters (where exclude takes precedence and flag order doesn't matter), --flatten to collapse the source directory structure, pipe to stream stdin to S3, server-side encryption with AWS KMS, ACL settings, S3 Transfer Acceleration, dry run, structured logging, and shell auto-completion. Credentials can come from --profile, --credentials-file, environment variables, EC2 or EKS IAM roles, or --no-sign-request for public buckets. Region is detected from flags, AWS_REGION, the AWS profile, or the bucket itself.

It suits developers, DevOps engineers and data engineers who move large numbers of files between local disks and object storage. Copying S3 objects larger than 5GB server-side is not yet supported.

Installation

# via Homebrew
brew install peak/tap/s5cmd
# via Go
go install github.com/peak/s5cmd/v2@master
# via Docker
docker pull peakcom/s5cmd

s5cmd usage output

Alternatives & related tools

add a badge to your README

Maintain s5cmd? Show that it's featured on trmn. The badge links back to this page.

Featured on trmn
Markdown
[![Featured on trmn](https://www.trmn.sh/badge/s5cmd.svg)](https://www.trmn.sh/tools/s5cmd?utm_source=badge)
HTML
<a href="https://www.trmn.sh/tools/s5cmd?utm_source=badge"><img src="https://www.trmn.sh/badge/s5cmd.svg" alt="Featured on trmn" height="20"></a>