miller vs xsv

miller vs xsv: compare their workflows, configuration choices, and practical tradeoffs.

Updated

short verdict

Miller is the starting point for new mixed-format transformations. Keep xsv in consideration for an established indexed CSV workflow while accounting for its unmaintained status.

Choose around the structure of your input and the operations you repeat. Named-field transformations across formats and indexed CSV access solve different parts of a data workflow.

this page is for

  • named-field transformations across data formats
  • indexed CSV access and column statistics
pick

miller

★ 10.1kGo
details

Use miller for named-field calculations, aggregation, and conversion across CSV, TSV, JSON, and JSON Lines.

best for

  • named-field transformations across data formats

strengths

  • Add computed fields, drop columns, sort, and aggregate records
  • Handle heterogeneous records with differing schemas
  • Most operations stream records instead of loading the entire input

weaknesses

  • Streaming applies to most operations, not every operation; choose a pipeline with the dataset size in mind.
pick

xsv

★ 10.8kRustarchived
details

Use xsv for existing CSV workflows that need indexes, column statistics, joins, or sampling.

best for

  • indexed CSV access and column statistics

strengths

  • Index a CSV file to support direct slicing
  • Compute statistics, frequency tables, joins, and regex searches
  • Use reservoir sampling to select random rows

weaknesses

  • The original project is unmaintained, which is a maintenance tradeoff for a new dependency.

Key differences

  • Data shape: miller works with named fields across CSV, TSV, JSON, and JSON Lines; xsv provides a CSV command toolkit.
  • Transformations: miller supports computed fields and aggregation; xsv provides commands for selection, joins, frequency tables, and sampling.
  • Access pattern: most miller operations stream records; xsv can index CSV input for direct slicing.
  • Dependency choice: miller fits mixed-format pipelines; an existing xsv workflow also carries the original project’s unmaintained status.

Choose miller when

  • Add computed fields, drop columns, sort, and aggregate records
  • Handle heterogeneous records with differing schemas
  • Most operations stream records instead of loading the entire input

Choose xsv when

  • Index a CSV file to support direct slicing
  • Compute statistics, frequency tables, joins, and regex searches
  • Use reservoir sampling to select random rows

FAQ

What if records do not all contain the same fields?
miller supports heterogeneous records and works with field names. That makes it relevant when log or JSON input varies from record to record.
When is an xsv index useful?
An index supports direct slicing of a CSV file, which is useful when you repeatedly need particular rows rather than processing the full file each time.
Does streaming mean every miller operation has constant memory use?
No. Most operations stream records, but memory behavior depends on the operation. Consider the dataset size and the transformations in the full pipeline rather than assuming every command keeps only one record.

related