If you have downloaded a text dataset in the last couple of years it probably arrived as a directory of .parquet files rather than one enormous JSONL. That is not fashion. Columnar storage changes what you can do without reading the whole thing.
Columnar means you can read one field
A row-oriented file stores record one in full, then record two in full. To read a single field from a million records you must touch every byte of all million records. A columnar file stores all the values of field one together, then all of field two.
For training data this is the whole argument. A dataset with a text field, a url, a language, a quality score and three provenance fields may be 90% text by volume. Counting the language distribution, or filtering to the rows above a quality threshold, reads only those small columns — often a few percent of the file. The same operation on JSONL parses every token of every document.
Row groups, and why file layout matters
Inside a Parquet file, rows are batched into row groups — typically tens or hundreds of megabytes each. Every row group stores per-column statistics: minimum, maximum, null count.
A reader can use those statistics to skip entire row groups without decompressing them. Filter for quality_score > 0.8 and any row group whose maximum score is 0.6 is discarded on the strength of its footer alone. This is predicate pushdown, and it is why a well-laid-out Parquet dataset answers a selective query in seconds.
It also explains the main way people make Parquet slow: a dataset written as thousands of tiny files, or with one row group per file, gives the reader nothing to skip and adds per-file overhead to every scan. Aim for files in the hundreds of megabytes, and sort on the column you filter by most so the statistics are selective.
| JSONL | Parquet | |
|---|---|---|
| Read one field | parse every record | read that column only |
| Filter by a value | parse every record | skip row groups by statistics |
| Typical size on disk | baseline | often 3–10× smaller |
| Append a record | trivial | rewrite a file |
| Human-readable | yes | no, needs a reader |
| Streaming row by row | natural | works, batched |
Reading it without loading it
The pattern worth learning is: scan cheap columns to decide, then fetch the expensive column for the rows you kept. Both pyarrow and the datasets library support this directly, and polars makes the lazy version particularly readable.
import pyarrow.dataset as ds
d = ds.dataset("corpus/", format="parquet") # a directory of files
print(d.schema) # no data read yet
# 1. survey using small columns only
langs = d.to_table(columns=["language"]).column("language").value_counts()
# 2. push the filter down, then pull text only for surviving rows
sub = d.to_table(
columns=["text", "language", "quality"],
filter=(ds.field("language") == "fr") & (ds.field("quality") > 0.8),
)
print(f"{sub.num_rows:,} rows kept")
# 3. hand it to training without materialising a python list
for batch in sub.to_batches(max_chunksize=1000):
texts = batch.column("text").to_pylist()
... # tokenise, train
When Parquet is the wrong answer
It is not universally better. Parquet is immutable in practice — appending means rewriting a file — so a dataset you are still collecting into is more comfortable as JSONL until it settles. It is binary, so you cannot inspect it with head or diff it meaningfully in version control.
And it is a poor container for large binary blobs. You can put image bytes in a column, and plenty of datasets do, but for heavy image and video training the ecosystem leans towards sharded record formats such as WebDataset tar shards, which stream sequentially and keep each sample's files together. The rule of thumb: Parquet for tabular and text, sharded records for large media, JSONL while the data is still moving.
Questions people ask
Does Parquet compress better than gzipped JSONL?
Usually yes, and for a different reason. Columnar layout puts similar values next to each other, so dictionary and run-length encoding work far better before the general-purpose compressor even runs. A field with few distinct values costs almost nothing. The gain is largest on repetitive metadata and smallest on free text, which is already high-entropy.
Which compression codec should I use?
Snappy is the common default and is fast to decompress, which matters when you are reading in a training loop. Zstandard gives noticeably smaller files at a modest decompression cost and has become the usual choice for datasets meant for distribution. Gzip is slower to read than either and worth avoiding for training data.
Can I stream a Parquet dataset I have not downloaded?
Yes. Because the footer carries the schema and row-group statistics, a reader can fetch the footer over HTTP range requests, decide which row groups it needs, and fetch only those byte ranges. This is how remote dataset streaming works without a local copy.
How do I pick a row group size?
Large enough that per-group overhead is irrelevant, small enough that a reader can skip usefully and does not have to buffer too much. Values in the low hundreds of megabytes are a reasonable default. If you filter heavily on one column, sorting by it before writing does more for query speed than tuning the size.