Parquet Row Groups Decide Whether Your Filters Skip Anything

Your query filters on one day and returns 200 rows. The engine reads 40 GB. The table is Parquet, the engine pushes the filter down, and the scan still touches every file you own. Nothing is broken: Parquet row groups only let an engine skip data when the layout cooperates.

Most pipelines write a layout that guarantees it cannot. Fixing that costs a sort, and it is usually worth more than any compression setting you could change.

Key takeaways#

• Parquet nests files into row groups, then column chunks, then pages.
• Skipping works on min/max statistics held per row group and per page.
• On unsorted data every row group spans the whole range. So nothing is skipped.
• The spec recommends row groups of 512 MB to 1 GB. Most tools default far lower.
• Sorting by the filter column matters more than any compression setting.

How Parquet row groups are laid out#

Parquet row groups sit in the middle of a four-level hierarchy, and the names matter because tuning happens at each one.

A file contains one or more row groups. A row group holds a horizontal slice of the table, and it contains one column chunk per column. Each column chunk contains one or more pages, which the spec calls the indivisible unit of compression and encoding.

The metadata sits at the end. A Parquet file finishes with the file metadata, then a four-byte little-endian length, then the magic bytes PAR1. Writing the footer last is what allows single-pass writing, since the writer does not know the offsets until the data is written.

That layout is also why a reader opens a Parquet file by seeking to the end first. It reads the footer, learns where every column chunk lives, and then fetches only what it needs.

What predicate pushdown actually needs#

Pushdown is the engine asking "can I skip this chunk without reading it". The answer comes from statistics.

Parquet stores min and max values at several levels, including the column chunk, the column index and the data page. So for a filter like WHERE event_date = '2026-09-03', the engine walks the footer, compares your predicate against each row group's min and max for event_date, and skips any group whose range cannot contain a match.

That is the whole mechanism. It is not an index, and it does not know which rows match. It only knows whether a match is possible.

Which leads to the failure everyone hits.

Why unsorted data destroys skipping#

Suppose you write a year of events, unsorted, into 100 row groups. Each row group receives a random sample of the year.

Every row group's event_date min is close to January and its max is close to December. Your one-day filter falls inside all 100 ranges. Therefore the engine skips nothing and reads the entire table, having faithfully performed predicate pushdown.

Now sort by event_date before writing. Each row group covers roughly three or four days. Your filter matches one or two groups, and the engine skips the other 98 on the strength of a few bytes of metadata.

Same file format, same compression, same predicate. The only difference is the order in which you wrote the rows.

This is the same idea as a database index, where field order decides whether a query can use it. The ESR rule for compound indexes is the relational cousin of this problem, and the intuition transfers directly: the physical order of your data determines which questions are cheap.

Choosing what to sort by#

You get one physical order per file, so it has to earn its place.

Sort by the column that appears in the most selective, most frequent filter. For event data that is almost always time. For a multi-tenant table it may be the tenant id, or tenant id then time.

Partitioning and sorting solve different halves. Partitioning by date puts each day in its own directory, so the engine prunes at the file level before opening anything. Sorting within the file lets it prune row groups after that. Use both: partition on the coarse dimension, sort on the fine one.

Be careful with partition granularity. Partitioning by hour on a small dataset produces thousands of tiny files, and that costs more in metadata and request overhead than it saves in pruning.

Sizing Parquet row groups, and why defaults disappoint#

The Parquet specification recommends large row groups, 512 MB to 1 GB, so that a group fits in a single HDFS block and permits larger sequential reads. Its reasoning is explicit: larger row groups allow larger column chunks, which makes larger sequential I/O possible.

Most engines default well below that. So a pipeline that writes without configuring anything produces row groups far smaller than the format assumes.

Small row groups are not automatically wrong. They give finer skipping granularity, which helps a highly selective filter. However, they multiply metadata, and the reader must parse the whole footer before touching a single value. Thousands of tiny row groups mean a large footer and a slow open.

The page size recommendation is 8 KB, and the trade sits the other way. Smaller pages allow more fine-grained reading for single-row lookups, while larger pages reduce space overhead from page headers and cut parsing cost.

The small files problem#

Sizing Parquet row groups has a bigger sibling. A streaming job writing every minute produces 1,440 files a day, each a few megabytes, each with its own footer.

The cost is per-file, not per-byte. Every file needs a metadata read, and on object storage every one of those is a request with real latency. A query over a month then opens 43,000 files before reading a single value.

Compaction is the fix, and it is ordinary maintenance rather than a clever optimisation. Rewrite yesterday's small files into a handful of large ones, sorted, once a day. Most table formats now do this on a schedule.

The tell is simple. If your query time scales with the number of files rather than the volume of data, you have this problem and no amount of row group tuning will address it.

Sorting also shrinks the file#

There is a second benefit that rarely gets mentioned, and on wide tables it can matter more than the skipping.

Parquet encodes each column chunk independently, and two of its encodings reward repetition directly. Dictionary encoding replaces repeated values with small integer references. Run-length encoding collapses consecutive identical values into a value and a count.

Sorting creates exactly the runs those encodings are built for. An unsorted status column alternates between a handful of values at random, so runs are one or two rows long. Sort by status, or by any column correlated with it, and the same data becomes long uninterrupted stretches.

The effect compounds across columns. Sorting by tenant_id clusters not just that column but everything functionally dependent on it, such as region, plan tier and account age. Each of those compresses better as a side effect of a sort you did for a different reason.

So the sort you choose for pushdown often pays twice, in bytes scanned and in bytes stored. That makes the write-time shuffle easier to justify than it first appears.

Bloom filters cover the case sorting cannot#

Sorting handles one column well. But equality filters on a high-cardinality column you did not sort by, such as a user id or a session token, get nothing from min/max, because every group's range spans almost the entire key space.

Parquet added Bloom filters for exactly this. A Bloom filter answers "is this value definitely absent" with certainty, and "possibly present" otherwise. False positives cost you a read you did not need. False negatives never happen, so correctness holds.

The format uses split block Bloom filters. Each one is a set of 256-bit blocks of eight 32-bit words, hashed with XXH64, using multiplication rather than modulo to pick bits. Writers place the filter data either after all row groups or between them, and ColumnMetaData records the offset so a reader can find it without scanning.

The documentation frames the trade against dictionary encoding directly. Dictionaries work well at low cardinality, and writers often skip them when distinct values grow too numerous, because the dictionary itself gets expensive. Bloom filters give you pushdown on those columns while using less space.

So the two mechanisms divide cleanly. Use sort order for range and time filters, and a Bloom filter for equality on a high-cardinality column. Enable them per column rather than globally, since a filter on a column nobody queries is pure overhead.

One limit worth knowing. Bloom filters answer equality only. A range predicate cannot use them at all, which is why sort order remains the primary tool.

Where this advice is wrong#

Four cases where sorting and large row groups are not the answer.

Point lookups by primary key. Parquet is a scan format. If you fetch single rows by id, you want a key-value store or an indexed OLTP table, and no layout tuning makes columnar storage good at this.

Small tables. Under a few hundred megabytes, the whole thing fits in memory after one read. Sorting buys nothing measurable, and the pipeline complexity is real.

High-cardinality filters with no dominant column. If queries filter on twenty different columns with equal frequency, one sort order cannot serve them. Consider a format with real indexes, or accept the scan.

Write-heavy, read-rare data. Sorting costs a shuffle at write time. For an archive someone queries monthly, paying that on every write is the wrong trade. Sort during compaction instead.

Detecting the problem#

Read the metadata rather than guessing. Every Parquet library exposes it, and the numbers tell you immediately whether skipping can work.

import pyarrow.parquet as pq
 
f = pq.ParquetFile("events.parquet")
print("row groups:", f.num_row_groups)
 
for i in range(min(f.num_row_groups, 5)):
    rg = f.metadata.row_group(i)
    col = rg.column(0)                      # the column you filter on
    print(i, rg.num_rows, col.statistics.min, col.statistics.max)

Look at the min and max across Parquet row groups. If consecutive row groups show near-identical ranges, your data is unsorted and pushdown is doing nothing. If the ranges step forward cleanly, skipping will work.

Two more checks are worth running. Compare bytes read against table size in your engine's query profile, since a filter that reads everything is the symptom in its purest form. And confirm statistics exist at all, because some writers omit them for certain types.

One caveat from the spec itself. Statistics rely on defined sort orders, and a few types are ambiguous — NaN handling for floating point, and the legacy INT96 timestamp. Filters on those columns may not skip reliably even when the layout is right.

A short checklist#

1. Sort by your most common selective filter column before writing.
2. Partition on the coarse dimension, sort on the fine one.
3. Target row groups in the hundreds of megabytes rather than the single digits.
4. Compact small files on a schedule.
5. Inspect row group statistics to confirm the ranges actually step.
6. Avoid filtering on float or INT96 columns where skipping matters.

The short version#

Predicate pushdown is not a property of Parquet. It is a property of your layout, and Parquet only supplies the statistics that make it possible. On unsorted data every row group spans the full range, so the engine dutifully checks each one and skips none. Sort by the column you filter on, keep row groups substantial, compact small files, and verify by reading the footer rather than trusting that the format is doing the work.

Sources: Apache Parquet format specification