Parquet V3 predicate pushdown to S3/Azure object storage ?

[ovidiubuligan]

Both S3 and azure blob storage support a features like Predicate Pushdown And Projection pushdown which drastically reduces the ammount of data to be read from parquet files. http://peter-hoffmann.com/2020/understand-predicate-pushdown-on-rowgroup-level-in-parquet-with-pyarrow-and-python.html .

Are these predicate pushdowns considered ? How hard would It be to implement in tempo ?

[joe-elliott]

So we do implement predicate pushdown but it is unrelated to our backend storage and works in all versions of Parquet.

The engine makes a fetch request here:

tempo/pkg/traceql/engine.go

Line 37 in 0542bc3

expr.Pipeline.extractConditions(req)

Each element of the AST is responsible for building its own conditions. Here is where the binary operator will create conditions and push them down to the fetch layer:

tempo/pkg/traceql/ast_conditions.go

Line 34 in 0542bc3

func (o BinaryOperation) extractConditions(request *FetchSpansRequest) {

In the fetch layer if an operation is present we will build an iterator that asserts the operation as close to the data as possible. If no operation is present we return the data all the way back to the engine and let it decide how to handle the situation.

tempo/tempodb/encoding/vparquet3/block_traceql.go

Line 1478 in 0542bc3

for _, cond := range conditions {

[ovidiubuligan]

Thank you for the insight , I actually wanted to link to an article about S3 Select that can actually do the filtering at the S3 level.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html

Searching further It may not necessarily always be better as the issue opened In ClickHouse more than 2 years ago with no interest:
ClickHouse/ClickHouse#19275

You can close the discussion.