Ingester: Validate completed blocks #4256
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
joe-elliott
requested review from
mdisibio,
mapno,
yvrhdn,
zalegrala,
electron0zero,
ie-pham,
stoewer and
javiermolinar
as code owners
Signed-off-by: Joe Elliott <[email protected]>
Signed-off-by: Joe Elliott <[email protected]>
Signed-off-by: Joe Elliott <[email protected]>
joe-elliott
force-pushed
the
validate-completed-blocks
branch
from
7b01ac7
to
3a0f62c
Compare
Signed-off-by: Joe Elliott <[email protected]>
Signed-off-by: Joe Elliott <[email protected]>
|
|
||
| // Validate will do a basic sanity check of the state of the parquet file. This can be extended to do more checks in the future. | ||
| // This method should lean towards being cost effective over complete. | ||
| func (b *backendBlock) Validate(ctx context.Context) error { |
There was a problem hiding this comment.
I wonder if it would not be simpler trying to open the file directly with parquet-go:
o := []parquet.FileOption{
parquet.SkipBloomFilters(true),
parquet.SkipPageIndex(true),
parquet.FileSchema(parquetSchema),
parquet.FileReadMode(parquet.ReadModeAsync),
}
reader := NewBackendReaderAt(ctx, b.r, DataFileName, b.meta)
_, err := parquet.OpenFile(reader, int64(b.meta.Size_), o...)
if err != nil {
return fmt.Errorf("failed to read parquet fike: %w", err)
}OpenFile seems to do essentially the same plus some other validations. Is this difference noticeable enough? Since this is at startup time maybe it isn't.
There was a problem hiding this comment.
OpenFile also unmarshals the thrift footer which is more costly. I was trying to avoid paying those allocs on startup.
zalegrala
reviewed
Nov 6, 2024
| // Validate will do a basic sanity check of the state of the parquet file. This can be extended to do more checks in the future. | ||
| // This method should lean towards being cost effective over complete. | ||
| func (b *backendBlock) Validate(ctx context.Context) error { | ||
| if b.meta == nil { |
There was a problem hiding this comment.
Is there a way to reassemble this information from data on disk and still avoid paying the allocs on startup like you mention above?
There was a problem hiding this comment.
possibly? what do you mean by "this information"?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Adds a
Validate()method to backend blocks and uses it in ingester startup to do basic confirmation of block health before reloading them. This prevents corrupt blocks from being loaded and eventually pushed to object storage.Which issue(s) this PR fixes:
Fixes #4166 as near as we can tell. At least it covers the instance seen internally.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]