Add --tombstone-probability and --compacted
#60
Conversation
Allows the `producer_worker` and `repeater_worker` to generate tombstone records.
| @@ -108,7 +109,10 @@ func (pw *ProducerWorker) newRecord(producerId int, sequence int64) *kgo.Record | |||
| pw.Status.AbortedTransactionMessages += 1 | |||
| } | |||
|
|
|||
| payload := make([]byte, pw.config.messageSize) | |||
| payload := pw.config.valueGenerator.Generate() | |||
There was a problem hiding this comment.
Not sure why valueGenerator wasn't used here before instead of an empty make([]byte)?
There was a problem hiding this comment.
Please make a PR to redpanda and check that it works as expect by pointing kgo at this commit sha before merging this PR. Otherwise if we merge and detect a problem we get a "head of line blocking" on making concurrent changes to kgo.
Suggestion: For testing compacted topics I think we can do more. kgo-verifier should keep track of the keys and and the end checkpoint the expected set of keys and associated values (last values produced for a key minus tombstones). Then during consumption we can track the same and compare at the end.
Or maybe we have a test like this already?
| @@ -108,7 +109,10 @@ func (pw *ProducerWorker) newRecord(producerId int, sequence int64) *kgo.Record | |||
| pw.Status.AbortedTransactionMessages += 1 | |||
| } | |||
|
|
|||
| payload := make([]byte, pw.config.messageSize) | |||
| payload := pw.config.valueGenerator.Generate() | |||
| @@ -53,7 +53,7 @@ type ValidatorStatus struct { | |||
| lastLeaderEpoch map[int32]int32 | |||
| } | |||
|
|
|||
| func (cs *ValidatorStatus) ValidateRecord(r *kgo.Record, validRanges *TopicOffsetRanges) { | |||
| func (cs *ValidatorStatus) ValidateRecord(r *kgo.Record, validRanges *TopicOffsetRanges, compacted bool) { | |||
There was a problem hiding this comment.
shall this be a field on the ValidatorStatus struct instead? it doesn't change for the lifetime entire lifetime anyway why pass it as a param then?
Ack, will do with an upcoming PR.
Yes, this is possible. Will push some follow up commits to this PR. |
Offset gaps exist in compacted topics. Suppress a warning log in the read workers if the user passes a `--compacted` flag as a parameter.
For verification of compacted topics, we can track the last expected key-value pair that will be seen after the log has been fully compacted.
As a means to verify the results along with `LatestValueProduced` for a compacted topic.
WillemKauf
force-pushed
the
tombstones
branch
from
b13a42a
to
28a4a05
Compare
|
Force push to:
|
See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters, and `LatestValueProduced`/`LatestValueConsumed` as return values in producer/worker status structs.
See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters, and `LatestValueProduced`/`LatestValueConsumed` as return values in producer/worker status structs.
See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters, and `LatestValueProduced`/`LatestValueConsumed` as return values in producer/worker status structs.
Adds two flags to
kgo-repeaterandkgo-verifier:--tombstone-probability, which allows for random production of tombstone records per the input parameter.--compacted, which indicates that the topic to be verified is compacted. A warning about gaps in consumed offsets will not be shown if this istrue.