Implement the ability to toggle dropping CLP processed field from the original generic record. #14534
Conversation
… original generic record.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #14534 +/- ##
============================================
+ Coverage 61.75% 63.99% +2.23%
- Complexity 207 1570 +1363
============================================
Files 2436 2675 +239
Lines 133233 146975 +13742
Branches 20636 22559 +1923
============================================
+ Hits 82274 94050 +11776
- Misses 44911 45970 +1059
- Partials 6048 6955 +907
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
|
||
| if (!_config.getRemoveProcessedFields()) { | ||
| to.putValue(key, value); | ||
| } |
There was a problem hiding this comment.
is this feature used for double ingestion?
There was a problem hiding this comment.
yes. if removeProcessedField is true, we won't add the original key and value to the output.
| @@ -54,6 +55,7 @@ public class CLPLogRecordExtractorConfig implements RecordExtractorConfig { | |||
| private final Set<String> _fieldsForClpEncoding = new HashSet<>(); | |||
| private String _unencodableFieldSuffix = null; | |||
| private String _unencodableFieldError = null; | |||
| private boolean _removeProcessedFields = false; | |||
There was a problem hiding this comment.
it seems the default value for removeProcessedFields is set to false. what's the current behavior, do we already by default keep the processedField e.g. message?
There was a problem hiding this comment.
The default behavior is changed. Previously, field is removed by default, now it's retained by default
There was a problem hiding this comment.
can we set removeProcessedFields=True by default to keep existing behavior. Thus, we can keep the current behavior without change tableConfig to existing tables.
There was a problem hiding this comment.
please add more comments for how to maintain the existing behavior.
There was a problem hiding this comment.
done. Added instructions on how to maintain the existing behavior via config.
deemoliu
added
backward-incompat
Allow the ability to toggle dropping CLP processed field from the original generic record inside
CLPLogRecordExtractor. One can toggle the removing the processed field via the following config:One can replicate the original behavior inside schemaConformingTransformerV2Config