Minion task progress tracking improvements

[shounakmk219]

Description

This PR tries to decouple the way how minion task progress is being tracked and stored so that we can reuse the implementations of MinionEventObserver and decide the underlying storage to be used based on the requirements.

This PR also adds some extra stats to track as part of the observer so that user can get more granular information about the minion task execution.

Introduce new MinionTaskProgressManager interface

The responsibility of storing the task progress data that's being monitored by MinionEventObserver will be delegated to this new interface MinionTaskProgressManager. It will expose the below methods to manage the MinionTaskProgressStats for each task run.

• init(PinotConfiguration configuration)
• getTaskProgress(String taskId)
• setTaskProgress(String taskId, MinionTaskProgressStats progress)
• deleteTaskProgress(String taskId)

Configuration

In order to plug the desired implementation of MinionTaskProgressManager there is a new config introduced

pinot.minion.taskProgressManager.class

If the config is not provided or minion is unable to load the instance of that class, the default implementation DefaultMinionTaskProgressManager will be used.
This is the same logic of how the existing progress is being maintained in minion memory.

Track more stats with MinionTaskProgressStats

The info maintained by MinionEventObserver in current implementation is not enough to get the granular view on a task's progress. A new wrapper class MinionTaskProgressStats is introduced to track and manage more stats.

Stats tracked right now

• Task start time
• Current task state (from MinionTaskState)
• Progress logs (timestamp, plain string log)

New stats added

• Stage (in addition to the MinionTaskState, track task type specific stages/states )
• Time spent on each stage
• startTime
• endTime
• progress log (timestamp, stage, log level, plain string log )
• input units (segments, files etc)
• #input units processed
• #generated segments

New interface methods on MinionEventObserver

Few new interface methods are added to MinionEventObserver along with the above improvements

• MinionTaskProgressStats getProgressStats() : to get all the stats for the task being tracked
• cleanup() : cleanup code which will be called when the observer is purged upon reaching the ttl threshold

Onboard MergeRollupTask

To try out the improvements, the MergeRollupTaskExecutor flow has been updated to enable tracking all the new stats. As the flow spans over SegmentProcessorFramework and SegmentMapper as well the tracking has been updated there as well

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 81.50943% with 49 lines in your changes missing coverage. Please review.

Project coverage is 63.76%. Comparing base (59551e4) to head (41ee679).
Report is 1587 commits behind head on master.

Files with missing lines	Patch %	Lines
...ava/org/apache/pinot/minion/BaseMinionStarter.java	0.00%	15 Missing ⚠️
...che/pinot/minion/event/MinionProgressObserver.java	87.87%	4 Missing and 8 partials ⚠️
.../core/segment/processing/mapper/SegmentMapper.java	22.22%	7 Missing ⚠️
...inion/event/BaseMinionProgressObserverFactory.java	0.00%	5 Missing ⚠️
...not/minion/event/EventObserverFactoryRegistry.java	0.00%	4 Missing ⚠️
...n/java/org/apache/pinot/spi/tasks/StatusEntry.java	92.30%	1 Missing and 1 partial ⚠️
...pinot/minion/event/DefaultMinionEventObserver.java	66.66%	1 Missing ⚠️
...inion/event/DefaultMinionEventObserverFactory.java	0.00%	1 Missing ⚠️
...minion/event/DefaultMinionTaskProgressManager.java	95.83%	1 Missing ⚠️
...apache/pinot/minion/event/MinionEventObserver.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14521      +/-   ##
============================================
+ Coverage     61.75%   63.76%   +2.01%     
- Complexity      207     1612    +1405     
============================================
  Files          2436     2711     +275     
  Lines        133233   151427   +18194     
  Branches      20636    23365    +2729     
============================================
+ Hits          82274    96558   +14284     
- Misses        44911    47624    +2713     
- Partials       6048     7245    +1197     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (+99.99%)	⬆️
integration	100.00% <ø> (+99.99%)	⬆️
integration1	100.00% <ø> (+99.99%)	⬆️
integration2	0.00% <ø> (ø)
java-11	56.27% <47.00%> (-5.44%)	⬇️
java-21	63.66% <81.50%> (+2.03%)	⬆️
skip-bytebuffers-false	63.75% <81.50%> (+2.00%)	⬆️
skip-bytebuffers-true	63.64% <81.50%> (+35.91%)	⬆️
temurin	63.76% <81.50%> (+2.01%)	⬆️
unittests	63.76% <81.50%> (+2.01%)	⬆️
unittests1	56.31% <47.00%> (+9.41%)	⬆️
unittests2	34.09% <72.45%> (+6.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.