Solver API - Expose score calculation speed?

[mcimbora]

The calculation speed is currently logged when the solving process finishes. Is it possible to expose the solver stats as a part of SolverJob interface / as a part of @PlanningSolution class (e.g. by a new @SolverStats object, or similar)?

The motivation is to gather the execution data over the longer time period / provide feedback to admin users without having to go over the log files & setting up the benchmarker.

DefaultSolver#outerSolvingEnded

logger.info("Solving ended: time spent ({}), best score ({}), score calculation speed ({}/sec), "
                + "phase total ({}), environment mode ({}), move thread count ({}).",
                solverScope.getTimeMillisSpent(),
                solverScope.getBestScore(),
                solverScope.getScoreCalculationSpeed(),
                phaseList.size(),
                environmentMode.name(),
                moveThreadCountDescription);

[triceo]

It is possible. We expose a whole bunch of metrics, score calculation speed being one of them.

[mcimbora]

Thanks for the help @triceo, this is probably it 👍 I'm having some difficulties querying the calculation speed once the solver finishes, I see the score calculation count meter is removed from the solver before the DefaultSolver#solve method returns. For me, the goal is to use some lightweight in memory storage (e.g. Atlas), just to query this piece of information & pass it further as a part of solution payload. I need to do some reading on the topic, maybe I'm just misunderstanding the concept.

                Metrics.globalRegistry.remove(new Meter.Id(SolverMetric.SCORE_CALCULATION_COUNT.getMeterId(),
                        solverScope.getMonitoringTags(),
                        null,
                        null,
                        Meter.Type.GAUGE));

[mcimbora]

Just to explain a bit using a pseudocode:

CustomMeterRegistry registry = CustomMeterRegistry.initialize

Solution solution = Solver.solve(problem)
Long calculationSpeed = registry.get(SolverMetric.SCORE_CALCULATION_COUNT)
solution.setCalculationSpeed(calculationSpeed)

solutionProcessor.process(solution)

[triceo]

This is fundamentally a monitoring API - it allows you to track the metric during the lifetime of the solver. After the solver has terminated, the metric no longer exists, I think that makes sense.

The benchmarker module uses the same metric; it hooks into it, listens to values, and stores that for its purposes. There is no dedicated logic in the benchmarker to measure score calculation speed - just a reference to the metric, which the solver exposes. Your code needs to do the same.

I understand that you would prefer to just get a simple number at the end of the solver. Although that request does make sense, at the moment I don't see ourselves creating an API for that any time soon.

[mcimbora]

Makes sense, thanks for the clarification. I'm closing the issue.

[triceo]

To maybe give a motivation for why we did it this way: using Micrometer, you can hook the solvers directly into Prometheus/Grafana, and that way your admins can track exactly what the solver is doing when. It is then the job of those systems, which are designed for monitoring, to format (and display and report) the data in any way that their users need.