Store everest results in ERT storage

[yngve-sk]

Issue
Resolves #8811

Base idea/documentation:

Store datasets by [batch, realization, perturbation] x [controls, objectives, constraints, objective_gradient, constraint_gradient]:

Exhaustive list of data stored PER BATCH :

• batch.json - contains info about the batch, batch_id and whether it is an improvement (aka merit flag, but the concepts are now unified for dakota and non-dakota runs)
• batch_constraints constraint values (and violations) for constraints, batch-wide
• batch_objectives objective values, batch-wide
• realization_controls - control values for geo-realizations, also includes simulation_id
• realization_objectives - objective values per geo-realization
• realization_constraints - constraint values per geo-realization
• perturbation_objectives - objective and control values per perturbation
• perturbation_constraints - constraint and control values per perturbation (Note/discussion point: control values could be pulled into separate table to avoid redundancy)
• batch_objective_gradient - Partial derivatives of objectives, given different controls. This dataset has one column per objective, and one row per control value, and the intersecting cells represent the partial derivative of the objective wrt that control value.
• batch_constraint_gradient - Partial derivatives of constraints, given different controls. This dataset has one column per constraint, and one row per control value, and the intersecting cells represent the partial derivative of the constraint wrt that control value.

Example data from math_func/config_advanced.yml (json format)

Exhaustive list of data stored PER OPTIMIZATION

• controls.json - control values for this batch
• realization_weights.json - realization weights
• nonlinear_constraints - conditions for constraints to satisfy (on average over the batch)
• objective_functions - objective function names, weights, and normalization

Example data from math_func/config_advanced.yml

Potential simplifications

The everest_data_api is currently used for plotting, but could be used (probably expanded a bit) to avoid doing direct (polars) dataframe manipulations elsewhere in the code, but currently they are done directly in the code.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 98.49624% with 6 lines in your changes missing coverage. Please review.

Project coverage is 91.89%. Comparing base (c4217b1) to head (a8af382).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
src/everest/everest_storage.py	98.49%	5 Missing ⚠️
src/everest/api/everest_data_api.py	97.56%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9161      +/-   ##
==========================================
+ Coverage   91.74%   91.89%   +0.15%     
==========================================
  Files         426      426              
  Lines       26538    26647     +109     
==========================================
+ Hits        24346    24488     +142     
+ Misses       2192     2159      -33     

Flag	Coverage Δ
cli-tests	39.80% <0.00%> (+0.14%)	⬆️
everest-models-test	34.58% <56.14%> (+0.42%)	⬆️
gui-tests	74.41% <0.00%> (+0.09%)	⬆️
integration-test	38.28% <85.96%> (+0.29%)	⬆️
performance-tests	51.60% <0.00%> (+0.01%)	⬆️
test	39.26% <97.24%> (-0.26%)	⬇️
unit-tests	74.04% <25.81%> (-0.20%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.