Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OLI API Flash update #1296

Merged
merged 44 commits into from
Mar 22, 2024
Merged

OLI API Flash update #1296

merged 44 commits into from
Mar 22, 2024

Conversation

veccp
Copy link
Contributor

@veccp veccp commented Feb 8, 2024

Fixes/Resolves:

This addresses some issues of #1279, dealing mainly with the Flash and OLIApi classes and adding test coverage for new functionality.

E.g.: move more general functions out of Flash class, rework Client and Flash frontend, improve compatibility with async #1303...

Summary/Motivation:

To allow for other flash calculations and frameworks to be used with OLI Cloud, some streamlining of functions was necessary.

Changes proposed in this PR:

  • Streamline data extraction functions
  • Enable Aqueous (H+ ion) thermo framework (required for corrosion analyzer)
  • Enable XSC private databank (required for surface complexation calculations, silica scaling)
  • Modify OLI stream output data storage: storing samples as array (no repetition of JSON keys for each sample)
  • Apply same modification method to all survey variables (either additive - "relative" or substitutive - "absolute")
  • Add JSON writing function
  • Update OLI API tutorial incorporating_oli_calculations
  • Clarify that Water Analysis is not required if you already have apparent species
  • Consistency and clarity improvements for some doc strings

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the license terms described in the LICENSE.txt file at the top level of this directory.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

@veccp veccp added the oli label Feb 8, 2024
@veccp veccp self-assigned this Feb 8, 2024
@ksbeattie ksbeattie added the Priority:Normal Normal Priority Issue or PR label Feb 8, 2024
Copy link

codecov bot commented Feb 8, 2024

Codecov Report

Attention: Patch coverage is 64.17657% with 211 lines in your changes are missing coverage. Please review.

Project coverage is 94.70%. Comparing base (32af858) to head (6bd9855).

Files Patch % Lines
watertap/tools/oli_api/flash.py 57.97% 174 Missing ⚠️
watertap/tools/oli_api/client.py 78.26% 35 Missing ⚠️
...s/oli_api/util/watertap_to_oli_helper_functions.py 81.81% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1296      +/-   ##
==========================================
+ Coverage   94.25%   94.70%   +0.45%     
==========================================
  Files         370      370              
  Lines       37950    38229     +279     
==========================================
+ Hits        35768    36204     +436     
+ Misses       2182     2025     -157     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@bknueven bknueven left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial comments.

watertap/tools/oli_api/client.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/client.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/client.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/client.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/client.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/flash.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/flash.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/flash.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/flash.py Outdated Show resolved Hide resolved
watertap/tools/oli_api/flash.py Outdated Show resolved Hide resolved
Comment on lines 320 to 326
" # if a DBS file has been retained from a previous session,\n",
" # its flash history and chemistry information can be summarized.\n",
" file_summary = oliapi.get_dbs_file_summary(dbs_file_id)\n",
"\n",
" # save chemistry information\n",
" chemistry_info = file_summary[\"chemistry_info\"]\n",
" write_output(chemistry_info[\"result\"], \"chemistry_info\")\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bit on getting the summary/flash history should be shown later, after walking through the main essentials.

@lbianchi-lbl
Copy link
Contributor

@lbianchi-lbl Fix (expected) failures in notebooks tests.

@lbianchi-lbl
Copy link
Contributor

lbianchi-lbl commented Mar 7, 2024

The OLI API CI notebooks job is failing seemingly because of `interactive_mode=True); maybe some change has been inadvertently reverted in the notebook file? EDIT It seems so, judging from b982cbd

image

@lbianchi-lbl
Copy link
Contributor

@adam-a-a For reference, this is how the OLI API CI checks look like:

image

@lbianchi-lbl
Copy link
Contributor

lbianchi-lbl commented Mar 8, 2024

@lbianchi-lbl Fix (expected) failures in notebooks tests.

This is done in bfcb306, which adds functionality to skip/xfail OLI notebooks if the credentials are not found in the environment:

image

@lbianchi-lbl
Copy link
Contributor

OK, I think I managed to get the Codecov upload to work:

image
image
image

@lbianchi-lbl
Copy link
Contributor

The OLI API CI jobs now also create a "pending" status check when the workflow is started:
image

@ksbeattie ksbeattie added Priority:High High Priority Issue or PR and removed Priority:Normal Normal Priority Issue or PR labels Mar 14, 2024
@adam-a-a adam-a-a self-requested a review March 19, 2024 19:21
"optionalProperties": dict(self.optional_properties),
"unitSetInfo": dict(self.output_unit_set),
}
if included_solids and excluded_solids:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why both cannot be specified at the same time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per OLI:

params.excludedSolids and params.includedSolids cannot be specified at the same time.

param
]
elif param in modified_clone["params"]["inflows"]["values"]:
d = d["waterAnalysisInputs"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did you remove the temperature/pressure catch here? If it is removed, then the relative/absolute value boolean for the survey definitely needs to be multi-dimensional.

If a user wants to sweep across temperature (more likely to be absolute) and concentration (more likely to be relative), how would this work? If T/P are handled explicitly as previously, we could maybe avoid that issue. As it is now, we definitely need to deal with it.

Copy link
Contributor Author

@veccp veccp Mar 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense - I'll make note to reimplement the catch for T/P and also pH

Copy link
Contributor

@adam-a-a adam-a-a left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - we will resolve any outstanding issues in subsequent PRs.

I want to reiterate that before we continue adding any new or altered functionality (or API changes in general), we should prioritize adding tests to cover the wide range of functionality introduced in this and previous PRs.

@lbianchi-lbl lbianchi-lbl merged commit 69481fc into watertap-org:main Mar 22, 2024
26 checks passed
@veccp veccp deleted the minor_oliapi_fixes branch April 24, 2024 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
oli Priority:High High Priority Issue or PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants