Refactor parsing of results files #176

wtbarnes · 2024-11-04T21:34:06Z

Fixes #86

This pull requests does a few things though is mainly focused on refactoring the parsing of the results files to try and simplify things:

Move all file parsing logic to standalone functions in pydrad/parse/util.py that return astropy.table.QTable objects rather than arrays
Properties on Strand are only dynamically generated if there are many of them or if they are unknown ahead of time (e.g. for the .ine files). Otherwise they are explicitly listed on Profile.
Quantities are now pulled out of the resulting tables by column name rather than relying on indexing an array. This should be more robust.
All results files are now optional except for the .amr files. If the .phy files are not read (or not present), the thermodynamic quantities are computed from the conserved quantities in the .amr files. See Function for calculating thermodynamic variables from conserved variables #86.

TODOs:

Docstrings for quantities on Profile
Create issue to track case where .amr file has an additional column for the electron mass density

codecov · 2024-11-04T21:36:10Z

Codecov Report

Attention: Patch coverage is 75.90361% with 40 lines in your changes missing coverage. Please review.

Project coverage is 71.30%. Comparing base (557e41a) to head (43bfc03).

Files with missing lines	Patch %	Lines
pydrad/parse/util.py	60.00%	18 Missing ⚠️
pydrad/parse/parse.py	88.28%	13 Missing ⚠️
pydrad/configure/configure.py	10.00%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #176      +/-   ##
==========================================
+ Coverage   70.41%   71.30%   +0.89%     
==========================================
  Files          11       12       +1     
  Lines         730      798      +68     
==========================================
+ Hits          514      569      +55     
- Misses        216      229      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

… arrays

jwreep · 2024-11-08T00:17:52Z

pydrad/parse/util.py

+    """
+    Parse the ``.hstate`` files containing the hydrogen level populations as a function of position
+    """
+    columns = [f'level_population_hydrogen_{i}' for i in range(1,7)]


level_population_hydrogen_3 is not really understandable at first glance. Level 6 is also not the 6th level, it's the ionized fraction.

Is there a better naming scheme we could use?

Also, perhaps adding a "neutral fraction" property, which is just the sum of levels 1 through 5, might be helpful. It's sometimes helpful to consider ions vs neutrals in the chromosphere.

I'm open to any better naming scheme! I think I probably named these somewhat arbitrarily as I didn't know what else to call them.

Also 👍 on a derived neutral fraction property. I would suggest that be its own PR.

Perhaps just simplifying hydrogen_level_i for the first five, and ionized_fraction for the last. I'm not sure if there's a much better name without abbreviations, but alternatively maybe HI_level_i and HII_fraction. Not sure.

jwreep · 2024-11-08T00:20:55Z

pydrad/configure/configure.py

@@ -415,7 +425,8 @@ def maximum_cells(self):
        refinement level and :math:`n_{min}` is the minimum allowed number of
        grid cells. Optionally, if the maximum number of cells is specified
        in ``config['grid']['maximum_cells']``, this value will take
-        precedence.
+        precedence. In general, it is better to set this value explicitly to a sufficiently
+        large value.


What is sufficiently large? Perhaps 1 Mm per cell at the minimum? I think a simple example number would be useful as a comment.

The default configuration now explicitly uses a value of 30000 which is what the HYDRAD GUI uses. Beyond that, I don't have a sense of what sufficiently large is. For low refinement levels (e.g. 4), the $2^Ln_{min}$ rule of thumb does not seem to hold.

pydrad/parse/parse.py

jwreep · 2024-11-08T00:26:01Z

pydrad/parse/parse.py

        """
-        Parse the hydrodynamics results file
+        Parse the adaptive mesh grid file


Suggested change

Parse the adaptive mesh grid file

Parse the adaptive mesh refinement (.AMR) file

jwreep · 2024-11-08T00:26:57Z

pydrad/parse/parse.py

        """
-        Parse the adaptive mesh grid file
+        Parse the hydrodynamics results file


Suggested change

Parse the hydrodynamics results file

Parse the physical variables (.PHY) file

Nitpicking just to clarify what the file extension means

jwreep · 2024-11-08T00:28:20Z

pydrad/parse/parse.py

-                self._grid_centers[i] = tmp[0]
-                self._grid_widths[i] = tmp[1]
+        if not self._phy_filename.is_file():
+            log.warning(f'{self._phy_filename} not found. Skipping parsing of phy files. Set read_phy=False to suppress this warning.')


Suggested change

log.warning(f'{self._phy_filename} not found. Skipping parsing of phy files. Set read_phy=False to suppress this warning.')

log.warning(f'{self._phy_filename} not found. Skipping parsing of .phy files. Set read_phy=False to suppress this warning.')

jwreep · 2024-11-08T00:47:19Z

pydrad/parse/parse.py

+    def electron_pressure(self) -> u.Unit('dyne cm-2'):
+        if hasattr(self, '_phy_data'):
+            return self._phy_data['electron_pressure']
+        gamma = 5/3


Maybe this shouldn't be hard coded? I think it's also found in the constants.h file. I don't think it would ever be updated, so probably not worrisome.

Agreed. I was mildly annoyed that this isn't already defined in astropy.constants. We could create a constants file and stick it in there.

I'd say astropy should add it, but the truth is that this isn't a constant, so much as a value specific to monatomic gas.

A constants file is probably a reasonable way to deal with this.

jwreep · 2024-11-08T00:48:02Z

pydrad/parse/parse.py

+    def ion_pressure(self) -> u.Unit('dyne cm-2'):
+        if hasattr(self, '_phy_data'):
+            return self._phy_data['ion_pressure']
+        gamma = 5/3


Ditto with my previous comment. Also, we're setting the same variable twice, and they should not differ between electrons/ions.

jwreep · 2024-11-08T00:48:35Z

pydrad/parse/parse.py

+
+    @property
+    @u.quantity_input
+    def ion_pressure(self) -> u.Unit('dyne cm-2'):


Perhaps worth adding a total pressure variable = P_e + P_H?

jwreep · 2024-11-08T00:51:34Z

pydrad/parse/util.py

+
+def read_ine_file(filename, n_s):
+    """
+    Parse ``.ine`` files containing ionization fraction as a function of position


Similar to the comments above, I wonder if it's worth just clarifying what the file extensions mean. For example,

Suggested change

Parse ``.ine`` files containing ionization fraction as a function of position

Parse Ionization Non-Equilibrium ``.ine`` files containing ionization fraction as a function of position

jwreep · 2024-11-08T00:53:02Z

pydrad/parse/util.py

+        lines = f.readlines()
+    # First parse all of the population fraction arrays
+    # NOTE: Have to calculate the number of elements we have
+    # computed population fractions for as we do not necessarily


This could be pulled from ./Radiation_Model/config/elements_neq.cfg, rather than calculated. Not sure if that's faster/slower, but it is an alternative option.

True. As I commented above, I've tried to make this only dependent on the results files themselves such that you can parse results even if all of the usual HYDRAD files aren't there. Reading from the neq file seems just as onerous as having to specify the number of coordinates.

wtbarnes marked this pull request as draft November 4, 2024 22:09

wtbarnes added 3 commits November 4, 2024 23:26

refactor poly fit interface

c76558b

refactor fitting methods for gravity and B fields

8c95b41

refactor trm parsing

ba82ec1

wtbarnes force-pushed the minor-refactoring branch from 91f7f3c to ba82ec1 Compare November 5, 2024 04:26

wtbarnes added 3 commits November 6, 2024 17:29

move all file parsing to external functions, return tables instead of…

014ced4

… arrays

fix amr parsing for variable columns

fa84101

add smoke test for conserved quantity calculation

43bfc03

wtbarnes changed the title ~~Minor refactoring of parsing and configuration~~ Refactor parsing of results files Nov 7, 2024

wtbarnes requested a review from jwreep November 7, 2024 20:22

wtbarnes marked this pull request as ready for review November 7, 2024 20:22

jwreep reviewed Nov 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor parsing of results files #176

Refactor parsing of results files #176

wtbarnes commented Nov 4, 2024 •

edited

Loading

codecov bot commented Nov 4, 2024 •

edited

Loading

jwreep Nov 8, 2024

jwreep Nov 8, 2024

wtbarnes Nov 8, 2024

jwreep Nov 8, 2024

jwreep Nov 8, 2024

wtbarnes Nov 8, 2024

jwreep Nov 8, 2024

jwreep Nov 8, 2024

jwreep Nov 8, 2024

jwreep Nov 8, 2024

wtbarnes Nov 8, 2024

jwreep Nov 8, 2024

jwreep Nov 8, 2024

jwreep Nov 8, 2024

jwreep Nov 8, 2024

jwreep Nov 8, 2024

wtbarnes Nov 8, 2024

	Parse the adaptive mesh grid file
	Parse the adaptive mesh refinement (.AMR) file

	Parse the hydrodynamics results file
	Parse the physical variables (.PHY) file

	log.warning(f'{self._phy_filename} not found. Skipping parsing of phy files. Set read_phy=False to suppress this warning.')
	log.warning(f'{self._phy_filename} not found. Skipping parsing of .phy files. Set read_phy=False to suppress this warning.')

	Parse ``.ine`` files containing ionization fraction as a function of position
	Parse Ionization Non-Equilibrium ``.ine`` files containing ionization fraction as a function of position

Refactor parsing of results files #176

Are you sure you want to change the base?

Refactor parsing of results files #176

Conversation

wtbarnes commented Nov 4, 2024 • edited Loading

codecov bot commented Nov 4, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wtbarnes commented Nov 4, 2024 •

edited

Loading

codecov bot commented Nov 4, 2024 •

edited

Loading