DM-46563: Ensure that arrow tables with null values return masked numpy array

[timj]

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes
• (if changing dimensions.yaml) make a copy of dimensions.yaml in configs/old_dimensions

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.69%. Comparing base (ca923c0) to head (c4e937f).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1086   +/-   ##
=======================================
  Coverage   89.69%   89.69%           
=======================================
  Files         360      360           
  Lines       47094    47110   +16     
  Branches     9693     9696    +3     
=======================================
+ Hits        42240    42256   +16     
  Misses       3480     3480           
  Partials     1374     1374           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[erykoff]

Looking at the tests

daf_butler/tests/test_parquet.py

Line 1188 in ca923c0

class ParquetFormatterArrowNumpyTestCase(unittest.TestCase):

I see that I never wrote a test to include masked columns with a numpy table. Perhaps it would be good to duplicate

daf_butler/tests/test_parquet.py

Line 1094 in ca923c0

def testWriteAstropyReadAsNumpyTable(self):

(or parametrize) with a include_masked=True call to _makeSimpleAstropTable()?

[timj]

There is definitely test code that now returns a masked numpy: testWriteReadArrowTableLossless.

It now returns:

[(0,-1.210854847584115,-0.6501103792944672,-0.7440781431041901,0.7740670343202368,0,teststring,b'teststring',2024-07-23T00:00:00.000000000,2024-07-23T00:00:00.000000,1649900760361600113,0.0,0.0,0.0,0.0,False,0,text,b'bytes'), (1,-1.3687127649177078,0.009401264757421805,-0.6416432248997462,-0.4179419528028183,10,teststring,b'teststring',2024-07-23T00:00:00.000000000,2024-07-23T00:00:00.000000,--,--,nan,--,nan,--,--,--,--), (2,-0.14690493805679222,-0.7649956326468196,-1.2992148587593215,1.5482050048469027,20,teststring,b'teststring',2024-07-23T00:00:00.000000000,2024-07-23T00:00:00.000000,2,2.0,2.0,2.0,2.0,False,2,text,b'bytes'), (3,-0.8108861108463714,-0.546826404097458,0.2956221589184724,-1.6930852478488179,30,teststring,b'teststring',2024-07-23T00:00:00.000000000,2024-07-23T00:00:00.000000,3,3.0,3.0,3.0,3.0,False,3,text,b'bytes'), (4,0.2879765811788807,-1.4030079687558348,-0.31508500206699963,0.8814814961596337,40,teststring,b'teststring',2024-07-23T00:00:00.000000000,2024-07-23T00:00:00.000000,4,4.0,4.0,4.0,4.0,False,4,text,b'bytes')]

so there are masked values for m_i8, for example. So calculating min/max for those should be enough to prove that the masked value is not included.

[erykoff]

Ah! That would also work.

[timj]

I've added a little test.

[timj]

Closed in favor or #1087 with the correct branch name.