21_07_22

Meeting Follow-up

https://pad.unistra.fr/p/AStroPyVo_follow-up

Goal of working with annotated data

This section is directly issued from the Vizier use-case. We suppose that we have to process with Astropy legacy data, e.g. a Vizier query responses, annotated with model components taken from MCT or PhotDM models. The expectations for the final can be summarized as follow:

• Facilitate the Astropy objects building
• Taking the best of most of the meta-data
• Make easier the identification/selection of the available quantities

Model components vs AStropy components

The table below shows up a rough equivalence between the VO models and the Astropy objects (See MCD note to get more details)

MCT	VO Model	Mango Support	AstropY
Position	MCT	yes	SkyCoord
RadialVelocity	MCT	yes	SkyCoord
Velocity	MCT	yes	SkyCoord
Time	MCT	yes	Time
GenericMeasure	MCT	yes	Quantity
Polarization	MCT	yes	Quantity
Photometry	PhotDM	yes	Quantity
PhotCal	PhotDM	yes	-
Flag/Status	Mango	yes	-

Reminder: The MANGO proposal is just model wrapping model classes (MCT, PhotDM, Mango specific ones) in open-ended docks.

• designed to be mappable on legacy data
• support quantity grouping

Important remarks:

• Most of the quantities issued from the VO models have Astropy counterparts.
• The coordinates system objects are not part of the Astropy API, they are embedded in the corresponding quantities.
• 2 lacks however:
  • Photometric calibrations (filter description)
  • flags and status

Missing Astropy items

• VO model instances may come with meta-data that are not in the Astropy classes (reduction status, UCD, vocabulary...).
• MCT measures come with error, which is not the case for Astropy
• Astropy does not support data grouping
• No way to iterate over the list of available measures

A path toward a model based API

• This API should reuse as much as possible Astropy classes
• It should avoid as much as possible using affiliated classes (should it?)
• It must extend the Astropy classes for the quantities that are not supported yet
• Astropy classes must be extended to support, missing metadata, errors and data grouping.

Which feature to add to Astropy objects

1. Just add missing classes
2. Extend the missing metadata to all classes (both astropy and DM natives)
3. Add the error support to all classes (both astropy and DM natives)
4. Add the data grouping support to all classes (both astropy and DM natives)
5. Add a measure selector

option	major argument	yes	no
1	gain too low		X
2	Missing Astropy feature: can be added by class derivation	Y
3	Missing Astropy feature: can be added by class derivation	Y
4	Would require all ASytropy quantity classes to have one common super-class		X
5	useful to get to row content in term of model	X

Which development strategy

1. Enabling Astropy to both extract mapping blocks and process them
2. Enabling Astropy to extract mapping blocks and writing another package, possibly affiliated, to process them
3. Writing another package, possibly affiliated, to both extract mapping blocks and process them
4. Writing an Astropy MR for extracting the mapping block and an Pyvo package withe the model classes

option	major argument	yes	no
1	Too much modification in Astropy to be accepted		X
2	Extra package won't be used because it will be out of reach for the regular users		X
3	The mapping block extraction must be part of Astropy		X
4	Solution conform with the AStrPyVO framework, greater development freedom with PyVO	X

Meeting notes

Attendees - 6 Laurent Michel LM, Renaud Savalle RS, Tom Donaldson TD, Omar Laurino OL, Mark CD MCD, Jesus Salgado JS

Meeting notes

LM: Presentation of a brief introduction on the subject, PyVO-DM The proposed model facilitate the way to map DMs into VOTables and how to read these VOTables in a efficient way in python using astropy (and PyVO) In the client side, there are two things to do with this annotation: extract the mapping block adn and process this mapping block

Also, write the python instances that map the data model instances (e.g position, photometry, etc)

The focus of today is design and astropy API compliant with the native API and supports working with models

As a result of the hackathon, pages under https://github.com/ivoa/modelinstanceinvot-code/wiki were created

Examples created by Vizier (Gilles Landais) described at https://github.com/ivoa/modelinstanceinvot-code/wiki/vizier-proposal as they have experience on the difficulties of mapping dm elements into vizier tables

Gilles propose to iterate on measures on the resources and for each one obtain the values (there is another examople for spectral data that has not been explored further)

There is an interpretation of this pseudocode into pyvo code including a query to the service, requesting dm_mapping. That could allow to created mangoinstances from the response and obtain a modelview using modelviewer

That allows the navigation of the result in terms of mangoObjects. It can be also iterated on the object parameters to get the values of the labels

Example of the parsing of a PhotometryFilter using the model description

MCD: the sentence containsMangoInstances could be false for, e.g. cubes or time series (the object is fully returned)

OL: how is this compared with the library RAMA?

TD: we need to have client code to navigate the vo/dml

LM: the example showed is based on mango but the idea is to create astropy classes to do the parsing. The process has been divided in three phases 1 parse VO/Table 2 work with the parameters inside classes 3 create classes

OL: Why you did not start with the library RAMA as most of the requirements are already implemented there?

LM: it was started the exercise as RAMA was not compatible with the mapping syntax

OL: yes, but this is a detail that could be implemented as other things are already there

The main target for today is to discuss about the model components vs astropy components at https://github.com/ivoa/modelinstanceinvot-code/wiki/21_07_22

MCD: if the idea is to create an skycoord instance from an annotation, the first thing is to study the constructor

TD: it is an overloaded constructor and "humans" knowing the location of the columns create the instances using a simple constructor version There is a affiliated package called SpecUtils that could be explored We just draft some ideas on the aspect of the API during the last interop. We could check RAMA and I have some ideas on how to handle this in astropy

LM: Table with the correspondance between the model instances and astropy types (in the case of polarization, photometry, etc it would be astropy quantity)

For cases like PhotCal and Flag, there is not a clear correspondance with astropy Also there is not clear link with magnitudes and error, it does not support data grouping and there is not a way to iterate over the list of available measures

MCD: it could be used an array for iteration LM: but associated values are difficult to link

The path is to reuse API astropy classes, prevent the use of affiliated classes, extend astropy for quantities, extend to support, missing metadata, errors and data grouping

TD: we should not be just limited to astropy classes as many metadata is not included into the astropy classes and it should be available for clients I think astropy core team will resist to include all these new classes but what we can do is upgrade the VOTable parser ensuring that we can load all the required metadata PyVO is the natural place to put these extended classes

LM: so the idea would be to extend astropy adding these classes to pyVO