Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: don't create a BytesIO stream but upload a Table directly #181

Closed
bsipocz opened this issue Nov 2, 2024 · 2 comments · Fixed by #182
Closed

BUG: don't create a BytesIO stream but upload a Table directly #181

bsipocz opened this issue Nov 2, 2024 · 2 comments · Fixed by #182
Labels
bug Something isn't working

Comments

@bsipocz
Copy link
Contributor

bsipocz commented Nov 2, 2024

In catalog_query we do some hackery, but in fact the upload can be an astropy table.

This snippet produces a failure, as is, after the underlying logic in pyvo has been cleaned (astropy/pyvo#614), but after debugging and digging around a bit I'm convinced this is an issue with the notebook.

OTOH, if we also want to enable an IO stream to be an upload table, we could certainly try that enhancement.

## In memory only, use an IO stream.
vot_obj=io.BytesIO()
apvot.writeto(apvot.from_table(mytable),vot_obj)
## (Reset the "file-like" object to the beginning.)
vot_obj.seek(0)
query="""SELECT mt.ra, mt.dec, cat.ra, cat.dec, cat.Radial_Velocity, cat.morph_type, cat.bmag
    FROM zcat cat, tap_upload.mytable mt
    WHERE
    contains(point('ICRS',cat.ra,cat.dec),circle('ICRS',mt.ra,mt.dec,mt.angDdeg))=1
    and cat.Radial_Velocity > 0 and cat.radial_velocity != mt.radial_velocity
    ORDER by cat.ra"""
#  Currently broken due to a bug.
#mytable2 = heasarc.service.run_async(query, uploads={'mytable':vot_obj})
mytable2 = heasarc.search(query, uploads={'mytable':vot_obj})
vot_obj.close()
mytable2.to_table()

with traceback:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File ~/munka/devel/pyvo/pyvo/dal/query.py:258, in DALQuery.execute_votable(self, post)
    257 try:
--> 258     return votableparse(self.execute_stream(post=post).read)
    259 except Exception as e:

File ~/munka/devel/pyvo/pyvo/dal/tap.py:1101, in TAPQuery.execute_stream(self, post)
   1098     raise DALServiceError(
   1099         "Cannot execute a non-synchronous query. Use submit instead")
-> 1101 return super().execute_stream(post=post)

File ~/munka/devel/pyvo/pyvo/utils/decorators.py:9, in stream_decode_content.<locals>.wrapper(*args, **kwargs)
      7 @wraps(func)
      8 def wrapper(*args, **kwargs):
----> 9     raw = func(*args, **kwargs)
     10     raw.read = partial(raw.read, decode_content=True)

File ~/munka/devel/pyvo/pyvo/dal/query.py:207, in DALQuery.execute_stream(self, post)
    201 """
    202 Submit the query and return the raw response as a file stream.
    203 
    204 No exceptions are raised here because non-2xx responses might still
    205 contain payload. They can be raised later by calling ``raise_if_error``
    206 """
--> 207 response = self.submit(post=post)
    209 try:

File ~/munka/devel/pyvo/pyvo/dal/tap.py:1129, in TAPQuery.submit(self, post)
   1126 url = self.queryurl
   1128 files = {
-> 1129     upload.name: upload.fileobj()
   1130     for upload in self._uploads
   1131     if upload.is_inline
   1132 }
   1134 response = self._session.post(
   1135     url, data=self, stream=True, files=files)

File ~/munka/devel/pyvo/pyvo/dal/query.py:1053, in Upload.fileobj(self)
   1051     return fileobj
-> 1053 fileobj = open(self._content)
   1055 return fileobj

TypeError: expected str, bytes or os.PathLike object, not BytesIO

During handling of the above exception, another exception occurred:

DALFormatError                            Traceback (most recent call last)
Cell In[26], line 14
      6 query="""SELECT mt.ra, mt.dec, cat.ra, cat.dec, cat.Radial_Velocity, cat.morph_type, cat.bmag
      7     FROM zcat cat, tap_upload.mytable mt
      8     WHERE
      9     contains(point('ICRS',cat.ra,cat.dec),circle('ICRS',mt.ra,mt.dec,mt.angDdeg))=1
     10     and cat.Radial_Velocity > 0 and cat.radial_velocity != mt.radial_velocity
     11     ORDER by cat.ra"""
     12 #  Currently broken due to a bug.
     13 #mytable2 = heasarc.service.run_async(query, uploads={'mytable':vot_obj})
---> 14 mytable2 = heasarc.search(query, uploads={'mytable':vot_obj})
     15 vot_obj.close()
     16 mytable2.to_table()

File ~/munka/devel/pyvo/pyvo/registry/regtap.py:963, in RegistryResource.search(self, *args, **keys)
    934 """
    935 assuming this resource refers to a searchable service, execute a
    936 search against the resource.  This is equivalent to:
   (...)
    960    if the resource does not describe a searchable service.
    961 """
    962 try:
--> 963     return self.service.search(*args, **keys)
    964 except ValueError:
    965     # I blindly assume the ValueError comes out of get_interface.
    966     # But then that's likely enough.
    967     raise dalq.DALServiceError(
    968         f"Resource {self.ivoid} is not a searchable service")

File ~/munka/devel/pyvo/pyvo/dal/tap.py:278, in TAPService.run_sync(self, query, language, maxrec, uploads, **keywords)
    249 def run_sync(
    250         self, query, *, language="ADQL", maxrec=None, uploads=None,
    251         **keywords):
    252     """
    253     runs sync query and returns its result
    254 
   (...)
    274     TAPResults
    275     """
    276     return self.create_query(
    277         query, language=language, maxrec=maxrec, uploads=uploads,
--> 278         **keywords).execute()

File ~/munka/devel/pyvo/pyvo/dal/tap.py:1117, in TAPQuery.execute(self)
   1103 def execute(self):
   1104     """
   1105     submit the query and return the results as a TAPResults instance
   1106 
   (...)
   1115        for errors parsing the VOTable response
   1116     """
-> 1117     return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)

File ~/munka/devel/pyvo/pyvo/dal/query.py:261, in DALQuery.execute_votable(self, post)
    259 except Exception as e:
    260     self.raise_if_error()
--> 261     raise DALFormatError(e, self.queryurl)

DALFormatError: TypeError: expected str, bytes or os.PathLike object, not BytesIO
@bsipocz bsipocz added the bug Something isn't working label Nov 2, 2024
@bsipocz
Copy link
Contributor Author

bsipocz commented Nov 2, 2024

The fix for the traceback is in pyvo in astropy/pyvo#617, and #182 should deal with the cleanup.

@bsipocz bsipocz linked a pull request Nov 2, 2024 that will close this issue
@bsipocz
Copy link
Contributor Author

bsipocz commented Dec 3, 2024

I'm closing this as the bug has been fixed upstream. I would think #182 can still go ahead as a cleanup though.

@bsipocz bsipocz closed this as completed Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant