Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support missing values #158

Open
dschristianson opened this issue Apr 21, 2023 · 0 comments
Open

Support missing values #158

dschristianson opened this issue Apr 21, 2023 · 0 comments
Labels

Comments

@dschristianson
Copy link
Contributor

Is your feature request related to a problem? Please describe.
basin3d doesn't support a common missing value. If a datasource's missing value is a numeric, it gets passed thru as if it is a result value.

For consideration:

  • have a basin3d missing value vocabulary
  • map missing value from data source to basin3d
  • catch and translate the datasource missing value
  • potentially: filter by missing value

Examples from USGS. Note, need to figure out what happens with non-numeric values like NaN, NA.

Using django-basin3d: ~/measurement_tvp_timeseries/?monitoring_features=USGS-09147500&observed_property_variables=RDC&start_date=2023-02-01&end_date=2023-02-10

Return: USGS missing value -999999 is converted to -28316.818683153
'''
"statistic": "MEAN",
"result_points": [
["2023-02-01T00:00:00.000",-28316.818683153],
["2023-02-02T00:00:00.000",-28316.818683153],
["2023-02-03T00:00:00.000",-28316.818683153],
["2023-02-04T00:00:00.000",-28316.818683153],
["2023-02-05T00:00:00.000",-28316.818683153],
["2023-02-06T00:00:00.000",1.5149513145],
["2023-02-07T00:00:00.000",1.5800800625999998],
["2023-02-08T00:00:00.000",-28316.818683153],
["2023-02-09T00:00:00.000",1.614060279],
["2023-02-10T00:00:00.000",-28316.818683153]
],
'''

Corresponding USGS direct query:
https://waterservices.usgs.gov/nwis/dv/?parameterCd=00060&startDT=2023-02-01&endDT=2023-02-10&sites=09147500&format=json&siteType=ST&siteStatus=all

"""
{"name":"ns1:timeSeriesResponseType","declaredType":"org.cuahsi.waterml.TimeSeriesResponseType","scope":"javax.xml.bind.JAXBElement$GlobalScope","value":{"queryInfo":{"queryURL":"http://waterservices.usgs.gov/nwis/dv/parameterCd=00060&startDT=2023-02-01&endDT=2023-02-10&sites=09147500&format=json&siteType=ST&siteStatus=all","criteria":{"locationParam":"[ALL:09147500]","variableParam":"[00060]","timeParam":{"beginDateTime":"2023-02-01T00:00:00.000","endDateTime":"2023-02-10T00:00:00.000"},"parameter":[]},"note":[{"value":"[ALL:09147500]","title":"filter:sites"},{"value":"[ST]","title":"filter:siteType"},{"value":"[mode=RANGE, modifiedSince=null] interval={INTERVAL[2023-02-01T00:00:00.000-05:00/2023-02-10T00:00:00.000-05:00]}","title":"filter:timeRange"},{"value":"methodIds=[ALL]","title":"filter:methodId"},{"value":"2023-04-21T15:38:38.319Z","title":"requestDT"},{"value":"95cb98e0-e05a-11ed-9320-2cea7f58f5ca","title":"requestId"},{"value":"Provisional data are subject to revision. Go to http://waterdata.usgs.gov/nwis/help/?provisional for more information.","title":"disclaimer"},{"value":"vaas01","title":"server"}]},"timeSeries":[{"sourceInfo":{"siteName":"UNCOMPAHGRE RIVER AT COLONA, CO","siteCode":[{"value":"09147500","network":"NWIS","agencyCode":"USGS"}],"timeZoneInfo":{"defaultTimeZone":{"zoneOffset":"-07:00","zoneAbbreviation":"MST"},"daylightSavingsTimeZone":{"zoneOffset":"-06:00","zoneAbbreviation":"MDT"},"siteUsesDaylightSavingsTime":true},"geoLocation":{"geogLocation":{"srs":"EPSG:4326","latitude":38.33143056,"longitude":-107.779225},"localSiteXY":[]},"note":[],"siteType":[],"siteProperty":[{"value":"ST","name":"siteTypeCd"},{"value":"14020006","name":"hucCd"},{"value":"08","name":"stateCd"},{"value":"08091","name":"countyCd"}]},"variable":{"variableCode":[{"value":"00060","network":"NWIS","vocabulary":"NWIS:UnitValues","variableID":45807197,"default":true}],"variableName":"Streamflow, ft³/s","variableDescription":"Discharge, cubic feet per second","valueType":"Derived Value","unit":{"unitCode":"ft3/s"},"options":{"option":[{"value":"Mean","name":"Statistic","optionCode":"00003"}]},"note":[],"noDataValue":-999999.0,"variableProperty":[],"oid":"45807197"},"values":[{"value":[{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-01T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-02T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-03T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-04T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-05T00:00:00.000"},{"value":"53.5","qualifiers":["P"],"dateTime":"2023-02-06T00:00:00.000"},{"value":"55.8","qualifiers":["P"],"dateTime":"2023-02-07T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-08T00:00:00.000"},{"value":"57.0","qualifiers":["P"],"dateTime":"2023-02-09T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-10T00:00:00.000"}],"qualifier":[{"qualifierCode":"Ice","qualifierDescription":"Value is affected by ice at the measurement site.","qualifierID":0,"network":"NWIS","vocabulary":"uv_rmk_cd"},{"qualifierCode":"P","qualifierDescription":"Provisional data subject to revision.","qualifierID":1,"network":"NWIS","vocabulary":"uv_rmk_cd"}],"qualityControlLevel":[],"method":[{"methodDescription":"","methodID":18989}],"source":[],"offset":[],"sample":[],"censorCode":[]}],"name":"USGS:09147500:00060:00003"}]},"nil":false,"globalScope":true,"typeSubstituted":false}
"""

dschristianson added a commit that referenced this issue Dec 21, 2023
vchendrix pushed a commit that referenced this issue Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants