Bnb/dh refactor #220

…added pytest.warns() catches for some intentional checks.

….dumps. fwp with bc in cli test. updated nc output.

…ed to string with ellipsis as a datetimeindex for the command string when time index is long.

…e too long to pass through cli and are non serializable.

…rg - can be defined right before writing.

… files

feat: default fwp_chunk_shape for None input fix: moved get chunk to after check for exisitng output file.

…ines when running on a head_node

fix: use low res to compute means / stds if features are lr only. fix: ensure correct feature order and feature divisions for dual samplers.

test: added full domain loading for rasterized h5 file

…om legacy np.random

… micro opts in loaders

…fixed cacher dims parsing

…downloader and type conversion to np.float32.

…s collection and batch handler init

…ys, which slows down the fwp.

…mputing param arrays passed to QDM() bc rex asserts that these are ndarrays and doesnt allow for da.core.Array types.

…et checks in h5 loading

…ata volume prior to node kick off.

…re then reshaped into n_obs. This is more efficient than building batches from indendent samples. This adds batch_size are an argument to sampler objects. This also allows us to remove lots of the previous batch building methods, like those using tf.data.Dataset

…ate collect test code to conftest

… level rename in cdsapi

…all available from dict and add if needed. cache loading for data handlers.

Use `rex Regridder` port

… test with cache data loading. DualSamplerCC test fix.

…ode.

@ppinchuk

-T_Dataset type removed. -Exo datahandler stuff moved to data_handlers/exo -typeagnostic rasterizer, loader, datahandler favored as base objects intead of type specific ones. -more explicit with args / kwargs in composite objects. -signature tests added for composite objects: data handler / batch handlers and https://github.com/NREL/gaps used to correctly resolve composite signatures and composite doc strings. thanks @ppinchuk! - docs branch built and checked. -cleaned up some sphinx syntax. -added some examples and explanation to Sup3rDatasets

vindex method for fancy indexing added signature check in batch factory and gaps derived docs / sigs in derived classes.

…c to Sup3rMeta meta class which all containers inherit.

…arison in stats test

…oved dim ordering logic in Sup3rX initialization. Only enforce dim order when returned from a numpy style indexing request

…mbers. Prep for replacing model step iteration with coarsening of highest res for each step. graceful batch handler thread exit.

…and then use xr.Dataset().coarsen for each model step. moved that simplified code from exo dir contents back into exo.py

…d check. fixed sphinx doc render - needed to define __name__ appropriately.

… addressed other pr comments.

…iner` data setter. sphinx_book_theme.

…map_blocks so this could be done with efficient mem use.

…e it was default without netcdf4 installed.

…h default engine + format

… can also be used to replace process pools just with scheduler='processes'

…emporal chunks for collection.

…n bump

…e cache if using hr topo in a temporal model.

…features and hr_exo_features

…e. much faster for large forward passes.

…ng now. added s_enhancements and t_enhancements to abstract models and now use this to determine exo shapes. removed temporal pad from SolarMultiStepGan, since this can be determined from the t_enhance arg and low res shape

…5 output_handlers

…anch is merged

…r modular calls

…y from parameters not file

…hould be a bool

…M to prevent large delta values

… issues with small window size so increased default to 120 (seasonal correction)

…tor formatting

… round data to zero anymore

…er data

… pass with correct output shape

…reduced mem use since number of spatial chunks tends to be much lower than number of time chunks.

… for now.

…ata time index can be shifted to start at the beginning of the day instead of at noon. GCM data frequently stamps daily data at noon instead of the beginning of the day. This caused an issue with the solar module thinking that given gan data had 48 time steps, since the time index had two unique day values, even though there were only 24 time steps from noon to noon on each day.

Gb/presrat updates 2

…ut files, with irradiance suffixes

… config file.

…ed to start at the zeroth hour, if it has not been shifted already.

…parate script if this is an issue.

…t features. now just repeat over time dimension in the forward pass padding method. this is much more performant for time independent features. this doesnt solve slow down issues with time dependent exo features like sza though.

…ndent features

…pendent

… deriver. typo in solar module.

…g. u_30m from u_10m and u_100m, with u pressure level array

…tine

… Added shape checks in ``test_access.py``

…azy load

…thout pre loading of data

…ied with .data

…dataset() which will default to a dask array manager if chunks is specified and load into memory as numpy arrays in chunks is None.

Bnb/compute on none chunks

Gb/bc debug

fixed double bias correction of features in bc qdm utility

A typo in the version of optiona dependency pydata-sphinx-theme. It should be 0.15 instead of 15.x .

Since it is an optional requirement in the package definition, let's add it as a conda source in pixi.

…for each separately.

Fix: Typos on optional dependencies related to doc

…er feature names or numpy style indices, but not both. All instances of ``ds[feature, id0, id1, ...]`` changed to ``ds[feature][id0, id1, ...]``. Features cannot be retrieved by index, only by name, and providing only dimension indices ``(ds[id0,id1,...])`` returns an array with trailing feature channel and spatiotemporal shape matching indices.

…setitem__`` which is no longer used after the index updates.

Updated Sup3rX access patterns - keys can be either features or numpy style indexing but not both

if n_writes request is too big, set to n_chunks instead of raising error

…ata attrs which have .hourly and .daily data so caching these needs to be handled slightly differently

added simple test on cc batching for daily boundaries

…ing, etc would not be applied to data loaded from cache.

Bnb/caching fixes

… time step

…. added tests for chunks=None with height interp derivation

…el_check keys. the latter is default False, as this takes a long time since it has to load arrays into memory to compute min / max levels. ) Modified the linear interpolation method to use the 2 closest levels rather than the two closest levels which also happen to be above and below the requested level. This speeds up the interpolation by orders of magnitude.

…ape in some cases.

…nks = auto and then load only the rasterized data into memory.

…ttle more speed.

…ter when loaded

Gb/bc kwargs

Commits on Aug 19, 2024

test fix

bnb32 committed Aug 19, 2024

Configuration menu

View commit details

Copy full SHA for a00fdaf

Browse repository at this point

Copy the full SHA

a00fdaf View commit details

Browse the repository at this point in the history

Commits on Oct 15, 2024

removed bottlenexck requirement

bnb32 committed Oct 15, 2024

Configuration menu

View commit details

Copy full SHA for 94975f9

Browse repository at this point

Copy the full SHA

94975f9 View commit details

Browse the repository at this point in the history

Commits on Nov 3, 2024

height parsing fix

bnb32 committed Nov 3, 2024

Configuration menu

View commit details

Copy full SHA for 508fadc

Browse repository at this point

Copy the full SHA

508fadc View commit details

Browse the repository at this point in the history

Bnb/dh refactor #220

Bnb/dh refactor #220

Commits on Jul 23, 2024

Commits on Jul 24, 2024

Commits on Jul 25, 2024

Commits on Jul 27, 2024

Commits on Jul 28, 2024

Commits on Jul 29, 2024

Commits on Jul 30, 2024

Commits on Jul 31, 2024

Commits on Aug 1, 2024

Commits on Aug 3, 2024

Commits on Aug 4, 2024

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Commits on Aug 7, 2024

Commits on Aug 8, 2024

Commits on Aug 9, 2024

Commits on Aug 11, 2024

Commits on Aug 12, 2024

Commits on Aug 15, 2024

Commits on Aug 16, 2024

Commits on Aug 18, 2024

Commits on Aug 19, 2024

Commits on Aug 20, 2024

Commits on Aug 21, 2024

Commits on Aug 22, 2024

Commits on Aug 27, 2024

Commits on Aug 28, 2024

Commits on Aug 30, 2024

Commits on Aug 31, 2024

Commits on Sep 4, 2024

Commits on Sep 12, 2024

Commits on Sep 13, 2024

Commits on Sep 15, 2024

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 20, 2024

Commits on Sep 21, 2024

Commits on Sep 24, 2024

Commits on Sep 26, 2024

Commits on Sep 27, 2024

Commits on Oct 14, 2024

Commits on Oct 15, 2024

Commits on Oct 16, 2024

Commits on Oct 17, 2024

Commits on Oct 21, 2024

Commits on Oct 22, 2024

Commits on Oct 23, 2024

Commits on Oct 24, 2024

Commits on Oct 25, 2024

Commits on Nov 2, 2024

Commits on Nov 3, 2024

Commits on Nov 4, 2024

Commits on Nov 5, 2024