Merge branch 'dev' into add_read_nwbmethod

NeurodataWithoutBorders · Nov 25, 2024 · f95ffc5 · f95ffc5
2 parents 2518fc8 + 7087f00
commit f95ffc5
Show file tree

Hide file tree

Showing 22 changed files with 267 additions and 124 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,26 +1,33 @@
 # PyNWB Changelog
 
-## PyNWB 2.8.3 (Upcoming)
+## PyNWB 3.0.0 (Upcoming)
 
 ### Enhancements and minor changes
-* Added `NWBHDF5IO.read_nwb` convenience method to simplify reading an NWB file. @h-mayorquin [#1979](https://github.com/NeurodataWithoutBorders/pynwb/pull/1979)
-* Removed unused references to region references and builders in preparation for changes in HDMF 4.0. @rly [#1991](https://github.com/NeurodataWithoutBorders/pynwb/pull/1991)
-* Added `pynwb.read_nwb` convenience method to simplif yreading an NWBFile written with any backend @h-mayorquin [#1994](https://github.com/NeurodataWithoutBorders/pynwb/pull/1994)
+- Added `pynwb.read_nwb` convenience method to simplify reading an NWBFile written with any backend @h-mayorquin [#1994](https://github.com/NeurodataWithoutBorders/pynwb/pull/1994)
+- Added support for NWB schema 2.8.0. @rly [#2001](https://github.com/NeurodataWithoutBorders/pynwb/pull/2001)
+- Removed `SpatialSeries.bounds` field that was not functional. This will be fixed in a future release. @rly [#1907](https://github.com/NeurodataWithoutBorders/pynwb/pull/1907), [#1996](https://github.com/NeurodataWithoutBorders/pynwb/pull/1996)
+- Added support for `NWBFile.was_generated_by` field. @stephprince [#1924](https://github.com/NeurodataWithoutBorders/pynwb/pull/1924)
+- Added support for `model_number`, `model_name`, and `serial_number` fields to `Device`. @stephprince [#1997](https://github.com/NeurodataWithoutBorders/pynwb/pull/1997)
+- Deprecated `EventWaveform` neurodata type. @rly [#1940](https://github.com/NeurodataWithoutBorders/pynwb/pull/1940)
+- Deprecated `ImageMaskSeries` neurodata type. @rly [#1941](https://github.com/NeurodataWithoutBorders/pynwb/pull/1941)
 
-### Documentation and tutorial enhancements
-- Added documentation example for `SpikeEventSeries`. @stephprince [#1983](https://github.com/NeurodataWithoutBorders/pynwb/pull/1983)
-- Added documentation example for `AnnotationSeries`. @stephprince [#1989](https://github.com/NeurodataWithoutBorders/pynwb/pull/1989)
+## PyNWB 2.8.3 (November 19, 2024)
 
-### Performance
+### Enhancements and minor changes
+- Added `NWBHDF5IO.read_nwb` convenience method to simplify reading an NWB file. @h-mayorquin [#1979](https://github.com/NeurodataWithoutBorders/pynwb/pull/1979)
+- Removed unused references to region references and builders in preparation for changes in HDMF 4.0. @rly [#1991](https://github.com/NeurodataWithoutBorders/pynwb/pull/1991)
+- Made gain an optional argument for PatchClampSeries to match the schema. @stephprince [#1975](https://github.com/NeurodataWithoutBorders/pynwb/pull/1975)
+- Added warning when writing files with `NWBHDF5IO` without the `.nwb` extension. @stephprince [#1978](https://github.com/NeurodataWithoutBorders/pynwb/pull/1978)
 - Cache global type map to speed import 3X. @sneakers-the-rat [#1931](https://github.com/NeurodataWithoutBorders/pynwb/pull/1931)
 
 ### Bug fixes
 - Fixed bug in how `ElectrodeGroup.__init__` validates its `position` argument. @oruebel [#1770](https://github.com/NeurodataWithoutBorders/pynwb/pull/1770)
 - Changed `SpatialSeries.reference_frame` from required to optional as specified in the schema. @rly [#1986](https://github.com/NeurodataWithoutBorders/pynwb/pull/1986)
 
-### Enhancements and minor changes
-- Made gain an optional argument for PatchClampSeries to match the schema. @stephprince [#1975](https://github.com/NeurodataWithoutBorders/pynwb/pull/1975)
-- Added warning when writing files with `NWBHDF5IO` without the `.nwb` extension. @stephprince [#1978](https://github.com/NeurodataWithoutBorders/pynwb/pull/1978)
+### Documentation and tutorial enhancements
+- Added documentation example for `SpikeEventSeries`. @stephprince [#1983](https://github.com/NeurodataWithoutBorders/pynwb/pull/1983)
+- Added documentation example for `AnnotationSeries`. @stephprince [#1989](https://github.com/NeurodataWithoutBorders/pynwb/pull/1989)
+- Added documentation example for `DecompositionSeries`. @stephprince [#1981](https://github.com/NeurodataWithoutBorders/pynwb/pull/1981)
 
 ## PyNWB 2.8.2 (September 9, 2024)
 
@@ -60,7 +67,7 @@
 ## PyNWB 2.7.0 (May 2, 2024)
 
 ### Enhancements and minor changes
-- Added `bounds` field to `SpatialSeries` to set optional boundary range (min, max) for each dimension of data. @mavaylon1 [#1869](https://github.com/NeurodataWithoutBorders/pynwb/pull/1869/files)
+- Added `bounds` field to `SpatialSeries` to set optional boundary range (min, max) for each dimension of data. @mavaylon1 [#1869](https://github.com/NeurodataWithoutBorders/pynwb/pull/1869)
 - Added support for NWB schema 2.7.0. See [2.7.0 release notes](https://nwb-schema.readthedocs.io/en/latest/format_release_notes.html) for details
 - Deprecated `ImagingRetinotopy` neurodata type. @rly [#1813](https://github.com/NeurodataWithoutBorders/pynwb/pull/1813)
 - Modified `OptogeneticSeries` to allow 2D data, primarily in extensions of `OptogeneticSeries`. @rly [#1812](https://github.com/NeurodataWithoutBorders/pynwb/pull/1812)

diff --git a/docs/gallery/domain/ecephys.py b/docs/gallery/domain/ecephys.py
@@ -31,7 +31,9 @@
 from dateutil.tz import tzlocal
 
 from pynwb import NWBHDF5IO, NWBFile
+
 from pynwb.ecephys import LFP, ElectricalSeries, SpikeEventSeries
+from pynwb.misc import DecompositionSeries
 
 #######################
 # Creating and Writing NWB files
@@ -77,10 +79,16 @@
 #
 # The electrodes table references a required :py:class:`~pynwb.ecephys.ElectrodeGroup`, which is used to represent a
 # group of electrodes. Before creating an :py:class:`~pynwb.ecephys.ElectrodeGroup`, you must define a
-# :py:class:`~pynwb.device.Device` object using the method :py:meth:`.NWBFile.create_device`.
-
+# :py:class:`~pynwb.device.Device` object using the method :py:meth:`.NWBFile.create_device`. The fields
+# ``description``, ``manufacturer``, ``model_number``, ``model_name``, and ``serial_number`` are optional, but
+# recommended.
 device = nwbfile.create_device(
-    name="array", description="the best array", manufacturer="Probe Company 9000"
+    name="array",
+    description="A 12-channel array with 4 shanks and 3 channels per shank",
+    manufacturer="Array Technologies",
+    model_number="PRB_1_4_0480_123",
+    model_name="Neurovoxels 0.99",
+    serial_number="1234567890",
 )
 
 #######################
@@ -241,6 +249,68 @@
 )
 ecephys_module.add(lfp)
 
+#######################
+# If the derived data is filtered but not downsampled, you can store the data in an
+# :py:class:`~pynwb.ecephys.ElectricalSeries` object in a :py:class:`~pynwb.ecephys.FilteredEphys` object
+# instead of a :py:class:`~pynwb.ecephys.LFP` object.
+
+from pynwb.ecephys import FilteredEphys
+
+filtered_data = np.random.randn(50, 12)
+filtered_electrical_series = ElectricalSeries(
+    name="FilteredElectricalSeries",
+    description="Filtered data",
+    data=filtered_data,
+    electrodes=all_table_region,
+    starting_time=0.0,
+    rate=200.0,
+)
+
+filtered_ephys = FilteredEphys(electrical_series=filtered_electrical_series)
+ecephys_module.add(filtered_ephys)
+
+################################
+# In some cases, you may want to further process the LFP data and decompose the signal into different frequency bands 
+# to use for other downstream analyses. You can store the processed data from these spectral analyses using a
+# :py:class:`~pynwb.misc.DecompositionSeries` object. This object allows you to include metadata about the frequency
+# bands and metric used (e.g., power, phase, amplitude), as well as link the decomposed data to the original 
+# :py:class:`~pynwb.base.TimeSeries` signal the data was derived from.
+
+#######################
+# .. note:: When adding data to :py:class:`~pynwb.misc.DecompositionSeries`, the ``data`` argument is assumed to be 
+#           3D where the first dimension is time, the second dimension is channels, and the third dimension is bands.
+
+
+bands = dict(theta=(4.0, 12.0), 
+             beta=(12.0, 30.0), 
+             gamma=(30.0, 80.0))  # in Hz
+phase_data = np.random.randn(50, 12, len(bands))  # 50 samples, 12 channels, 3 frequency bands  
+
+decomp_series = DecompositionSeries(
+    name="theta",
+    description="phase of bandpass filtered LFP data",
+    data=phase_data,
+    metric='phase',
+    rate=200.0,
+    source_channels=all_table_region,
+    source_timeseries=lfp_electrical_series,
+)
+
+for band_name, band_limits in bands.items():
+    decomp_series.add_band(
+        band_name=band_name,
+        band_limits=band_limits,
+        band_mean=np.nan,
+        band_stdev=np.nan,
+    )
+
+ecephys_module.add(decomp_series)
+
+#######################
+# The frequency band information can also be viewed as a pandas DataFrame.
+
+decomp_series.bands.to_dataframe()
+
 ####################
 # .. _units_electrode:
 #
@@ -269,6 +339,10 @@
 
 #######################
 # The :py:class:`~pynwb.misc.Units` table can also be converted to a pandas :py:class:`~pandas.DataFrame`.
+#
+# The :py:class:`~pynwb.misc.Units` table can contain simply the spike times of sorted units, or you can also include
+# individual and mean waveform information in some of the optional, predefined :py:class:`~pynwb.misc.Units` table
+# columns: ``waveform_mean``, ``waveform_sd``, or ``waveforms``.
 
 nwbfile.units.to_dataframe()
 
@@ -287,44 +361,50 @@
     description="shank0",
 )
 
-
-spike_events = SpikeEventSeries(name='SpikeEvents_Shank0',
-                                description="events detected with 100uV threshold",
-                                data=spike_snippets,
-                                timestamps=np.arange(20),
-                                electrodes=shank0)
+spike_events = SpikeEventSeries(
+    name='SpikeEvents_Shank0',
+    description="events detected with 100uV threshold",
+    data=spike_snippets,
+    timestamps=np.arange(20),
+    electrodes=shank0,
+)
 nwbfile.add_acquisition(spike_events)
 
-#######################
-# Designating electrophysiology data
-# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-#
-# As mentioned above, :py:class:`~pynwb.ecephys.ElectricalSeries` objects
-# are meant for storing specific types of extracellular recordings. In addition to this
-# :py:class:`~pynwb.base.TimeSeries` class, NWB provides some :ref:`modules_overview`
-# for designating the type of data you are storing. We will briefly discuss them here, and refer the reader to
-# :py:mod:`API documentation <pynwb.ecephys>` and :ref:`basics` for more details on
-# using these objects.
-#
-# For storing unsorted spiking data, there are two options. Which one you choose depends on what data you 
-# have available. If you need to store the complete, continuous raw voltage traces, you should store the traces with
-# :py:class:`~pynwb.ecephys.ElectricalSeries` objects as :ref:`acquisition <basic_timeseries>` data, and use
-# the :py:class:`~pynwb.ecephys.EventDetection` class for identifying the spike events in your raw traces.
+############################################
+# If you need to store the complete, continuous raw voltage traces, along with unsorted spike times, you should store
+# the traces with :py:class:`~pynwb.ecephys.ElectricalSeries` objects as :ref:`acquisition <basic_timeseries>` data,
+# and use the :py:class:`~pynwb.ecephys.EventDetection` class to identify the spike events in your raw traces.
+
+from pynwb.ecephys import EventDetection
+
+event_detection = EventDetection(
+    name="threshold_events",
+    detection_method="thresholding, 1.5 * std",
+    source_electricalseries=raw_electrical_series,
+    source_idx=[1000, 2000, 3000],
+    times=[.033, .066, .099],
+)
+
+ecephys_module.add(event_detection)
+
+######################################
 # If you do not want to store the raw voltage traces and only the waveform 'snippets' surrounding spike events,
-# you should use :py:class:`~pynwb.ecephys.SpikeEventSeries` objects.
-#
-# The results of spike sorting (or clustering) should be stored in the top-level :py:class:`~pynwb.misc.Units` table.
-# The :py:class:`~pynwb.misc.Units` table can contain simply the spike times of sorted units, or you can also include 
-# individual and mean waveform information in some of the optional, predefined :py:class:`~pynwb.misc.Units` table 
-# columns: ``waveform_mean``, ``waveform_sd``, or ``waveforms``.
+# you should store the snippets with :py:class:`~pynwb.ecephys.SpikeEventSeries` objects.
 #
-# For local field potential data, there are two options. Again, which one you choose depends on what data you
-# have available. With both options, you should store your traces with :py:class:`~pynwb.ecephys.ElectricalSeries`
-# objects. If you are storing unfiltered local field potential data, you should store
-# the :py:class:`~pynwb.ecephys.ElectricalSeries` objects in :py:class:`~pynwb.ecephys.LFP` data interface object(s).
-# If you have filtered LFP data, you should store the :py:class:`~pynwb.ecephys.ElectricalSeries` objects  in
-# :py:class:`~pynwb.ecephys.FilteredEphys` data interface object(s).
+# NWB also provides a way to store features of spikes, such as principal components, using the
+# :py:class:`~pynwb.ecephys.FeatureExtraction` class.
+
+from pynwb.ecephys import FeatureExtraction
+
+feature_extraction = FeatureExtraction(
+    name="PCA_features",
+    electrodes=all_table_region,
+    description=["PC1", "PC2", "PC3", "PC4"],
+    times=[.033, .066, .099],
+    features=np.random.rand(3, 12, 4),  # time, channel, feature
+)
 
+ecephys_module.add(feature_extraction)
 
 ####################
 # .. _ecephys_writing:

diff --git a/docs/gallery/domain/ophys.py b/docs/gallery/domain/ophys.py
@@ -86,12 +86,17 @@
 #     :align: center
 #
 # Create a :py:class:`~pynwb.device.Device` named ``"Microscope"`` in the :py:class:`~pynwb.file.NWBFile` object. Then
-# create an  :py:class:`~pynwb.ophys.OpticalChannel` named ``"OpticalChannel"``.
+# create an  :py:class:`~pynwb.ophys.OpticalChannel` named ``"OpticalChannel"``. The fields
+# ``description``, ``manufacturer``, ``model_number``, ``model_name``, and ``serial_number`` are optional, but
+# recommended.
 
 device = nwbfile.create_device(
     name="Microscope",
     description="My two-photon microscope",
-    manufacturer="The best microscope manufacturer",
+    manufacturer="Loki Labs",
+    model_number="ABC-123",
+    model_name="Loki 1.0",
+    serial_number="1234567890",
 )
 optical_channel = OpticalChannel(
     name="OpticalChannel",

diff --git a/docs/gallery/domain/plot_behavior.py b/docs/gallery/domain/plot_behavior.py
@@ -105,17 +105,13 @@
 #
 # For position data ``reference_frame`` indicates the zero-position, e.g.
 # the 0,0 point might be the bottom-left corner of an enclosure, as viewed from the tracking camera.
-# In :py:class:`~pynwb.behavior.SpatialSeries`, the ``bounds`` field allows the user to set
-# the boundary range, i.e.,  (min, max), for each dimension of ``data``. The units are the same as in ``data``.
-# This field does not enforce a boundary on the dataset itself.
 
 timestamps = np.linspace(0, 50) / 200
 
 position_spatial_series = SpatialSeries(
     name="SpatialSeries",
     description="Position (x, y) in an open field.",
     data=position_data,
-    bounds=[(0,50), (0,50)],
     timestamps=timestamps,
     reference_frame="(0,0) is bottom left corner",
 )

diff --git a/docs/gallery/general/plot_file.py b/docs/gallery/general/plot_file.py
@@ -55,7 +55,6 @@
 
     * **Optical physiology and imaging:** :py:class:`~pynwb.image.ImageSeries` is the base type
       for image recordings and is further refined by the
-      :py:class:`~pynwb.image.ImageMaskSeries`,
       :py:class:`~pynwb.image.OpticalSeries`,
       :py:class:`~pynwb.ophys.OnePhotonSeries`, and
       :py:class:`~pynwb.ophys.TwoPhotonSeries` types.
@@ -92,7 +91,6 @@
       :py:class:`~pynwb.behavior.EyeTracking`.
 
     * **Extracellular electrophysiology:** :py:class:`~pynwb.ecephys.EventDetection`,
-      :py:class:`~pynwb.ecephys.EventWaveform`,
       :py:class:`~pynwb.ecephys.FeatureExtraction`,
       :py:class:`~pynwb.ecephys.FilteredEphys`,
       :py:class:`~pynwb.ecephys.LFP`.
@@ -297,10 +295,12 @@
 # object with text information about a stimulus and add it to the stimulus group in 
 # the :py:class:`~pynwb.file.NWBFile`. 
 
-annotations = AnnotationSeries(name='airpuffs',
-                               data=['Left Airpuff', 'Right Airpuff', 'Right Airpuff'],
-                               description='Airpuff events delivered to the animal',
-                               timestamps=[1.0, 3.0, 8.0])
+annotations = AnnotationSeries(
+    name='airpuffs',
+    data=['Left Airpuff', 'Right Airpuff', 'Right Airpuff'],
+    description='Airpuff events delivered to the animal',
+    timestamps=[1.0, 3.0, 8.0],
+)
 
 nwbfile.add_stimulus(annotations)
 
@@ -593,8 +593,7 @@
 
 ####################
 # .. [#] Some data interface objects have a default name. This default name is the type of the data interface. For
-#    example, the default name for :py:class:`~pynwb.ophys.ImageSegmentation` is "ImageSegmentation" and the default
-#    name for :py:class:`~pynwb.ecephys.EventWaveform` is "EventWaveform".
+#    example, the default name for :py:class:`~pynwb.ophys.ImageSegmentation` is "ImageSegmentation".
 #
 # .. [#] HDF5 is the primary backend supported by NWB.
 #

diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -168,6 +168,10 @@ def __call__(self, filename):
 nitpick_ignore = [('py:class', 'Intracomm'),
                   ('py:class', 'BaseStorageSpec')]
 
+linkcheck_ignore = [
+    r'https://training.incf.org/*'  # temporary ignore until SSL certificate issue is resolved
+]
+
 suppress_warnings = ["config.cache"]
 
 # Add any paths that contain templates here, relative to this directory.

diff --git a/src/pynwb/behavior.py b/src/pynwb/behavior.py
@@ -26,8 +26,6 @@ class SpatialSeries(TimeSeries):
             {'name': 'data', 'type': ('array_data', 'data', TimeSeries), 'shape': ((None, ), (None, None)), # required
              'doc': ('The data values. Can be 1D or 2D. The first dimension must be time. If 2D, there can be 1, 2, '
                      'or 3 columns, which represent x, y, and z.')},
-            {'name': 'bounds', 'type': list, 'shape': ((1, 2), (2, 2), (3, 2)), 'default': None,
-             'doc': 'The boundary range (min, max) for each dimension of data.'},
             {'name': 'reference_frame', 'type': str,
              'doc': 'description defining what the zero-position is', 'default': None},
             {'name': 'unit', 'type': str, 'doc': 'The base unit of measurement (should be SI unit)',
@@ -38,7 +36,9 @@ def __init__(self, **kwargs):
         """
         Create a SpatialSeries TimeSeries dataset
         """
-        name, data, bounds, reference_frame, unit = popargs('name', 'data', 'bounds', 'reference_frame', 'unit', kwargs)
+        name, data, reference_frame, unit = popargs(
+            'name', 'data', 'reference_frame', 'unit', kwargs
+        )
         super().__init__(name, data, unit, **kwargs)
 
         # NWB 2.5 restricts length of second dimension to be <= 3
@@ -49,7 +49,6 @@ def __init__(self, **kwargs):
                           "The second dimension should have length <= 3 to represent at most x, y, z." %
                           (name, str(data_shape)))
 
-        self.bounds = bounds
         self.reference_frame = reference_frame
 
     @staticmethod