Question: How to insert new data into a recording object. Purpose: "remove_artifacts" by inserting our own cleaned data

[borrepp]

Hello,

Contex:
We have a recording that includes Intracortical Microstimulation (ICMS) multiple times during the session. The data is in NWB format originally.
A collaborator has implemented an artifact removal using ERAASR (https://github.com/djoshea/eraasr) and provided a Matlab file with:

1. A matrix of [nSnippet X nChans X nStim] which contains snippets (length = nSnippet) of the high-pass filtered & centered cleaned traces for all the channels (nChans) from all the ICMS epochs (nStim)
2. An array of size [nStim] which contains the start_sample_index of each snippet.

Now, I'm working out how to insert these clean traces into the "filtered & centered" recording object that I got from the same NWB file. Afterward, I will use "neuroconv" to export this recording into a new NWB file (which I have already tested but without the snippet replacement).

Question:
Is there a function to insert these snippets into the recording object?

I hope my question makes sense, if not I can explain anything in further detail.

Thanks for any thoughts and comments on how to approach this.

Best
Pepe

[h-mayorquin]

Hi, man.

First the simplest answer:

If your data fits into memory the easier thing is to use the NumpyRecording:

https://github.com/h-mayorquin/spikeinterface/blob/0bf2b08248b836c6323524c1f54cf3690cd6c5f8/src/spikeinterface/core/numpyextractors.py#L23-L53

This will allow you to just put your data as a numpy array and run any spikeinterface routine your might want. This gives you a lot of flexibility but is limited by memory.

Now do I get your right that you want a recording object that has the following behavior:

get_traces(start_frame=a, end_frame=b)
1) normal recording object if the object is outside the snippet
2) the snippet data if the object on a snipet

The most similar thing that I can think off is InjectTemplatesRecording but it does not seem to fit perfectly.

@alejoe91 , what are your thoughts?

Also, I did this for you and some other users:
#3587

[borrepp]

Hello @h-mayorquin

Thanks for the tips.

It is a dataset with 32 channels at 30K, so it might not be feasible to load it into memory. But I can try to process individual channels, save them as a binary folder, and later combine them within spikeinterface. Is there a function to concatenate channels?

I saw your request to get non-electrical NWB timeseries¡ Thanks¡¡ that will be useful for filtering and processing our analog signals from behavioral channels. THANKS¡¡

Best
Pepe

[h-mayorquin]

from spikeinterface.core import aggregate_channels

You can ctrl + f this here to get a longer explanation.
https://github.com/SpikeInterface/SpikeInterface-Training-Edinburgh-May24/blob/main/hands_on/cookbooks/14_00_cookbook.ipynb

Be mindful that there might be a better way than the per channel processing, I would wait a bit for @alejoe91 or someone else to comment.

But at least that's a way.

[borrepp]

Cool,

Thanks,

Yes, I will wait. I'm still finishing our pipeline to export the curation from phy.
Reinserting the cleaned snippets will be the next step, so there is no rush.

Best
Pepe

[samuelgarcia]

Hi.
RemoveArtifactsRecording do something similar but in lazy way : no need to inject in a file or memory.
It has some mode : zeros, average, ...
You could also write your own RemoveArtifactsRecording by inheritence for this.

[borrepp]

Hello @samuelgarcia

Thanks for the tip. I will check "RemoveArtifactsRecording" to write our own.

We wouldn't like to insert zeros or averages. The lab implemented a method (https://github.com/djoshea/eraasr) that allowed us to remove the artifacts and keep the single-unit activity evoked by the microstimulation. Then, we would like to replace the signal that has been processed on our end and "inject" it into a filtered spike interface recording.

As a separate note. Would it be feasible for the project to implement the ERAASR method at some point in the future?

Thanks again for your help.
Best
Pepe

[samuelgarcia]

As a separate note. Would it be feasible for the project to implement the ERAASR method at some point in the future?

Yes if you want to extend this class by adding ERAASR you can make a PR.

[Djoels]

Not sure if helpful, but you can have a look at how SHYBRID has done this some time ago.

It uses a numpy memmap as data structure in the background, which AFAIK doesn't require the data to be in memory.

It then uses some simple subtract and insert primitives to adjust the data (remove a template in one place, add it in another) as needed.

[samuelgarcia]

Inserting spike from templates into a recording is already done and totaly lazy in
spikeinterface.generation.hybrid_tools.generate_hybrid_recording()
Note that our implemntation take in accout the true motion vector detected from the recording and inject spikes given this motion vector.

[borrepp]

Thanks @Djoels and @samuelgarcia

I will check these functions. This processing step is the next thing on my list but I'm not working on that right now. As soon as I get back to this part, I will let you know which strategy works better for us.

Thanks again for all the ideas and options.

Best
Pepe