Request for feature addition: Visualization of Neuron Activations and Clustered Neuron Firings

[cristianestojeda]

Is it possible to include a method to visualize neuron activations and clustering neurons by activation values from https://jalammar.github.io/explaining-transformers/?

Some method to visualize this kind of figures:
https://jalammar.github.io/images/explaining/activations-4.PNG
https://jalammar.github.io/images/explaining/activations_1.PNG
https://jalammar.github.io/images/explaining/activations_2.PNG

It would be very interesting to incorporate those visualizations as improve the interpretability of the Transformer architecture.
Kind regards.

[jalammar]

Oh interesting. I'm curious why you'd find them useful. I was curious at how they'd look, but it was not a visual that is easily interpretable by humans. Ecco still collects the activations and you can plot them. That visual was using heatmaps to plot those values. That investigation turned to NMF factor analysis which brought forth more human-understandable patterns.

Could you shed some light on what you'd like to achieve? I might still have that code lying around somewhere. I couldn't find it from a quick search, but it's likely still around. I don't have a lot time to embed it into Ecco, but maybe just share it in its raw state if I find it.