Editorial Decision

[distillpub-reviewers]

Decision on ‘Multimodal’

Editorial decision for Distill | 2020 | Sam Greydanus

Reviewer 1 (even split between 4s and 5s)
Concerns about subjective neuron labeling in specific cases, eg “kids’ art.” Calls for some discussion of related work related to grandmother neurons, multimodal models. Authors responded to these comments.
“Many feature visualizations, produced by neural networks are strikingly beautiful. Plots and diagrams are well made and provoke readers for further exploration.”

Reviewer 2 (mostly 4s, a few 3s and one 2-4)
Positive comments about structure, readability, and significance. Concerns about potential mislabeling of neurons (“responded to things that seemed incongruous with their stated meaning”), type of dataset used to train model, and some small text-level issues. See “Thoughts about neurons.”
“Basically, I'm sold on most of the main claims, but I have some quibbles with some individual neurons.”

Reviewer 3 (split between 4s and 5s; one 3 for replication)
Expressed concerns about reproducibility along with acceptance of limitations given context. [Note: since this review, a whitepaper on the model has been released]. Overall very positive about the significance of results, writing quality, and scientific hygiene. A number of text-level comments and questions which the authors addressed.
“The results are significant. The writing is clear and focused. The figures generally support the claims in the text quite well.”

Editorial decision

This article presents novel interpretability results which are important in the context of machine learning and neuroscience. These results consist of detailed feature visualization of a recently-published multimodal model called CLIP. The article clearly shows that this model has individual neurons which respond to high-level semantic concepts such as ‘spiderman’ or ‘jealousy.’ It makes extensive use of OpenAI Microscope in an effort to make the visualization process (somewhat) more accessible and interactive for readers. The article also uses previously unpublished interpretability tools in order to perform its analysis; in particular, it introduces ‘faceted feature visualization’ which allows that authors to probe multiple instantiations of a concept (eg as a face, image, or image of text).

The main weaknesses pointed out by reviewers include 1) the fact that some of the neuron labels do not exactly match the feature visualizations and dataset examples, making the process of neuron labeling a somewhat subjective process and 2) the analysis of this model is difficult to reproduce, as some of the models, datasets, and code is not yet open source. The authors responded to the first of these concerns by discussing the measures they had taken to avoid bias in neuron labeling. The editors believe these measures were sufficient. The authors responded to the second of these concerns by referencing a recently-released whitepaper about the CLIP model and discussing measures they had taken to make feature visualization and dataset examples more accessible via improvements to Microscope. The editors believe these measures were sufficient. The reviewers made a number of other small recommendations and comments, to which the authors appear to have responded.

Editorial decision: This paper represents a significant contribution to interpretability research in machine learning. The authors have taken appropriate measures to correct the shortcomings pointed out by reviewers. Accept.