Plot for viewing images

[hunse]

When running a model that takes images as input, we need a good way to view the input images.

[cchan987]

#404 Allows for custom html. Does this suffice for a way to have images displayed?

[jgosmann]

I think it still requires to write Javascript to display images as the transmitted image data has to be parsed and drawn onto a canvas. It might be possible to encode that data directly into the HTML, but I'm not sure.

The <img> tag won't at the moment work as the server needs to be configured to deliver that image when a certain URL is requested.

[tcstewar]

Does this suffice for a way to have images displayed?

That's not what this issue is talking about. This issue is talking about dealing with a Node whose output value is, say, a 10x10 image, and it'd be nice to actually display that image rather than doing a Value plot with 100 lines on it.

[Seanny123]

User-interface wise, would this just be another option in the context menu or is there some other way that you'd like the user to access this option?

[tcstewar]

User-interface wise, would this just be another option in the context menu or is there some other way that you'd like the user to access this option?

Hmm... good question. Part of me think it's be really neat for this to be an option on the Value plot itself (toggling back and forth between this mode and the normal line mode), but that's rather difficult to implement given our current framework. So I think for the first implementation, we just have it as another option in the context menu (as it was in Nengo 1.4), and then look to merging the two in a future PR.

[Seanny123]

Why is it difficult to implement in our current framework? Is it because
the SVG can't handle click events, like I seem to remember you mentioning
earlier?

On Sun, Aug 30, 2015 at 1:07 PM, tcstewar [email protected] wrote:

User-interface wise, would this just be another option in the context menu
or is there some other way that you'd like the user to access this option?

Hmm... good question. Part of me think it's be really neat for this to be
an option on the Value plot itself (toggling back and forth between this
mode and the normal line mode), but that's rather difficult to implement
given our current framework. So I think for the first implementation, we
just have it as another option in the context menu (as it was in Nengo
1.4), and then look to merging the two in a future PR.

—
Reply to this email directly or view it on GitHub
#382 (comment).

[tcstewar]

Why is it difficult to implement in our current framework?

Just that the entire Value class is written assuming that there's just one basic graph it shows, so it'd basically be a rewrite of that whole thing to make it toggleable

Is it because
the SVG can't handle click events, like I seem to remember you mentioning
earlier?

Nope, and that thing I was mentioning earlier I was wrong about. What I was remembering is that markers on SVG lines can't handle click events, which is why this time when I implemented the Connection lines the triangle arrow is done as a separate SVG path, rather than as a line marker.

[Seanny123]

@tcstewar didn't a version of this get included in a demo of Eric's ImageNet thing? Where is that code stored and how did you implement it in that case?

[hunse]

Here's a simple version of the code I've been using:

import nengo

import base64
import PIL
import cStringIO

images  # (n_images, n_channels, height, width) ndarray
image_shape = images.shape[1:]

with nengo.Network() as model:
    image_node = nengo.Node(nengo.processes.PresentInput(images, presentation_time=0.2))

    # --- image display
    def display_func(t, x):
        values = x.reshape(image_shape)
        values = values.transpose((1,2,0))
        # values = values * 255.
        values = (values / 255. + 0.5).clip(0, 1) * 255.

        values = values.astype('uint8')

        png = PIL.Image.fromarray(values)
        buffer = cStringIO.StringIO()
        png.save(buffer, format="PNG")
        img_str = base64.b64encode(buffer.getvalue())

        display_func._nengo_html_ = '''
            <svg width="100%%" height="100%%" viewbox="0 0 100 100">
            <image width="100%%" height="100%%"
                   xlink:href="data:image/png;base64,%s"
                   style="image-rendering: pixelated;">
            </svg>''' % (''.join(img_str))

    display_node = nengo.Node(display_func, size_in=image_node.size_out)
    nengo.Connection(image_node, display_node, synapse=None)

#755 is working on getting a better interface for providing the HTML, so that I can build this right into a PresentImages process in nengo_extras so this can be done automatically.

[tcstewar]

I think the _nengo_html_ approach and the eventual #755 approach make sense for full-fledged images (where there are various different colour options and shape options to worry about). But it still might be handy to have a built-in way of displaying high-dimensional value data as a greyscale pixel grid. This should just be a small variation on the current firing rate plot (but showing the decoded value, rather than filtered spikes).