PathGan: Visual Scan-path Prediction with Generative Adversarial Networks

Marc Assens	Kevin McGuinness	Xavier Giro-i-Nieto	Noel O'Connor

A joint collaboration between:

Insight Centre for Data Analytics	Dublin City University (DCU)	UPC Image Processing Group

Abstract

We introduce PathGAN, a deep neural network for visual scanpath prediction trained on adversarial examples. A visual scanpath is defined as the sequence of fixation points over an image defined by a human observer with its gaze. PathGAN is composed of two parts, the generator and the discriminator. Both parts extract features from images using off-the-shelf networks, and train recurrent layers to generate or discriminate scanpaths accordingly. In scanpath prediction, the stochastic nature of the data makes it very difficult to generate realistic predictions using supervised learning strategies, but we adopt adversarial training as a suitable alternative. Our experiments prove how PathGAN improves the state of the art of visual scanpath prediction on the iSUN and Salient360! datasets.

Publication

Find the pre-print version of our work on arXiv.

Please cite with the following Bibtex code:

@inproceedings{Assens2018pathgan,
title={PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks},
author={Marc Assens, Xavier Giro-i-Nieto, Kevin McGuinness, Noel E. O'Connor},
journal={ECCV Workshop on Egocentric Perception, Interaction and Computing (EPIC)},
year={2018}
}

You may also want to refer to our publication with the more human-friendly Chicago style:

Marc Assens, Xavier Giro-i-Nieto, Kevin McGuinness, Noel E. O’Connor. “PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks”, ECCV Workshop on Egocentric Perception, Interaction and Computing (EPIC), 2018.

Models

The model is composed by two deep neural networks, the generator and the discriminator, whose combined efforts aim at predicting a realistic scanpath from a given image.

Model Architecture:

The model is implemented in Keras, which at its time is developed over Tensorflow.

• [Scan-path generator model for Salient 360! dataset (250 MB)]

Examples

We provide examples of predicted object sequences for two datasets.

1. iSun

2. Salient360! Benchmark & Challenges

Our model ranked #2 in the ICME'18 Grand Challenge on Salient360!).

• Scanpaths generated for tasks 3 & 4 of the ICME Salient360 Grand Challenge 2018
• Results for Model Type 3: Prediction of Eye-gaze Scan-paths
• Results for Model Type 4: Prediction of Head-gaze Scan-paths

The big dot indicates the first fixation of the scanpath:

Acknowledgements

We especially want to thank our technical support team:

Albert Gil

We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GeForce GTX Titan Z and Titan X used in this work.
The Image Processing Group at the UPC is a SGR14 Consolidated Research Group recognized and sponsored by the Catalan Government (Generalitat de Catalunya) through its AGAUR office.
This work has been developed in the framework of projects TEC2013-43935-R and TEC2016-75976-R, financed by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF).