This repository contains the code for submission paper

WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning.

It should be emphasized that our proposed WinDB uses C++ to read Tobii fixation data, so you only need to prepare a Tobii device without any additional charging software and with a simple configuration to very easily run WinDB . However, the current way of collecting fixation for panoramic video data based on HMD is very complicated, requiring the installation of a large amount of software, such as Unity, Steam, VIVEPort..., and even expensive HMD and computer hosts equipped with high-end GPU cards.

Table of Contents

• Requirements
• Tobii Installation
• Main Steps
• Detailed Procedure of Eye Tracking Data:
  • 1. WinDB Generation
  • 2. Fixation Collection
  • 3. Fixation Generation
  • 4. Fixation Learning
  • 5. Evaluation
• PanopticVideo-300 Dataset

Requirements.

• Visual Studio 2019
• Matlab2016b
• python3.6.4
• pytorch1.10.0
• CUDA10.2
• Opencv python and C++
• Tobii Eye Tracking installation package (TobiiGhost.1.7.0-Setup.exe, Tobii_Eye_Tracking_Core_v2.16.8.214_x86.exe)

Tobii Installation

• 1 Install Tobii_Eye_Tracking_Core_v2.16.8.214_x86.exe and TobiiGhost.1.7.0-Setup.exe (License.pdf).
• 2 Start the Tobii Eye Tracking and calibration.

Main Steps

1. WinDB Generation -> 2. Fixation Collection -> 3. Fixation Generation -> 4. Fixation Learning -> 5. Evaluation

Detailed Procedure of Fixation Learning Data:

1. WinDB Generation

Figure 1. The overall pipeline of our new HMD-free fixation collection approach for panoptic data. Compared to the widely-used HMDbased method, our WinDB approach is more economical, comfortable, and reasonable.

• 1. Generate the longitude (lon.txt) and latitude (lat.txt) of WinDB;
     python ERP2WinDBLonLat.py
• 2. From ERP to WinDB based on LonLat (lon.txt, lat.txt) of WinDB.
     python ERP2WinDB.py

2. Fixation Collection

Figure 2. The existing HMD-based method compares the advantages (+) and disadvantages (-) with our WinDB approach.

• 1. Open the start.sln with Visual Studio 2019;
• 2. Config property pages of start.sln;
• 3. run the start.spp and the fixation location(x, y) will be saved in PeopleID.txt.

3. Fixation Generation

• 1. Convert the fixation location(x, y) of WinDB to ERP;
     fixation location(x, y)->WinDB location(theta, phi)->ERP Location(m, n)
     python Location2WinDB.py
• 2. Smooth the fixation of ERP on the Sphere.
     ERP Location(m, n)->Sphere Location(theta, phi)->Sphere Smooth->saliency
     python SphereSmooth.py

4. Fixation Learning

Figure 3. The motivation of the newly proposed model. Subfigures A and B illustrate the “fixation shifting” phenomenon — very common in our set. Our model has devised “a very simple yet effective” architecture, which performs spatiotemporal self-attention to alleviate the fixation shifting-induced longdistance misalignment problem.

• 1. The Training Process
     Python main.py --- Train=True
• 2. The Inference Process
     Python main.py --- Test=True
• 3. The Model Weight
     Model.pt (51.2MB)
• 4. Results
     Results

5. Evaluation

• 1. The score of every Testing set clip
     MatricsOfMyERP.m
• 2. The score of the All testing set
     MatricsOfMyALLERP.m

PanopticVideo-300 Dataset

Figure 4. The semantic categories of PanopticVideo-300 dataset. All fixations in our set are collected by WinDB.

• Video Clips (300):
  Training set: 240 clips;
  Testing set: 60 clips.

Please note that to meet the blind requirements, this code hides the issue and related personal information.