Home Abstract Video Dataset Paper ☰

Abstract

Watching a 360° sports video requires a viewer to continuously select a viewing angle, either through a sequence of mouse clicks or head movements. To relieve the viewer from this “360 piloting” task, we propose “deep 360 pilot” – a deep learning-based agent for piloting through 360° sports videos automatically. At each frame, the agent observes a panoramic image and has the knowledge of previously selected viewing angles. The task of the agent is to shift the current viewing angle (i.e. action) to the next preferred one (i.e., goal). We propose to directly learn an online policy of the agent from data. We use the policy gradient technique to jointly train our pipeline: by minimizing (1) a regression loss measuring the distance between the selected and ground truth viewing angles, (2) a smoothness loss encouraging smooth transition in viewing angle, and (3) maximizing an expected reward of focusing on a foreground object. To evaluate our method, we build a new 360-Sports video dataset consisting of five sports domains. We train domain-specific agents and achieve the best performance on viewing angle selection accuracy and transition smoothness compared to [51] and other baselines.

Video Overview

Sports-360 dataset

Following resources are provided:


Code (GitHub)


Sports-360 videos (4KB)


Sports-360 labels (370KB)


Sports-360 features (3GB)

CVPR 2017

Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video

Hou-Ning Hu*, Yen-Chen Lin*, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, Min Sun (*indicate equal contribution)

Oral Presentation

 Paper (High-resolution) Paper (arXiv) 
@inproceedings{HuLinCVPR17,
  title     = {Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video},
  author    = {Hou-Ning Hu and Yen-Chen Lin and Ming-Yu Liu and Hsien-Tzu Cheng and Yung-Ju Chang and Min Sun},
  year      = {2017},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}
}

CHI 2017Project Website

Tell Me Where to Look: Investigating Ways for Assisting Focus in 360° Video

Yen-Chen Lin, Yung-Ju Chang, Hou-Ning Hu, Hsien-Tzu Cheng, Chi-Wen Huang, Min Sun
Paper (High-resolution) Paper (DOI) 
@inproceedings{LinCHI17,
  title     = {Tell Me Where to Look: Investigating Ways for Assisting Focus in 360° Video},
  author    = {Yen-Chen Lin and Yung-Ju Chang and Hou-Ning Hu and Hsien-Tzu Cheng and Chi-Wen Huang and Min Sun},
  year      = {2017},
  booktitle = {ACM Conference on Human Factors in Computing Systems (CHI)}
}