Intelligent Technologies for Interactive Entertainment. 10th EAI International Conference, INTETAIN 2018, Guimarães, Portugal, November 21-23, 2018, Proceedings

Research Article

Scene Reconstruction for Storytelling in 360 Videos

Download
197 downloads
  • @INPROCEEDINGS{10.1007/978-3-030-16447-8_12,
        author={Gon\`{e}alo Pinheiro and Nelson Alves and Luis Magalh\"{a}es and Lu\^{\i}s Agrellos and Miguel Guevara},
        title={Scene Reconstruction for Storytelling in 360 Videos},
        proceedings={Intelligent Technologies for Interactive Entertainment. 10th EAI International Conference, INTETAIN 2018, Guimar\"{a}es, Portugal,  November 21-23, 2018, Proceedings},
        proceedings_a={INTETAIN},
        year={2019},
        month={4},
        keywords={360 videos Storytelling Scene Reconstruction Binaural sound Computer vision Computer graphics 3D reconstruction People detection People tracking},
        doi={10.1007/978-3-030-16447-8_12}
    }
    
  • Gonçalo Pinheiro
    Nelson Alves
    Luis Magalhães
    Luís Agrellos
    Miguel Guevara
    Year: 2019
    Scene Reconstruction for Storytelling in 360 Videos
    INTETAIN
    Springer
    DOI: 10.1007/978-3-030-16447-8_12
Gonçalo Pinheiro1,*, Nelson Alves1, Luis Magalhães2, Luís Agrellos3, Miguel Guevara1
  • 1: Centro de Computação Gráfica
  • 2: University of Minho
  • 3: GMK
*Contact email: goncalo.pinheiro@ccg.pt

Abstract

In immersive and interactive contents like 360-degrees videos the user has the control of the camera, which poses a challenge to the content producer since the user may look to where he wants. This paper presents the concept and first steps towards the development of a framework that provides a workflow for storytelling in 360-degrees videos. With the proposed framework it will be possible to connect a sound to a source and taking advantage of binaural audio it will help to redirect the user attention to where the content producer wants. To present this kind of audio, the scenario must be mapped/reconstructed so as to understand how the objects contained in it interfere with the sound waves propagation. The proposed system is capable of reconstructing the scenario from a stereoscopic, still or motion 360-degrees video when provided in an equirectangular projection. The system also incorporates a module that detects and tracks people, mapping their motion from the real world to the 3D world. In this document we describe all the technical decisions and implementations of the system. To the best of our knowledge, this system is the only that has shown the capability to reconstruct scenarios in a large variety of 360 footage and allows for the creation of binaural audio from that reconstruction.