Research Article
Scene Reconstruction for Storytelling in 360 Videos
@INPROCEEDINGS{10.1007/978-3-030-16447-8_12, author={Gon\`{e}alo Pinheiro and Nelson Alves and Luis Magalh\"{a}es and Lu\^{\i}s Agrellos and Miguel Guevara}, title={Scene Reconstruction for Storytelling in 360 Videos}, proceedings={Intelligent Technologies for Interactive Entertainment. 10th EAI International Conference, INTETAIN 2018, Guimar\"{a}es, Portugal, November 21-23, 2018, Proceedings}, proceedings_a={INTETAIN}, year={2019}, month={4}, keywords={360 videos Storytelling Scene Reconstruction Binaural sound Computer vision Computer graphics 3D reconstruction People detection People tracking}, doi={10.1007/978-3-030-16447-8_12} }- Gonçalo Pinheiro
Nelson Alves
Luis Magalhães
Luís Agrellos
Miguel Guevara
Year: 2019
Scene Reconstruction for Storytelling in 360 Videos
INTETAIN
Springer
DOI: 10.1007/978-3-030-16447-8_12
Abstract
In immersive and interactive contents like 360-degrees videos the user has the control of the camera, which poses a challenge to the content producer since the user may look to where he wants. This paper presents the concept and first steps towards the development of a framework that provides a workflow for storytelling in 360-degrees videos. With the proposed framework it will be possible to connect a sound to a source and taking advantage of binaural audio it will help to redirect the user attention to where the content producer wants. To present this kind of audio, the scenario must be mapped/reconstructed so as to understand how the objects contained in it interfere with the sound waves propagation. The proposed system is capable of reconstructing the scenario from a stereoscopic, still or motion 360-degrees video when provided in an equirectangular projection. The system also incorporates a module that detects and tracks people, mapping their motion from the real world to the 3D world. In this document we describe all the technical decisions and implementations of the system. To the best of our knowledge, this system is the only that has shown the capability to reconstruct scenarios in a large variety of 360 footage and allows for the creation of binaural audio from that reconstruction.