360 stream to AR app for image-based lighting and real-time reflections
With this proof of concept app I aimed at using the RICOH THETA S camera to capture 360 image and use it for image-based lighting and reflections in an augmented reality application.
This app is capable of loading an equirectangular image from the phone memory, downloading any image from the Internet or connecting to a PC and using a real-time video stream from the RICOH THETA S camera. The equirectangular image (or stream) is then converted into a cubemap and used for image-based lighting and reflections in the Unity3d game engine.
I find this project compelling as it’s something 360 cameras are not usually used for and augmented reality applications can surely benefit from some additional realism.
A few plug-ins were used to ease the development process as the goal of this app is to prove that the RICOH THETA S camera can be beneficially used in an augmented reality application.
The Vuforia AR plugin was used as it’s the one I’m most familiar with, but I’m also developing versions for Kudan and Wikitude.
- Android app tested on Galaxy S6 - Android 5.1.1
- PC Streaming app tested on Windows 10.
- RICOH THETA S camera
Additional software/plug-ins used
3D models used for demonstration
The technology and its direction shown in this work, "360 stream to AR app for image-based lighting and real-time reflections", will be very important in the expression in the future virtual reality and augmented reality applications. There are many submissions that focus on IBL (Image-Based Lighting) in this contest. Among the peers, this work that implements IBL for AR applications using 360-degree images of RICOH THETA was given high marks by the judges from the viewpoint of "Does it inspire imagination and sense of innovation?", and was selected as Second Prize.
Interview with the award winners
I'm very humbled to be one of the awarded contestants of the RICOH THETA x IoT Developers Contest. This year's contest had some amazing applications by very passionate people and I'm sure most will be developed into products. The competition was very well organised and I would like to thank the team behind it and the sponsors for this pleasant evening at Miraikan. The diversity of applications we saw is a proof of the versatility of the RICOH THETA S camera and I'm sure this is just the beginning of a very healthy ecosystem.