NTT is promoting research and development of technology that reproduces the spatial, temporal changes, and vibrations of a wide-area outdoor environment in a virtual world, reproducing the experience of the real world.

Based on the latest results, “SUBARULEVORG presents 2023 Japan Cup Cycle Road Race” was held from the 13th (Friday) to the 15th (Sunday) with Utsunomiya City and the Japan Cup Cycle Road Race Association, a non-profit organization. In the recreated ultra-realistic metaverse space “XR Sports Space,” they conducted an experiment to create a co-running experience that feels like running with the athletes. Through this experiment, they aim to recreate the experience of competing on the spot, transcending time and space, and create a new remote experience through ultra-realistic metaverse space.

The technical point is that a three-dimensional video space is generated from images from a plurality of moving cameras, and images from any viewpoint can be generated. By interpolating the occluded areas caused by moving objects in the real world through image selection and image restoration based on separately shot video, a wide-area video space is reproduced.

In addition, they can separate individual sound sources such as players and spectators in the real world from video and sound data captured and recorded by moving cameras and microphones, and analyze the spatial spread of sound, including the location of individual sounds. Generate an acoustic space that includes A natural acoustic space is reproduced by individual sound source localization that combines sound source separation and depth estimation based on object identification, and sound field reconstruction using camera and microphone position information.