IBC2023 Accelerator Project: Real-Time XR Sport Edge

The Challenge

The specific challenge as stated on the project site is:

Continuing R&D from the award-winning 2021 Accelerator project that started to explore 5G location based XR, this revived 2023 Accelerator will build on the latest, cutting-edge innovations in interactive digital athletes, motion capture, AI and high-speed edge content delivery for live XR sports workflows for virtual 3D worlds, and the potentials into OTT platforms.

The aim of this Challenge is to broadcast digital twin XR Sports using combat MMA (Mixed Martial Arts) into live, immersive high-end graphics, spatial and social audio and real-time, edge-compute deliverability to fans in a 3D virtual world via VR headsets, computers, mobile devices and also simultaneously to an outdoor Location-Based Experience venue at The Outernet in London.

The Problem

As XR (eXtended Reality) systems are beginning to expand beyond gaming, the world of sport is taking interest. Nariman Tootoonchi, Creative Technology Manager at Outernet Global Ltd, described the problem this project is taking on. “This project aims to tackle the challenge of pushing the limits of real-time motion capture data integration with a digital skeleton in Unreal Engine,” he said. “At Outernet, our focus for this project is on efficiently rendering real-time content on screen, optimising computing resources required to display the 3D characters and Unreal content on the LED canvas within The Now Building which is our largest space in the district.”

Register now for IBC2023

Rob Oldfield, CEO and co-founder, Salsa Sound Ltd, one of the Participants, explains the innovative nature of the project. “Metaverse and XR experiences are normally the privilege of bespoke high-end local servers with heavily post-produced content,” he said. “We wanted to do it live on the edge in real-time for a great immersive, personalised and accessible audience experience either over an HMD or large environment delivery.”

The Goal

The project builds on the work of the 5G XR LBE Accelerator project from 2021, which was selected as Project of the Year. Building on the latest developments in motion capture and hight-speed content delivery the focus of the project is ‘digital twin’ powered XR sports in an immersive environment with the following objectives:

• Real-time capture of high-speed XR sports (Mixed Martial Arts) with 2 or more athletes and participants.
• Low-to-no latency delivery via edge technologies.
• Creating and using virtual avatars of real-world athletes, that can be augmented with VFX.
• High-quality, photorealistic graphics in immersive environments
• Audiences in virtual 3D worlds can view on multiple devices (VR headset, mobile, computer).
• Audiences will participate and spectate on multiple devices (VR headset, mobile, computer, LBE venue).
• Interoperable Talent ID labelling for Provenance Verification & Automation/

“The accelerator program,” said Tootoonchi, “allows us to activate innovative workflows to push boundaries of pre-existing rendering and tracking technology to effect real-time rendered 3D content on screen. These proofs of concepts are invaluable for our future development and our content offering utilising real-time rendering.”

“What sets this solution apart is its innovative method of streaming motion capture data across long distances, such as between cities, with minimal latency,” he continued. “Upon reaching the Outernet London graphics machines, the data undergoes rapid decompression and is seamlessly retargeted within Unreal Engine. This process enables real-time control of 3D characters and certainly a unique live experience.”

Oldfield highlighted the collaborative goals of working in the Accelerator. To him, it’s a huge advantage, “to get a chance to work on a ground-breaking project with an array of exciting companies, incentivise the development of new technology and create new industry relationships.”

Participants

One of the main benefits of working within the IBC Accelerator programme is access to a wide range of organisations in a collaborative environment. Participating organisations get access to technologies, production spaces, media, and knowledge that they might not have within their own companies.

“Engaging with external organisations has yielded notable benefits when compared with an exclusively in-house approach,” noted Tootoonchi. “Outernet’s role in the project is that we are a destination, focusing on content production and delivering immersive experiences through our screen-equipped spaces fitted with advanced surround audio systems which is ideal for live graphic content. While internal management of 3D content creation is conceivable, the project’s scope necessitates collaboration with technology partners to harness specialised solutions effectively.”

“Implementing and addressing this challenge necessitates specialised technologies, resources, and expertise,” he said. “Outernet London boasts an entertainment district featuring five distinct spaces, each equipped with LED and immersive sound installations. To effectively present content on the main space’s expansive 26,000-pixel LED canvas, a combination of specialised hardware and software is essential. Leveraging a real-time graphics solution is optimal, considering that the graphics machines are interconnected in a cluster configuration to accommodate the LED canvas. For content production and playback, Ventuz is employed extensively, while Unreal Engine is reserved for specific projects, particularly those emphasising 3D character animation.”

“We have relished the experiences of the other companies, also part of the accelerator,” said Oldfield. “We’ve benefitted from their expertise and experience, and it’s helped us adapt our solutions to this project but also our wider suite of products and the industry as a whole.”

The Accelerator Process

IBC’s Accelerator programme is unique in the industry for providing a safe place for a variety of companies, sometimes competitors, to come together and collaborate on solutions that benefit the entire industry.

Tootoonchi credits this environment for the success of the project. “This project would have been significantly challenging to realise without the support of the Accelerator programme, which played a crucial role in aligning stakeholders and technology providers to successfully execute the project,” he said.

“Certainly, we are enthusiastic about participating in upcoming Accelerator projects. Engaging in similar ventures offers valuable learning opportunities for our young business and immersive space. As we continuously evolve our approach to content creation and storytelling, being a part of projects that integrate emerging technologies in entertainment, film, video games, broadcast, and events allows us to refine our strategies and stay at the forefront of innovation in these industries.”

There have been many returning participants from year to year during the existence of the Accelerator programme. Tootoonchi sees the value of Outernet’s involvement and would welcome the opportunity to be involved with the programme again. “We are enthusiastic about participating in upcoming Accelerator projects,” he said.

“Engaging in similar ventures offers valuable learning opportunities for our young business and immersive space. As we continuously evolve our approach to content creation and storytelling, being a part of projects that integrate emerging technologies in entertainment, film, video games, broadcast, and events allows us to refine our strategies and stay at the forefront of innovation in these industries.”

Oldfield sees significant advantages to this way of working. “This project would not have happened if not part of an accelerator as each company would have only been working in their own silos,” he said. “Bringing the consortium together for the accelerator showed the sum is greater than the value of its parts and we can do more together. From a Salsa Sound perspective, we have had the opportunity and motivation to develop new real-time audio streaming capability using SRT directly from our MIXaiR software which positions us better for cloud and edge deployment for other projects too.”

The Showcase at IBC

Each Accelerator programme culminates in a showcase at IBC2023. Catch the Real-Time XR Sport Edge project, showcasing its Proof of Concept and findings at IBC2023, Monday 18th of September, from 12:15-13:15 at the Innovation Stage.

Oldfield describes what visitors to the Innovation Stage can expect to see, “We are showcasing our technology with live MMA fighters volumetrically captured with audio and video, rendered live in Unreal Engine and pixel-streamed for live location based-experience on a large screen and immersive audio system, and head-mounted displays,” he said. “We will present a live trial that incorporates this capture and delivery systems with multiple commentators with avatars and collective sound of remote fans around the globe.”

Benefits

Tootoonchi sees significant benefits from his company’s participation in the Accelerator programme. “Outernet’s efforts to attain real-time rendering of Unreal Engine content on screen epitomise our drive to exploit the spatial features we have in our spaces to their maximum potential,” he said.

“This project is ideal as proof of concept for us to connect live data and Unreal real-time rendered content therefore expanding our capability to deliver captivating experiences aligning with our ongoing mission to push the boundaries of audience engagement through innovative technological applications.”

Oldfield sees benefits of participation that go beyond this specific challenge. “We would definitely be interested in being involved in future Accelerator projects as it helps speed up development and delivery of new solutions and is a great opportunity to work with potential new clients and benefit from the collective brain of the consortium – there’s strength in team,” he said.

“Especially as a small start-up company, we really value this cross-pollination of ideas and the fast-paced development of the project. It’s fun.”

Read more IBC2023 Accelerators in full