- il y a 5 semaines
Beyond the Broadcast: Capturing and Delivering Immersive Live Events in VR
Catégorie
🤖
TechnologieTranscription
00:00Hello, everyone.
00:02So, thank you for being here.
00:04I'm Alexis Fabregoul.
00:06I'm an engineer in artificial intelligence
00:08and multimedia networking.
00:09And today I'm going to talk about shows,
00:12immersion, and also innovation.
00:17I think it will interest many of you here
00:19because we all missed events that were important to us.
00:24Be it a concert of your favourite artists
00:26or a football game.
00:27And that's why last year I co-founded Liveo
00:32where we imagined a new way, a unique way
00:35to live these events remotely
00:38while staying as close to the real thing as possible.
00:42And to do so, there is a very spectacular device
00:46which is virtual reality headsets.
00:49And I guess all of you here already know
00:52what is a virtual reality headset.
00:54But just to be clear,
00:55it is a niche market.
00:59I would agree with that.
01:01But we have already 10 million devices sold every year
01:06with a significant growth.
01:09And last year, there has been a big change.
01:13A new contestant.
01:15So we all hear that Apple launched their VR headsets.
01:20But the thing is that this VR headset
01:23is not for teenagers playing video games in their bedrooms.
01:26People who buy this headset,
01:28they are asking for high quality, immersive content.
01:33This is the first use case.
01:34And the only provider so far has been Apple.
01:38So when the headset was released in San Francisco for the first time,
01:43I went there a few months to observe how it could reshape the industry of virtual reality.
01:50What I saw is just what I said.
01:53This is a new audience that is buying VR headsets.
02:00And Apple is the only provider of high quality content.
02:04It has been 16 months since the release of their headsets.
02:08However, there is only two hours of content that is available 16 months after the release.
02:15So every user is asking for more and more content.
02:20And that's what we are working on.
02:22In fact, the Apple device is the first one of a new generation headset.
02:27I would say that the Quest 3 is in between an affordable and premium headset.
02:32But Meta is working on a premium device.
02:34Just like Pico, Huawei, Asus, Lenovo.
02:38They are all working to release their headsets.
02:42And Samsung with Google will have their own headsets launched in a few weeks from now.
02:48Today it's 40 million people who have a headset.
02:51And they use it and they need more content.
02:55So we are building an application, Live.io,
03:00where the users will find curated live shows
03:03with a constantly enriched library of content.
03:06We have been working with the French National Dance Theatre,
03:11with Dragon Productions.
03:13We are looking to worldwide famous shows.
03:17And we are working also for social features to share it with friends and family.
03:21If you have a cousin who lives abroad and with whom you share a passion for an artist.
03:28With Live.io, we will be able to connect together and experience an event
03:33and talk during the event and share emotions.
03:38To do that, to reach a realism and to be able to share it with your friends and family,
03:45we have to reinvent the whole broadcast pipeline.
03:48But also the thing is that with VR, the users can watch wherever they want.
03:54And with Live.io, they will be able to choose where they want to be at the venue.
03:59They can be on stage in the public.
04:02And to do that, the first thing is to think the capture setup.
04:07Apple has developed their own proprietary solution.
04:11So we spent also 10 months to find a good setup to reach the same quality,
04:16but for less expensive than what they are doing.
04:20Because we know exactly how they did that.
04:22But quality and cost is not the only factor.
04:26Also, in the music industry, we have to build something non-intrusive
04:30because we don't want anything between the artist and the public.
04:34So our setup can be safely placed anywhere in the public or on stage.
04:40We are not the first one to try to do immersive content in VR.
04:46But so far, the experiments were either low resolution, very pixelated,
04:52or it was 3D rendered and it looks like Fortnite.
04:56That's not what artists and producers want.
04:58With Live.io, we are doing 12K resolution.
05:01So with the stereoscopic, you have the immersion,
05:06you feel that you are at the stage, at the venue.
05:11And it is not expensive.
05:12It is not half a million to produce the content.
05:14It is the same cost as for a TV production.
05:18So maybe before I continue, I will do a demo with a few things we captured recently.
05:27So let me just connect to the screen.
05:36And now, finger crossed.
05:40Yes, you can see yourself on the screen.
05:43So we also cross the fingers to have the audio.
05:48There is a bit of flickering, maybe, I don't know.
05:52So let me first immerse myself in the Palais de Chaillot, the French National Dance Theater.
06:04So for a few minutes.
06:07I hope you can see what I see because I can't see you anymore.
06:11It will be very quick.
06:13But right now, I'm in front of the Eiffel Tower.
06:16I can see my hands.
06:17I can see the French National Theater on my left.
06:20And I really feel like the people are real.
06:24I have the 3D effects. You can't feel it right now.
06:28There is a Dolby Atmos soundtrack that is playing with head tracking.
06:33It is not very important for this scene.
06:36But if I go a bit forward, I can hear the music.
06:42I can hear the dancer moving on the stage.
06:45If I turn my head on the right, I could see a friend.
06:50If I have someone, for instance, my co-founder is currently in Canada.
06:56If he had a headset and if he was available, I could have just called him so he could appear
07:02next to me.
07:04But the sound is fixed, but all around me.
07:09And you can have the best place at the venue.
07:13I'm just next to the dancers.
07:16With a high frame rate, it's very, very smooth.
07:20Maybe it's a bit not that smooth on the screen, but it's because of the mirroring.
07:27And before I continue with the slide, let me just show another use case we did with the International Automobile
07:36Federation.
07:37We went on a car race in France.
07:43So here, I'm in between the cars.
07:47I'm living the life of what we call a marshal.
07:49So the people who are in charge of the security just before the race.
07:58So here's a marshal.
08:00I can just see the cars.
08:02There is the sound that is...
08:04Oh, there is also music for this one, but the sound of the motors is very important for people who
08:10enjoy cars races.
08:15I can be inside the car also.
08:19So if you want to watch a bit more, just come at the booth.
08:23I will have to continue with the slides.
08:25But I hope you had a few glances, a little glances on what it can be.
08:34Thank you.
08:39So you had not the opportunity, just as I did, to hear the Dolby Atmos track, but it is very
08:46important.
08:46In the cinema industry, we say that sound is responsible for half the immersion quality.
08:52I would say that for virtual reality, it's even more.
08:56So what we do is that we have 360 microphones that record at each point of view the sound of
09:03the location.
09:05And we mix it with closed source mix.
09:08So we have either what we call ambisonics or Dolby Atmos mix at the end.
09:14And with the Vision OS, but also any platform of VR headset, we can then interpret that format and have
09:23a head tracking.
09:24So when the car is on my right, I can hear that the car is on my right.
09:29And when I look at the car, then the sound is in front of me.
09:33And that's very important. And so far, no one has been doing that except Apple and us.
09:40Also, we are a bit different than what is doing Apple because we are using AI and Apple is not
09:47that good for now in AI.
09:49I'm sure they are working a lot on that.
09:52There is a lot of applications for AI in the post-production world.
09:57I will not dig into every aspect.
10:01The main one, I would say, is for now the super-resolution models for upscaling the image.
10:07And this is very important for two reasons.
10:10First, you can't just have a 4K content that is very fine for TV and cinema.
10:17Looking at 4K content in VR headset is very pixelated because you are looking at just a portion of the
10:24image.
10:24What we are doing is 12K video.
10:28So you could either try to find sensors that are doing 12K.
10:34There is only one on the market today.
10:36But it can't be output in real time.
10:39The max resolution we can retrieve from a camera today, even the most professional ones, is 4K.
10:46So we managed to work with 4K content.
10:51And we have our AI models that are doing super-resolution to transform a 4K video to 12K video.
10:58And we compare it with what Apple did in Paris with the same shot.
11:02And we are slightly better than what they have without AI.
11:08And then, this is important. Why? Because as we can retrieve the 4K content in real time.
11:14Who say real time? Say live events.
11:17You could attend a football game, a car race, a concert in real time with just one second latency.
11:26Because our AI models are running in the cloud with our NVIDIA partnership.
11:31And because we are doing that, we can interfere the data in real time.
11:38Apple is forced for now with their solution to first record, then post-process, and then publish the video.
11:46So we think we'll be the first one to be on the major events in real time with high quality.
11:54Meta has been doing real time events, but it's very low resolution.
12:00This is an incredible opportunity. I hope you already understood that.
12:04But if you are an artist, you can now engage with international fans in the best way possible just after
12:13the on-site presence.
12:14But we can't be always on-site. For instance, when the PHD won the Champions League,
12:21a lot of fans from Decay could not attend the games if they had the opportunity to see it in
12:29VR.
12:30Just for that, they would have buy a headset that is very expensive.
12:35But also, it is an opportunity because very few artists have been on this platform for now.
12:41And they will be able to do online ticket selling and merchandising.
12:47In fact, it's a new distribution channel.
12:49Who says new distribution channel?
12:52Says that we have to control the whole pipeline.
12:54How it works is that we are sending people on-site with our own cameras and microphones.
13:00And then we post-produce the content.
13:03We deliver it on our application and we share revenue with the people in the music industry
13:09who is in deep need to increase their revenue and to renew their audience
13:15because the young public is going less and less to the venues.
13:21But I talk a lot about our B2C application for live shows.
13:26But as you saw, we are also doing car races
13:30because the technology we are developing, the expertise we have can interest a lot of companies.
13:36It can interest sport leagues. It can interest brands who are doing catwalks.
13:41In fact, anyone who has content to broadcast can be interested by our services.
13:47The fact is, our live view platform is for live shows.
13:51So anything else, we will be doing it through white labeling.
13:59So before your questions, what we've been doing is first to design the high-end capture setup
14:07to reach the Apple quality.
14:08We managed to work out the post-production workflow because even today it's very hard
14:14to be able to correctly correct the distortion, to align the stereoscopy,
14:19to use AI to be able to reach the final format we have.
14:25And we developed an application with a player to play the video, the spatial audio.
14:30And after all that, we started this spring to capture real events.
14:37So with the French National Dance Theater, with the International Federation of Automobile,
14:42with the Dragon production, and a lot of big groups that I'm not authorized to talk about right now.
14:50But we have things that are planned for this summer.
14:54And we hope to publish our application before the end of the year, once we have a consistent library.
15:02The vision we have for the coming years is to be the major actor in immersive videos.
15:10I think there is a huge opportunity. We have to make choice.
15:14But to help also the young people who today are every night consuming Netflix and TikTok.
15:22At the same time, there is no place for quality content.
15:26What we offer is a way to immerse in quality events with no distraction.
15:32And I think it's very important to help these people to discover again culture,
15:38not only of their country, but of every country.
15:43Thank you very much for listening. I'm eager to hear your questions.
15:47Feel free to contact us. Feel free to go to our booth to try the experience.
15:53Thank you. Great. Great.
15:56Very interesting. Yeah, very interesting product you have there.
15:59Any questions for our speaker?
16:02I have questions.
16:03Yeah.
16:04So if I understood correctly, you're basically quite ahead of the game compared to the competitors, right?
16:13I mean, you're basically built a platform and you're also providing a service to film and capture content
16:19to then make it available on this platform like a library, right?
16:22And apparently, you're the first guys doing it, according to you.
16:25You said that the big giants are looking at it like Apple, but...
16:27Yeah. I mean, we are doing...
16:31Apple has been very good to push forward the quality.
16:35Yeah. It was very needed. So we spent time to go in the same direction and to reach that.
16:41Yeah.
16:41Meta has been pushing the live events, but not in the same quality.
16:44Yeah.
16:45So we've been doing to do all of that live and good quality.
16:51So our AI models I talk about, they are not quite running in real time right now.
16:56So we still have some work to make it real time.
16:58But I think it's the best strategy to be the first one to do so.
17:02Okay.
17:02So far, we are able to do just like Apple, good immersive content, almost in real time, but not totally.
17:10Because I'm surprised that Apple only has two hours of content for...
17:14Yeah.
17:15Wow. It's surprising.
17:17All right.
17:17Last question then.
17:19Ah, we have a question.
17:20Yeah.
17:20So do you need your cameras or specific cameras or it can be its regular cameras or...
17:28So this is a setup we spent months to build with partnership.
17:34I would say on the shelf components, but it is a custom rigs with several cameras that we synchronize with
17:41specific mics also and specific lenses.
17:45The whole combination is very important.
17:47We tried dozens and dozens of setups before finding a good one.
17:50And we are very satisfied now.
17:53And also it is not too expensive.
17:56It doesn't need millions of investment to do so.
18:00Yeah.
18:00Yeah.
18:03All right.
18:04Thank you.
18:05Are you using professional grade cameras or like something like a, you know, DSLR kind of with cameras?
18:15We are using professional grade cameras that are used for cinema.
18:21Okay.
18:22We...
18:23Okay.
18:23Very, very fast.
18:24Very fast.
18:25Yeah.
18:26You talked sports entertainment.
18:28Yeah.
18:28Where is this headed long term?
18:30Long term, our platform will be entertainment.
18:34We hope to be the main actor to help other industry like sport to get a presence in the immersive
18:41world.
18:42But I think, as I said, catwalks and a lot of events could be interesting in an immersive format.
18:50Great.
18:51And you guys are a French company, right?
18:52Yeah.
18:53We are based in Paris.
18:53Ladies and gentlemen, Olivia.
18:56Thank you.
18:58Thank you.
18:59Thank you.
Commentaires