In 2017, UFA celebrated its 100th anniversary, making it one of the oldest entertainment brands in the world. With more than 2,500 program hours broadcast each year, today's UFA Group presents itself as a high-performance program creator who has steadily expanded its market leadership as a film and television producer in Germany in recent years. UFA programs inspire and inspire millions of viewers every day.
The production units UFA FICTION, UFA SERIAL DRAMA, and UFA SHOW & FACTUAL operate under the umbrella of UFA. The extensive portfolio offers a unique variety of programs. High-quality fiction productions such as TV movies, series, and TV events are realized by UFA FICTION. Journalistic-oriented formats, reality, game, panel, dating, comedy, and music shows are all created by UFA SHOW & FACTUAL. UFA SERIAL DRAMA is a leading producer of industrial series productions.
UFA SERIAL DRAMA is probably the only German production company, with an embedded technology department. We see ourselves as the keepers of know-how within UFA. Most German production companies run a hire-and-fire policy. This means that the technical know-how keeps flowing away and must be rebuilt at the start of every production. Our technology department, on the other hand, is a bit like the library in the movie „The Name of the Rose“ We have a few „bad“ books, containing the mistakes we made of which we learned how not to do it, and many books on how to do it right and effective. This enables us to handle such large quantities of media production.
How it started and why
We are not just keepers, we are also pioneers. Not only do we support our current productions, but we also keep an eye on the market for emerging new technologies and forms of media. Obviously, XR has been one of the trending topics for quite a few years now.
The XR market is growing steadily. By 2021, a global market value of $ 215 billion is forecast.
https://www.statista.com/statistics/591181/global-augmented-virtual-reality-market-size/
The XR video entertainment market segment is expected to have a market share of $ 3.2 billion. This market offers new opportunities for UFA "beyond entertainment" in the areas of industry, e-learning, etc.
We figured, if UFA wants to enter this market, we need suitable means of production that will do justice to the "UFA genes". We needed a tool, which would allow to bring the core competencies of UFA -storytelling and the conveyance of real emotions- almost seamlessly into the XR world. We soon decided that 360° did not cover our needs. Working with animated CG Avatars, was also not an option. We had the opportunity to visit the Microsoft Volumetric Studios in Redmond, US and do a volumetric shoot in this facility. Volumetric video technology makes it possible to capture and digitize people and objects. In a volumetric studio, people and objects are captured from all sides, not to create regular movies, but realistic, natural-looking, authentic 3D representations - a real-time 3D scan. It creates Holograms, a sequence of 3D geometries and textures, of real people and objects. Upon returning to Germany, we found out Fraunhofer is working on volumetric technologies as well. We encouraged the Fraunhofer team to take their technology to the next level -with our help.
1st Production “Gateway to Infinity”
We convinced the Fraunhofer Institute of the necessity to produce a proof of concept for the “3D Human Body Reconstruction (3DHBR)” technology of the Fraunhofer Heinrich Hertz Institute HHI. The studio set-up for this production was still rather experimental. We used a set-up of 16 cameras, which allowed to shoot a range of about 220°. Upon completion of this production, it was decided to abandon the green screen set up in favor of a completely white studio. This white backdrop serves a dual purpose. It is both lighting and provides a perfect backdrop for a luminance keying workflow. We basically had 3 systems (green screen, stereoscopic cameras and light) competing for the limited space in the studio. By eliminating the green screen, the amount of systems was reduced from 3 to 2. This way the light is more even and green spill is no longer an issue.
2nd Production “An Entire Lifetime”
Our second production “An Entire Lifetime” was conceived as part of the exhibition "The UFA - History of a brand" in the German Cinematheque. In this VR Experience, we follow actress “Maria” (played by Franziska Brandmeier) and Director “Fritz” (Herbert Knaup) through the history of cinema. We experience life on a film set of the early 1920s. In the course of the VR experience, the viewer experiences how technical tricks and illusions were used to deceive the audience even then. Surprisingly, however, this set world turns out to be an illusion - and Maria takes the viewer on a visual journey through the history of UFA.
The Studio set-up for “An Entire Life” was already very close to the Volucap Studio. It was a bit smaller and we encountered a few acoustic issues. The closed cylindrical construction acted as a giant acoustic reflector. This problem was solved in the Volucap studio by incorporating materials with the correct acoustic characteristics. This production was awarded at Laval Virtual, a leading global conference and exhibition of virtual technologies and applications in France. With the Laval Virtual Award, the international jury of experts honors projects characterized by high technical quality and innovation.
So why did UFA become a stakeholder in the Volucap venture? We normally go by the motto: “no need to buy a cow, if you want to drink a glass of milk”. We decided in this case we need to make an exception to this rule. This is why:
Volumetric production facilities are in short supply. Having direct access is a big plus.
Thanks to UFA's involvement, the Fraunhofer Institute was able to create a full-fledged production tool from a basic volumetric video research project. Since this innovative technology still offers a lot of potential for further development and optimization of the workflows, we must not leave the field to the engineers and should be able to continue to influence its development. The partnership with the Fraunhofer HHI, Arri and cooperation with Microsoft are the best conditions to push the technical limits even further.
UFA needs experience and talent in the field of VR/AR. The public funding conditions require that the Volucap Studio offers opportunities for research and experimentation. By participating in Volucap Studio, UFA can take advantage of these opportunities. Due to the strong commitment of the Filmuniversity Potsdam to engage with the technology and the medium VR/AR within the framework of this research project, a well-trained talent pool emerges which develops a relationship with UFA via the Volucap Studio and is available to tackle future projects.
The VR/AR market is very complex. UFA needs a network of partners from technology and industry and vice versa. These partners are gathered at the Babelsberg Mediatech hub -of one of Germany's 12 digital centres supported by the Federal Ministry for Economic Affairs and Energy, the meeting place of Germany's digital elite. With the Volucap Studio at the heart and as USP of the Mediatech hub, UFA moves directly to the heart of this network by joining this partnership.
Creative and technical takeaways from our XR productions so far
Creative takeaways
The fact that -at that time- we did not reach the level of technical perfection we would like to, didn't not seem to bother the audience. They are surprised at how realistic and near the actors are. Although computer animation can sometimes achieve a higher level of quality, it still remains animation and not real action and emotion. The audience realizes: One is synthetic, the other is real. We suspected the appeal this has with our audience, but not to that extent. If we manage to raise the quality bar for realistic volumetric video, the experience will become even more immersive.
Many classically trained filmmakers struggle to make the transition to XR. When a new kind of media emerges, creatives tend to build on what they already know. In the early days of filmmaking, they used Theater as a reference. Early films were short films consisting of one long, static, and locked-down shot -like a play in a Theater. Each film ran as long as there was film in the camera. It took filmmakers almost 10 years to figure out the principle of editing. This is where we stand with XR. Directors and Storytellers need to develop a completely new language and leave behind the “old” one. We should embrace all these experiments as, without them, the industry would not progress. At the 75th Annual Venice Film Festival, the co-curator of the festival’s VR program, Michel Reilhac, said, ‘if cinema is the art of playing with time, VR is the art of playing both with time and space. And we are just beginning to understand what that means.’
And these are a few of the things we have learned so far.
Every Character is a Hero
XR calls for each character on stage to have a full storyline for the whole time they are on stage since the director cannot control where the audience will choose to look or which character they will decide to follow. Of course, the director can utilize such tools as sound or visuals to draw the audience to a particular point, so-called „calls to action“. But even still, there is no guarantee the audience will respond to these signals and that is what the director should prepare for. The double-edged-sword of control is it may feel inauthentic. Since XR is the medium about your viewer’s freedom, I suggest you embrace it. Instead of spending a considerable amount of time and energy attempting to control something that you fundamentally can’t, you should focus on delivering a world of wonders — a unique experience that makes people feel.
In film where the director utilizes editing and camera to control what the audience sees and who it follows, the hero is often the only character who is given a full story arc. In some cases, a supporting character or two will also have their own arc to complete, but the other supporting roles only serve to push the main characters’ journey of discovery forward and to completion. But since it is different in XR, it is imperative that the writer lets go of the concept of the single hero or protagonist. Each character will need to be developed fully for the time they are on screen.
So, approach each character as a hero and develop his her storyline thoroughly, take your time, trust your audience to find the story, and let them become active agents and co-directors of the story. If you do all this, then you will have successfully cracked storytelling for XR.
Take Your Time When Telling the Story
Nowadays, not one second is wasted or dallied with when it comes to film and the cinema. Each moment on-screen counts and is as precious as gold. But when it comes to XR, for the reasons mentioned above, it is the overall scene that matters, not each single moment. So, in XR, the pace is much slower than that of film. This allows for the audience to look about the stage, catch up with the action and even follow it if they miss a beat. Slow down the unfolding of events and let each scene breath, then you can trust your audience to find their own story.
Trust Your Audience
Discovery is a vital component to keep in mind regarding storytelling for XR. XR asks of us as storytellers and directors to allow our audience agency and participation. The viewers are an active participant in how they will experience the story, leaving each to walk away with an entirely unique experience. This is what we should strive for as XR content creators. But this asks of us to relinquish control and trust the audience to find their own story, which is difficult for filmmakers since many of us have spent years harnessing the control freak within us because that is what is needed when it comes to movies.
The Audience is Not Only a Character, But Also a Co-Director
The beauty of allowing the participant to discover the story is that the experience will not only be utterly unique for each and every participant, but also for each and every time they partake. Making it something that we will want to enjoy over and over again. This applies maybe less so to 360 Video since it only offers one level of agency to look around in 360. Virtual Reality, however, permits for more active interaction because it enables spatial interaction and manipulation. The 4th wall can be completely torn down. In XR we can remove any barrier between the world of the experience and the user.
It is this which is one of the biggest promises of XR, especially regarding monetization of this medium. XR encourages the audience to watch the experience multiple times.
Go Easy on the Audience
Our viewers, most likely, aren’t used to XR films and for most of them, this could be their first VR experience. Because of such a huge jump in information density from tv screen to immersive environment, think about how you would gently ease them into the story.
XR is so intense it needs to be experienced in bite-size chunks, two-to-three hours of high action or drama would be too much in XR. Instead, you’ll revisit the experience again and again. Thus, a longer experience divided into chapters is the right way to go.
For decades we have been passive observers, sucked in by linear, director-crafted narratives. In XR, audiences have the freedom to choose where and what they look at, with almost all of the traditional director's tools taken away. This requires a different focus and participation compared to traditional films. Over the length of a feature film, this will be very demanding for your audience.
Test, test, test.. and test more.
Product and services are always tested before release into the public. So should your VR creation. Since we have established a developmental relationship with our work, we are not the right person for testing our design. We might be too critical or forgiving of our creation. Ideally, we should show our work to random people that we don’t know since they have no vested interest in outcomes and won’t stroke our ego. Their observations, feelings, and especially reactions watching our creation are pure gold.
Technical takeaways
XR is a multidisciplinary challenge
There are crucial differences to film production: The post-production workflow is more like creating a computer game. We build a CG world in which the volumetric characters act. We use tools from the gaming industry such as Unity, Unreal or Cry Engine. We also process dialogues, atmos, sounds, etc. in these engines to create a binaural sound. This means we need a much wider range of skill sets. We need to set up a production pipeline which combines classical filmmaking know-how, with coding, CG and many others
What is it like to shoot in a volumetric Studio?
We have a fixed studio set-up. This means there are no set-up times for camera, light, sound, etc. Apart from that, it is pretty similar to a green screen shoot. The challenges for actors and director of working in a void space are similar too. That makes everything slightly more complex. We support this with very precise pre-visualization using storyboards or 3D-animatics in 3D as visual planning aids for shooting.
Here’s looking at you.
At some point, we will need to be able to interact with the recorded figure. One of the basic needs is to be able to establish eye contact with the volumetric figures. In the distant future, it might even be possible to develop interactivity controlled by artificial intelligence.
Volumetric Video is a work in progress
Improving quality and production speed is an ongoing challenge. We have no technical standards to go by.
In some cases, the huge amount of data still limits the distribution to local based entertainment. But if we cannot distribute the product to the audience, it will have no chance of success. So, the question is how to compress a 3D format? Fortunately, the Fraunhofer Institute -the data compression specialists- is one of the Volucap stakeholders.