VIRTUAL PRODUCTION A Global Innovation Opportunity for the UK - National Centre for Immersive Storytelling
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
VIRTUAL PRODUCTION
A Global Innovation
Opportunity for the UK
StoryFutures Academy Immersive Skills Report 2021
By Prof. James Bennett, Dr. Claude Heath,
Fiona Kilkelly and Prof. Peter Richardson
National Centre for Immersive Storytelling
VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK
CONTENTS
EXECUTIVE SUMMARY 04
METHODOLOGY 05
SECTION 1: WHAT IS VIRTUAL PRODUCTION? 06
SECTION 2: THE VP SKILLS MANDALA 08
Case Study: Hazfilm 10
Case Study: Wilder Films 13
Authors SECTION 3: THE OPPORTUNITY 14
Prof. James Bennett is Director of StoryFutures Creative Cluster and Co- Case Study: Space Film & VFX 16
Director of StoryFutures Academy and the StoryTrails UnBoxed project. He
is Professor of Television and Digital Culture at Royal Holloway, University of SECTION 4: THE CHALLENGE 18
London. When he’s not immersed in the future of storytelling, he likes running, Case Study: ILMxLAB 20
sea swimming and Vegemite.
SECTION 5: THE SKILLS GAPS 22
Dr. Claude Heath is Senior Research Fellow at StoryFutures in the
ethnography and mapping of creative workflows in the media industry, using 5.1 Technical Skills 22
participatory co-design, drawing and visual research methods. His research Communications 22
background is in human-computer interaction and cognitive science. He also
has an international track record as a visual artist with work held in national Integration of Film-Games Cultures and Tools 22
and private collections worldwide.
Workflow Integration 24
Fiona Kilkelly is Acting Head of Immersive at StoryFutures Academy, the
VP Supervision 24
UK’s National Centre for Immersive Storytelling run by the National Film and
Television School and Royal Holloway, University of London. Fiona established VP Production Planning and Budgeting 24
Immerse UK, the UK’s immersive technology network in 2016, is Co-Founder
of XR Health Alliance UK and has worked in the strategic development of
5.2 Creative and Communicative Skills 26
emerging technologies within the creative industries for over 20 years. Universal 3D Skills for VP 26
Prof. Peter Richardson is Professor Of Creative Industries at Royal Holloway, Common Art Department Games Engine Skills 26
University of London, he has been Principal Investigator on three major EU-
funded projects researching emerging cinematic technologies. Peter has 28 Plugin Design Skills 28
years experience as a Director of music videos, commercials, documentaries, Technical Direction (TD) Skills 28
opera, visual effects supervision in film, and recently as Producer and
Executive Producer on VR and VP projects. Asset Building Skills 28
General Coding / Engineering Skills 28
Technical Artist Skills 30
Case Study: The Garden Studios 31
© The Authors, StoryFutures Academy, copyright 2021.
SECTION 6: Interim Conclusions 32
Cover image: © Wilder Films / Fireworks
This project has received funding from the AHRC-UKRI Audiences of the Future Case Study: The View from the Games Industry 33
Programme. Grant Reference AH/S003622/1
The text of this document (this excludes, where present, logos, production stills)
may be reproduced free of charge in any format or medium provided that it is
reproduced accurately and not in a misleading context.
The material must be acknowledged as The Authors’ (Bennett, Heath, Kilkelly,
Richardson) copyright and the document title specified. Where third party
material has been identified, permission from the respective copyright holder
must be sought.
ISBN: 978-1-8384779-2-9
3
EXECUTIVE SUMMARY METHODOLOGY
Virtual Production (VP) is becoming a ubiquitous property in the form of tools, methods and in our This interim report is primarily qualitative and informs our understanding of VP with six of our
methodology for the achievement of high-quality film, world-renowned ability to make the highest quality based on the experience of Virtual Production interviewees drawn from the crew. The detailed focus
television and media productions. A game-changing content. This is not simply a technology opportunity, from the perspective of film and television. Semi- on this case study helps to frame our understanding
development, VP combines virtual and augmented it is one that brings together Science Technology structured interviews with 25 leading industry of other VP projects, in terms of the differences that
reality with computer generated imagery. All this is Engineering and the Arts: a true STEAM opportunity professionals were conducted over two months in exist between productions and the skills gaps that
made possible by real-time game-engine technology. for our Creative Industries. the summer of 2021. are common to all productions.
Exemplified by high end Hollywood productions such Throughout this research we found an industry Each interview lasted between 30mins and 2 hours The qualitative data analysis tool MAXQDA2020
as The Mandalorian and in television by the BBC’s that recognised that whilst current costs were and most were one-to-one meetings, while some was used to analyse the interviews, developing
Virtual Production presentation studio for the Tokyo high, Virtual Production would be normalised and were joint interviews (ILMxLAB, DNEG). At least two themes presented within the data, sorted and
Olympics, Virtual Production takes traditional screen correspondingly cheaper within 5 years. However, individuals from each film discipline were interviewed. clustered by discipline, role, Department, skills,
media production methods to a new dimension. ‘hands-on’ experience is still low. Appetite for This first set of interviews are part of a continuing challenge and opportunity points, pain-points,
VP capitalises on technology developed for games training is therefore veracious: Epic, Screenskills, series of consultations with industry planned for touchpoints, educational background and training.
development, notably Epic’s Unreal Engine, and Digital Catapult, National Film and Television School 2021-2 and beyond, regarding the disruptive impact A review of current vacancies and job descriptions
delivers stunning, photorealistic sets, environments StoryFutures Academy, and others, have all launched of VP on the media sector. from large and medium sized companies, further
and audience experiences in real-time. training programmes in 2020-21, with ScreenSkills informed and highlighted the reported gaps in
seeking to establish national training standards Our primary case study, Fireworks, the VP short film
VP offers efficiencies and enhancements for workforce competencies.
for VP. As the then Minister for Creative Industries directed by Oscar winner Paul Franklin, significantly
both live action (Production) and visual effects
(Postproduction). VP allows for a reduction of the Caroline Dinenage commented, to ensure the UK
costs and risks associated with VFX, which typically stays ahead of the creative curve there is a need to
account for 10-20% of overall budget: virtual sets “develop our world class screen workforce [to] help
not only save on production time and costs, but the UK’s film and TV sectors build back better”.[1]
might also be more sustainable, reducing carbon This Interim StoryFutures Academy Skills Report
footprint by removing travel and transportation provides a map of the VP skills needed across the
costs of location filming. Instead of flying crews, film and TV industry. As with the roll out of the
equipment, actors and sets to locations across the technology itself, the skills mapping will need to
world, Producers can design and build locations in grow more widely to include games, theatre, and live
the computer, which are then played back on large production, and also look more broadly across the
LED screens known as Volumes. UK creative industries. Our Report identifies both
Such Volumes are being rolled out across the UK to the hard and the soft skills demands of the creative
service the growing need for this technology. This industries required to translate innovation practice
represents a massive investment in the technology for into economic growth. It shows how VP changes
companies such as Netflix, Sky, ILMxLAB and Warner the way key Departments operate - internally and
Bros. This is because the opportunity is also massive. with other Departments - during Preproduction,
Traditionally the technologies and methodologies of Production and Postproduction.
the games and film and TV industries have developed However, the report has identified many gaps and
in parallel, with very little crossover from one field pain points in the workflows and pipelines of both
to the other. With VP the opportunity exists for a Production and Postproduction talent in the UK,
crossover of competencies allowing companies to revealing a significant skills shortage. Demand for
expand from games (UK 2020 video game market talent and training far outstrips supply. There is a
worth £5.3 billion) to high-end TV (UK 2020 spend on clear need for highly creative, technical artists who
HETV production: £1.49million) to film (UK 2020 total can write code, problem-solve and communicate
spend feature film production: £1.37billion) and into effectively with creatives while fixing complex
immersive (UK value 2021 prediction: £5.7billion). With technical issues; all within the fast-paced film and TV
the potential to benefit all these sectors individually set environment. This shortage needs to be tackled
and collectively, it is crucial that the UK ensures that it urgently if the UK is to leverage its reputation for
has not only cutting-edge VP infrastructure, but also innovation and creative excellence across film, TV,
the skills capacity to develop its own new intellectual VFX, games and computing.
© Wilder Films / Fireworks
1
Quoted in Televisual, 10/02/21, “National Standards Plan for Virtual Production”, https://www.televisual.com/
news/national-standards-to-be-developed-for-virtual-production/
4 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 5
SECTION 1
WHAT IS VIRTUAL PRODUCTION?
Fundamentally, VP ‘rolls up’ the previous linear that only engages in the filmmaking process VP is a complex array of techniques and technologies, incorporated into virtual or VFX people, creatures
Production process of film and television into a within a linear sequence. Most radically, traditional and consists of four main areas: Visualisation, Motion or other assets in order to animate them. Mocap
dynamic Production world, bringing together Postproduction roles, skills and processes have Capture, Hybrid Camera and LED Live-Action: first came to prominence in the 1980s, and whilst
departments that had previously worked sequentially moved into Preproduction and on-set, whilst, at the an established part of the film industry it remains
1. Visualisation: Using 3D VFX assets to visualise and
into simultaneous interaction (see Figure 1). There same time, they retain a role in a final polish of a film challenging both technically and artistically.
plan a task. Various forms are available and include:
is not one node, department or role on the Mandala before release.
a) P
itchvis: where imagery and scenes are 3. Hybrid Camera (‘simulcam’ green-screen hybrid):
created from a project to help raise finance or This is compositing (combining) digital VFX with
communicate a director’s vision. live-action camera footage in real-time. Simulcam
Figure 1: The Virtual Production Pipeline was originally developed by companies such as
b) P
revis: allows use of ‘virtual scouting’ for shot Weta Digital (Lord of the Rings) and by producer/
and sequence planning where a digital set director James Cameron (Avatar). It offers improved
or location is created allowing key crew to results compared to shooting with a green screen
experience and interact with a scene via a head background, because visualising the digital and
mounted display. physical simultaneously helps directors and actors
c) Postvis: merges temporary VFX with shot to gain a better spatial understanding of the scene.
live action which can provide placeholder 4. L
ED Live-action: This replaces shooting against
sequences for editorial. green screens with shooting against LED panels
d) T
echvis: combines filmed footage with virtual or “Volumes” displaying the final-quality VFX.
assets often used to prove physics for complex This is an advance on 2D video projection and
scenes, such as explosions etc, where camera requires rendered scenes to be ready to be ‘played’
n traditional filmmaking (Above) pipelines move through
I data or lens choices may affect believability. out onto the LED wall via a game engine. This
clearly identifiable and demarcated phases. method shifts the emphasis from Postproduction
e) Stuntvis: the visualisation of stunt sequences.
In VP on the other hand (Below) these phases overlap and to Preproduction, since key creative decisions are
iterate, and an extended duration of Preproduction. 2. M
otion Capture (‘Mocap’): Camera and computer made before principal photography begins, allowing
capture of human motion which is then crews to shoot with completed VFX sequences.
LED Live-action volume: panels of LED lights display the scene, at the
StoryFutures Academy/Final Pixel VP Bootcamp, 2021 at ARRI Mixed
Reality Studio. Image courtesy of ARRI / CT Group.
6 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 7
ala
and
M
kills
S
n
io
uct
od
l Pr
ua KEY
irt
SECTION TWO Roles
V
e
Th
Understanding the world of VP
2:
re
gu
The VP Skills Mandala
Fi
Skills
The Virtual Production Skills Mandala (see Figure 2)
represents VP-specific roles and skills rather than
traditional filmmaking. It builds upon work by EPIC
visualising the cyclical nature of the VP workflow.2
The Mandala was shown to interviewees towards
the end of each meeting and developed with their
input. The diagram is therefore iterative rather
than exhaustive, with relationships between roles
and skills not fixed. It captures different views of
organisational structures, workflow cycles, and
role descriptions while mapping these into the VP
world as it emerged across disciplines.
Creative and directorial roles are shown as blue
circles - these roles may help to shape but may
not directly handle the data produced for the
Virtual Production end product. Alongside
these are the more technical roles - shown as
yellow circles. Arrayed alongside the roles are
the specific skills, aptitudes, and abilities that
may, or may not be, involved in the carrying
out of that role, depending on the production.
On this first mapping, the greatest
concentrations of new skills associated
with VP can be seen, unsurprisingly, in the
emergence of a ‘VP Department’ and an On-
Set VFX Department. But it also significantly
impacts Production Design, whilst all other
Departments are forced to adapt existing skills
and evolve new ones, including finding new
ways to communicate between Departments.
Reading the Mandala like a compass, we can
observe the preponderance of creative, non-
data-handling roles (blue) in the North East
quadrant, whilst in the East and West, technical
data-handling roles (yellow) tend to dominate - as
the Virtual Production Department prepares the
final assets and scene modelling and animation.
Compared with linear production, 3D modelling and
other technical skills appear earlier on in the cycle. For
example, in Fireworks, Unreal Sequencer was used by
the Director to generate storyboards for the film and
the Production Designer created 3D SketchUp models
with detailed architectural and texture references for
the VP Department. These assets may not have been
directly utilised in the final product, but help to frame
the film during all phases of the Production.
Download a copy of the Mandala here https://www.
storyfutures.com/uploads/docs/VP-Skills-Mandala_
November-2021.pdfNovember-2021.pdf
2
The Virtual Production Field Guide, Volume 2, Epic Games, 2021, p.148. See:
https://cdn2.unrealengine.com/Virtual+Production+Field+Guide+Volume+2+v1.0-5b06b62cbc5f.pdf
8 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 9
The VP Skills Mandala
In the South East, at approximately 135 degrees, and The Preproduction phase
South West (200 degrees) we find the Technical of the project starts with CASE STUDY | HAZFILM
Artists roles unique to VP. These roles are shown as the Writer, Director
yellow nodes with green fringes. This denotes their and Producer
distinctive blend of skills from both ends of the beginning to shape Name of Interviewee: to be an intern or a runner, make the Director’s
spectrum of filmmaking skills. Technical Artists the central ideas Haz Dulull or Producer’s coffee. Now I think there’s a place
have been described as embodying the meeting and techniques where interns can be hands-on with a Virtual
Job Title Production such as moving lights or doing a lot
of mathematical and creative skills, in activities that will be needed
that merge creative and technical problem- for the film. At Director/ Producer of data transferring, managing the Unreal engine
solving through collaboration. approximately 45 scenes, or doing backups on the day.
Company
degrees the Art
The metaphor of reading the map of VP skills as Hazfilm The Skills Challenge of VP
Department becomes
a compass does not, however, provide a complete
more closely involved, It is all in the prep, we Previs the hell out of a
understanding of VP. The compass points help Virtual Production Role
with the Production shoot. But not only do we do pre-vis, we tech-vis
identify regions of production, but the Mandala also
Designer starting work on their There isn’t really a solid job description of what a as well. Which meant that we go onto the Virtual
describes a multi-layered process cycle that unfolds
ideas for the look and feel of the project. At or after Virtual Production Supervisor is, because it’s fairly Production stage, we measure it, we get floor
over time. The radial format conveys the forward
this stage the Director of Photography (DoP) may new … essentially [it is] someone that understands plans, we rebuild that stage in Maya. We even
movement of the project as a whole, and different
become involved in informing the decision-making real-time, someone that understands visual had footage playing in Maya on a make-believe
Departments appear in more or less traditional order,
related to how best to achieve the desired look and effects, but also someone that understands how to LED screen to really plan it. The thing about
but the analogy with linear production ends there. As
feel. Other early contributions will follow from the communicate to the director on how to get his or Virtual Production, amazing as it is creatively and
one practitioner described it:
Virtual Production Department, about how best to her vision using VP. technically, it’s very expensive. You can’t really
‘The workflow... is not linear... It’s all about approach the filming from a technical, modelling, have the luxury to experiment, spend a couple of
As a Virtual Production supervisor on a VP shoot,
being able to jump between tasks and having animation and aesthetic perspective. hours to see what you can do and then do a tech
you are jumping between the actors, DoP, digital
the responsiveness to manage the competing recce day. There’s a lot you learn the minute you
village and the Virtual Production team. The VP
pressures. The workflow is more like a jumble or come on the set and the minute you start putting
At or before 90 degrees supervisor and the VFX Supervisor work in hand
bundle of jobs in the centre, all squashed together, on the graphics.
on the VP Mandala, but you are also working hand-in-hand across all of
and we have to find a way to handle each project
the Production those teams. That’s why roles like Virtual Production There’s a lot of decisions you can make instantly…
most efficiently; a core of tools and processes,
phase begins. The Supervisor are critical and we are going to see the thing I love about VP is that when you’re
with moving parts all around it.’
Virtual Production demand for this role pop up in quite a lot of places. relating to production and executives, sign-
The VP methodology demands that Departments
Department offs are much quicker. With Virtual Production,
move in and out of the active centre of the The Changing Role and Relationship to
creates the 3D because you know your shoot date is in two
production at different times and for different other Departments
digital assets that months’ time, and you have two months to make
purposes. Rather than being linear or classically
will be used to set Relationships between the Departments are a decision creatively, what the environments
cyclical, the workflow is like a condensed and
the scene, as both slightly different and we have a new Department are, what the look is, what the film is, it forces
evolving bundle of tasks in the centre of the
backdrop to filming as well in VP, the Virtual Art Department, or VP executives and decision makers to work with
production, to be dipped into as and when needed.
on-set and as a light Department. I would hover between the DoP, the directors, to sign off the director’s vision.
Around this is a core of skills, tools and processes,
and reflections source first AD, the actors, and the Production Designer if
with moving parts all around. In this way, the Mandala
for this filming. It is here there’s something specific for set dressing. The Skills Demand
is a map of the different roles and skills required of
where many of the hybrid Those cross-departmental conversations should
each department and the ways these interact in the I’m finding my artists have changed. I’m hiring
roles exist, artists who possess a combination of be happening on the Virtual Production shoot. All
world of production. Most importantly, gone are the a lot of people from the video game industry.
technical and creative expertise (circled in green). As Departments should get an element of training
strict dependencies of linear production – meaning Because here’s the thing: I’m making an animated
work progresses, the assets mature, with feedback during the tech recce, and get the Gaffers involved
that previously ‘Pre’ and ‘Post’ departments are now film. It’s a 90-minute movie. We’ve got a deadline
from the Production team and Production Designer. in the conversation with the Virtual Production
‘on-set’, with roles, functions and tasks overlapping to hit. You’ve got to be smart. And I find people
Meanwhile the Virtual Production Department begin to supervisors so they’re aware and know how to help
and merging. that work in the video game industry, in terms of
test and set the technical game engine parameters that and supplement the Virtual Production streams. CG artists, they have one word on their mind that
will govern the lighting, colour and other visual effects
artists in film and television don’t usually have.
- including the animation of some of the 3D assets. At The Opportunity
And that word is ‘optimisation’. They have to
this, and at all stages, close collaboration between the
I find VP is a good place for people that are trying optimise it to work on a game engine and it has to
Director and the Heads of all Departments (HoD’s) will
to break into the industry. Back in the day, you had work really, really well.
drive the film project to completion.
10 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 11
The VP Skills Mandala
The Camera Department different to a traditional shoot: visual effects (VFX)
prepare the set from for example, are no longer a ‘Post’ task, having been
CASE STUDY | WILDER FILMS
around 180 degrees. completed ‘in-camera’ in real-time during the filming
The Director of on-set. Any further VFX required will add time and
Name of Interviewee: The Opportunity Photography will cost to a Virtual Production and may result in the
Annalise Davis I felt that Virtual Production took a familiar setting be coordinating same visual effect having to be carried out twice.
and gave us a new perspective on that world, on between Director This underlines the fact that careful planning of shots
Job Title
how we see those people, and elevating the story and the technical and effects during Preproduction is always needed,
Producer camera team, with much effort shifting to significantly earlier in the
in really interesting ways. So, for me, it was very
Company much a story-driven and character-driven decision; as well as the project cycle.
it didn’t ever change the story of Fireworks, it only Lighting and Grip
Wilder Films
enhanced it. When we realised we could retool teams. The DoP will
Postproduction comes
Fireworks using Virtual Production, Steve Jelley also liaise closely with
Virtual Production Role to a conclusion with
from Dimension Studio said, “Come and visit our Visual Effects and Virtual
final colour grading
With VP, as a producer, you can have more set, and you’ll see what I mean,” and I understood Production supervisors to
and other finishing,
control over the production, and it allows for more it properly for the first time when I walked on ensure that the real-time visual effects created
bringing the VP
flexibility on set. Similarly, for a designer and a the set; up until that point it was a very technical by technical artists in the Virtual Production
cycle full circle,
DoP, it’s brilliant, because they have more input process with words I didn’t fully understand. After Department will merge seamlessly with the
360 degrees. An
into what goes on screen, compared to working visiting the set, I could see how the technology filmed action on-set. The team under the Virtual
important piece of
with green-screen. could serve the story, and we realised that it could Production supervisor calibrates and maintains
work that is adjacent
work; what Virtual Production allowed us to do the camera tracking system - without which
The Changing Role and Relationship to to the VP Mandala
was open up the scope of the story and the ways there would be none of the spatial and temporal
other Departments is work carried out
we could engage with the characters in that world. synchronisation between the camera and the virtual
prior to all of the above
My number-one goal with any film is around set displayed on the LED volume, which together
The Skills Challenge of VP in the recruitment of
story. Not just telling a good story, but ensuring make up the ‘storyworld’ for cast and crew alike to
crew to a project. The VP
everybody’s telling the same story. When you work within. It is this close integration of technology
Gaffers on set talk about light using different Skills Mandala shows an
have a Virtual Production, you’re working with and story that produces the effect of visual parallax
language from VP gaffers’ lights. Also, on a initial mapping of the roles and skills for Talent
a Director, obviously, and you’re also working between virtual background and real foreground
film set, if the AD who’s running the day says Strategist and Director of Production, who will lead
closely with a Production Designer and a as the camera moves around in this ‘world’, and it
to a cinematographer, “How long?” and the recruitment. The skills in these roles include talent
cinematographer. All those people are part of that is this that makes Virtual Production filmmaking so
cinematographer says, “Five minutes,” it’s going casting for Technical Artist and other roles such as
storytelling conversation. On a VFX-heavy film convincing for viewers.
to be five minutes. But if the AD says to an VP R&D Engineers to assist in the development of new
with larger mixed technical and creative teams, person on a Virtual Production, “How long?” and forms of VP. Significantly, the Director of Production
there is a risk that the key creative input from the they say, “Five minutes,” it could be five minutes, To the West of the is never simply looking for technical skills, they
cinematographer and designer may get lost. but sometimes it can be 20 minutes, sometimes Mandala, at 225 profile new employees for their aptitudes, interests
As a producer, I worked very hard with the AD four hours. Where every minute costs money, this degrees, the Editorial and approach to working collaboratively in time-
department, because I wanted the technical is a very real challenge. Department has pressured situations.
element not to overshadow the team, the cast received the data
The Skills Demand from which the final
and their performances. On Fireworks, we made
sure the actors didn’t feel that they came second Because the film industry and film sets work product will be
to the technology, even though we waited for the in a very specific way, and there are cultural formed, after which
technology a lot more than we waited for differences between games and film, and the film enters the
the actors! If you allow the technology to take over, communication can be a challenge. final Postproduction
then you’re not respecting the actors and their phase (‘Post’). Here
A film aesthetic is very different from a games
process, and you need to do that to get the Sound Post and
aesthetic; narrative design is a different way of
good performances. Picture Post crew will
looking at the world. There will be real demand
add any additional effects
for Unreal technicians who can work on film sets,
needed to complete the
and are trained in that ‘setiquette’. As well as film
production design and round off
and VFX technicians who can learn to work in the
the look and feel so that the story can be delivered
Unreal world.
according to the original vision of the Director and
their Production team. This portion of the cycle looks
Courtesy of ARRI / CT Group
12 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 13
SECTION 3
THE OPPORTUNITY
As Screen Daily reported in March 2021, the carry out re-shoots and pickups via the game engine the industry consortium created to drive sustainability, is so big, and there are so many tiny little things that
pandemic has accelerated the growth and popularity Unreal, and use virtual sets as a source of lighting which indicated that VP might have been “130 times need to happen, I think interns can play a big part in
of Virtual Production for its ability to help conquer and colour (Haz Dullul, Director/Producer, Hazfilm). more effective” compared to going on location that” (Dullul, Director/Producer, Hazfilm) and shorten
travel and social distancing measures. But the Fireworks director Paul Franklin adds that key (Russell Shaw, VP Supervisor, Quite Brilliant). what is traditionally a long multi-decade journey to
scale of the opportunities makes VP skills ones the production design opportunities emerge through the rise to the level of Director or VFX Supervisor, for
“industry workforce must get to grips with.” [3] deliberate merging of the virtual and physical: Beyond efficiencies, second-order business instance. Recent graduates can also take advantage of
opportunities also expand around VP: for example, this freedom to operate. A graduate from 2019 found
“in fact, we needed to consider them as a whole,
Creative Opportunities: Opening Pandora’s Box creating scans or models of well-known locations or his university training in Contemporary Media Practice
even though they’re two very different worlds,
unique props to be licensed as digital assets to other to be extremely useful - for example, he was able to
VP has great “untapped potential in terms of they’re these different imaginary spaces that exist
productions. The ability to reuse digital 3D assets transfer the skills from the marked documentation
aesthetics, in terms of understanding of storytelling” within the same visual world”
and virtual scenes is also likely to have an impact modules of the course to current VP projects, which
according to creatives working in VP, potential that Paul Franklin, Director, and Creative Director, DNEG.
on driving productions towards a more efficient is an area in need of development to support evolving
goes beyond the current usages of VP “essentially
approach to budgeting and scheduling of resources workflows. This earlier involvement, at the ideation
as a green-screen proxy, or a back-projection proxy” Business Opportunities
between and across productions. Following the stages of a project, is an opportunity:
(Jamie Lapsley, Production Designer). Realising
VP is reaching quickly into all areas of production, increasing appreciation of the business potential of
this potential depends on seeing the creative “because I’m doing a lot more than just a technical
although, so far, there has not been a full feature film VP, one client-facing VFX Supervisor reported that he
opportunities of the real-time physics engine, the role at the moment, I’m being brought on a lot
made using VP. Nevertheless, as a measure of the could see that “commercials people were just literally
ability of the game engine to create “a reality” with sooner, which is also beneficial for me, getting
range of VP, “there is now not a single [key] show on thinking, ‘We just don’t need a set’,” and are becoming
its own logic. The engine can provide “instantaneous more experience in more aspects of the pipeline”
Sky that does not use immersive tech in some way” aware of the time and money that can be saved in this
feedback, and intuitive control” and can liberate James Codling, VP Stage Technician, Final Pixel.
(Richard Nockles, VR Creative Director, SkyVR). Real- way (Gary Brown, VFX Supervisor, Ghost VFX). For
the creativity of Directors and others in this way
world factors play a part here, since “the reason why studio owners, LED equipment and virtual sets can be
(Paul Franklin, Director, DNEG). This can be a two- After all, says Production Designer Jamie Lapsley,
people are doing it is... because it’s going to make things used for production of commercials, and can be re-
way exchange between technicians and creatives, “there’s a zero cost way into this - Blender and
quicker and therefore cheaper” (Mark Flanagan, EMEA used for further seasons on Series-based productions,
as noted by a Director of Photography, “I said to Unreal” (Lapsley, Production Designer), and it is
Education, EPIC). This speed makes VP a very different or on sequels and other content for film projects.
an Unreal Artist, ‘Can we do this, can we do that?’ partly this that promises to deliver a diversity of
proposition up and down the production hierarchy, with
They’re like, ‘No one’s ever said that to me before.’ backgrounds into VP. Practitioners acknowledge that,
Executives able to make immediate decisions: Entry Opportunities
So these are the good things that would help drive to-date, when recruiting for roles in immersive “it has
new stuff” (Andreas Neo, DoP). Crucial to unlocking “They’re like, ‘Wait, what? This never happens. We Many of our interviewees related the fact that VP “is tended to be white men” (Peng and Dobbs-Beck,
the potential of VP is providing the space for expect a week later to get some notes, and we sit a good place for people that are trying to break into ILMxLAB) and that with VP there is an opportune
departments to communicate and understand the around and do a whole bunch of notes on email.’ the industry” (Dullul, Director/Producer, Hazfilm). moment to change this. The current skills shortages
skills and technical parameters involved in any task or I’m like, ‘No, you can just address the notes now.’ With the number of VP stages expanding rapidly, in Unreal for VP mean that the employment net is
creative process. As one interviewee explains: So that’s a game changer for them” and the unprecedented demand for content across being cast wide by companies, however, this has not
Haz Dullul, Director / Producer, Hazfilm. many platforms, there is a moment of opportunity helped to remedy the prevalent gender imbalance:
“those on set need to know if [something is]
for entry-level runners, interns and graduate trainees
definitely impossible… or is a slightly more “It is at the level of your, unfortunately, boy in
Efficiency is therefore a key driver for the uptake to gain valuable experience on-set in VP. Rather than
involved process that will take more time?” the bedroom. I wish it was a girl in the bedroom,
of VP. This comes at a time when efficiency gains standing aside deferentially, these entrants would be
Russell Shaw, VP Supervisor, but it’s not”
that have been made in recent years may have well advised to “ask for something to do… because VP
Quite Brilliant at the Garden. Simon Frame, VFX Supervisor, Netflix.
reached their limit. VP offers new opportunities,
VP creates an opportunity for creatives to have a such as previsualisation, which can reduce re-
stronger say at the beginning of projects, putting shoot requirements that often run to 20% of total
their stamp on the look and feel of the production, budget. LED shooting techniques reduce the cost
and ensuring that early creative conversations of Postproduction, including compositing and
and decisions are perpetuated throughout the rotoscoping, and can often lead to faster production of
cycle. Producers working in VP gain an improved screen-test versions of projects, thereby reducing the
level of control over certain aspects of production overall time needed to reach a final product. Virtual
and planning. Creative opportunities are linked to sets can also save on production time and costs:
innovative VP techniques, enabling practitioners to travel for the crew is minimised, as is transportation
push the envelope with “insane hybrid virtual camera and the cost of filming on location. Quite Brilliant VP
moves” and to “blend Virtual Production cameras studio at The Garden undertook a carbon analysis on
with my real cameras,” test out lighting, plan and a recent VP shoot with Ad Green and We Are Albert,
3
Baughan, N. (2021) ‘Training: How Virtual Production will become essential beyond the pandemic’, Screen Daily, 17/03/21.
See: https://www.screendaily.com/features/training-how-virtual-production-will-become-essential-beyond-the-pandemic/5158106.article
14 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 15
© Wilder Films / Fireworks
The VP Skills Mandala
CASE STUDY | SPACE FILM & VFX
Name of Interviewee: The Opportunity
Matthew Nelson VP is going to become the norm over the next
Job Title three to four years because it reads instantly.
You’ve got lights interacting with physical objects
Producer
that are actually there and real so it’s definitely
Company 100% believable.
Space Film & VFX
It’s currently a very expensive way of shooting.
But I think over the next two or three years, the
Virtual Production Role
price will come down. There’ll be more and more
I work in the production side, and that’s all about skilled people available to do it and there’ll be
communication, it’s all about information, sharing more and more assets available.
it, and getting it. Whereas in the visual effects
world, and a lot of the characters are... not all of The Skills Challenge of VP
them, but some of them can be quite introverted At the moment, it’s still expensive to shoot in VP.
themselves. They like what they like, and that’s To build a studio with a volume is probably going
what they do. Whereas directors or actors, they’re to cost more than flying out to shoot in a location
the opposite, aren’t they? and doing it. So, at the moment, if you’re going to
do it, I think you’ve got to get a lot more added
The Changing Role and Relationship to
value and residual use out of using VP.
other Departments
VP will definitely affect roles in that they will need As a company we’ve put VP options forward for
to evolve to adapt to VP. It may not, for example, a production but perhaps the Director’s not used
affect the Wardrobe Department … but certainly VP before and Directors will probably still go out
the roles and relationship between the set designer and shoot it on a location rather than do it with a
and the Director will change. VP set.
I think Postproduction will become more of a I’ve also had that as a problem with people
priority… VFX is going to really be helping in the working in games; they’re brilliant at that and
look of a film, working closer with the DoP and it’s good for games, but for film it’s a completely
the director and the whole creative and technical different discipline.
process. They will be involved at the beginning and
The Skills Demand
have a stronger say in how the overall vision
is delivered. We need people who are passionate about visual
effects … because there’s so much to learn. 3D
skills are going to be even more important. That’s
where we’ll be focusing our hiring, over the next
two or three years. But the challenge is that
globally there’s a shortage of VFX artists.
16 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK Courtesy of ARRI / CT Group 17
SECTION 4
THE CHALLENGE
This balancing act is reflected in the integration of “By the time we find out [about a potential VP
physical and virtual sets: project], the client’s signed off on the job. They want
to get it done and shot and delivered in six weeks,
“You try and merge the worlds together. It’s tough,
and it’s on air”
it’s tough” Andreas Neo, DoP.
Matthew Nelson, Managing Director, Space Film and VFX.
The primary aim, therefore, is to create an
VP has a different timescale to traditional filmmaking,
appropriate environment for the specific constraints,
one where “fixing it in post” is effectively eliminated:
challenges and opportunities of VP. Professionals
“it forces decisions to be made much quicker, which I
consider this by asking:
think is a big plus. And it’s because of the way Virtual
“How can we fit that creative vision within these Production is. It’s all in-camera. And they know
practical constraints? And if that’s a balanced kind that fixing it in Post is very expensive” (Haz Dullul,
of relationship, you tend to end up with the best Director/Producer, Hazfilm).
work” Vic Wade, Head of Training, DNEG.
Courtesy of ARRI / CT Group
Amongst the many technical challenges of VP is
the real-time “integration” of set, actors, the virtual
Technical and Logistical:
scene and visual effects. This process challenges all
Broadly, we might think of the challenge in terms of bring to the table, and we’re looking to the interactive “faster, cheaper, and better”
aspects of production, both hardware and software,
the colour-coding of roles and skills in our Mandala: [real-time domain experts] for the interaction design. As Virtual Production workflows develop, there is as well as the skills needed to operate VP technology:
technical and creative. As per the Mandala, in roles The magic happens when they both bring the best of a constant process of adaptation to the fact that from developing new DMX lighting rigs to creating
like the Technical Artists, these challenges often their past but are willing to learn from one another” at present “the technology that they’re introducing innovative solutions to improve sound performance
overlap, whilst the need to build new and effective (Vicki Dobbs-Beck, Vice Executive in Charge, is essentially always in beta mode” (Andreas Neo, against the LED volumes. In this fast-paced landscape
workflows is a challenge that unites all of the roles, President Immersive Content Innovation, ILMxLAB). DoP). This is an R&D moment in cutting-edge of innovation developing repeatable and documented
processes and skills detailed on the Mandala. But A way forward is: technology that promises to transform the way workflows becomes crucial: “If you wanted to do it
here we outline them separately to help identify the we make film, television, games and immersive all properly, you would look to have everything in
“to actually acknowledge the difference in
challenges and then, in the next section, specific experiences. Alongside this, there are nevertheless the engine from the beginning.” and use the real-
approach that these two industries have…
skills gaps and touchpoints for these. In so doing, we strict limitations on how to plan camera moves so time engine for previsualisation and shot planning
the goal then is to figure out a way to take the best
hope to also point out where these two challenges that camera tracking systems are not compromised, before coming to the VP stage (Jon Wadelton, VP
of both” Julie Peng, Director of Production, ILMxLAB.
need to be tackled together to increase the and on-set time must be planned for recalibrating the Consultant). Similarly, Production Designers Caroline
opportunity of VP. This is not simply about the world of games system for each lens and scene change. Greville-Morris and Jamie Lapsley argue that attaining
entering films: the overriding challenge is to find the optimal form of handover from Production
Creative and Communicative: ways of minimising any potential “disconnect” when Key logistical and technical challenges include Designer to the VP Department is essential:
Making “the Magic Happen” companies expand into VP and come into closer the fact that the cost of VP is prohibitive for most
contact with the real-time computing mindset as a independent or low-budget television productions “Any assets we [the Art Department] generate
The changing place of roles within the film or low-budget films, especially when compared to need to be able to be taken in either direction
production pipeline as well as the introduction of result, says Julie Peng. Communication, as ever, is
key – with our interviewees stressing the importance green screen techniques which are relatively cheap. [digital and physical] without any more input
roles from the games sector creates new kinds of Where opportunities for VP exist in long-form high- from us, because of the way that the technology
creative and communication challenges. As Producer of soft skills as well as technical competencies to
make the most of the VP opportunity. This new end television and film, the costing and planning of VP is moving. Now we are going to [need to] make
Annalise Davies explains, “film sets work in a very shoots is an area much in need of understanding. VP is models, Unreal Engine models, of every single
specific way, and to have games people within that emphasis on the importance of communication
will help asset-producers working in new Virtual a new type of production, where there may be savings environment that we are creating for the film”
world is a big cultural issue,” which has an impact on the one hand, due to the economies of filming Jamie Lapsley, Production Designer.
on how people communicate up and down a long Production Departments to manage a different kind
of relationship with clients, one that is now more on-set compared to filming on location, while on the
established on-set “chain of command.” This is a Ultimately, this new technology, processes and
strongly mediated by Production Designers – playing other, there may be potential hidden R&D technical
two-way problem, of film adapting to games and workflows all comes down to developing a rapid
a role earlier, or higher up, the production pipeline. costs. As Paul Franklin reflects, “I think people
vice versa: for those coming from games, a VP skills pipeline. In the current market, where such skills
There is a need to counteract old habits where: just come on to it with unrealistic expectations.”
shoot is “just a different aesthetic, a different set of are scarce, companies discuss a fierce talent retention
Scheduling and budgeting for VP requires a heavy
guidelines that you wouldn’t get if you’re doing a “people doing real-time assets [are seen] as being battle and a fear of ‘brain drain’ as tech companies
up-front investment of time and money into the
game compared to doing something that’s for a VP separate to people doing the backend, a little bit of poach the brightest screen industries’ professionals.
Preproduction phase, before a single piece of data is
stage” (Jon Wadelton, VP Consultant). However, true that [divide is still] being created” Even for large film studios, like Lucasfilm, it is “hard
captured on-set, a fact that clients may not appreciate
innovation and new forms of screen entertainment Jon Wadelton, VP Consultant. to compete [with] big companies like Microsoft
fully until their first VP project has been undertaken.
only emerge from the convergence of these worlds: [who] are very invested in Virtual Production and
“we’re looking for the aesthetic that the film folks Small scale commercial projects may find this Immersive development” (Julie Peng, Director of
particularly challenging at present: Production, ILMxLAB).
18 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 19CASE STUDY | ILMXLAB The Opportunity
For apprentices, I think there are very few barriers
Name of Interviewees: to working in VP, it’s open for everyone ... In some
Vicki Dobbs-Beck of these emerging areas, the growth can be much
and Julie Peng quicker than the traditional journey up the ladder.
The people in the ILM Virtual Production teams,
Job Titles be that on the LED stage or running the real-time
Executive in Charge,, Immersive teams that are operating all of the visuals that go
Content Innovation with overall responsibility for on the displays, have come from almost every single
ILMxLAB (Dobbs-Beck); Director of Production with discipline within the VFX side. They’re modellers,
responsibility for talent strategy, ILMxLAB (Peng). texture artists, layout artists, compositors. There’s
Company not really one discipline to transition into VP from.
ILMxLAB, Lucasfilm In our roles, almost every single role requires
some level of programming. So whether it’s
in Unreal Blueprints, which is more of a visual
Virtual Production Role
based programming language, node based, or,
The Virtual Production group at ILMxLAB is seen you’re writing C++ plus code, whether you’re an
as two teams. One is the on-set team, and the experienced designer, level designer, technical
other is everyone preparing the content to be designer, gameplay engineer, Technical Artist,
used on-set. I think those two ‘mind-sets’ are even an animator or asset artist, environment and
quite different. The latter is a little bit more of a character artist, all require some level of scripting at
traditional workflow, CG asset creation essentially, some point in your day.
everything has to be really efficient rendering-
The Skills Challenge of VP
wise. The on-set mind-set, however, is about being
adept at real-time troubleshooting. It’s like working One of the challenges of VP is to help these two
on a live action shoot essentially. One of the groups eventually get closer and closer to the
training opportunities is to actually acknowledge same mind-set so that we weren’t speaking two
the difference in approach that these two have and different languages.
figure out a way to merge the best of both. We need to write job descriptions and have job
titles that appeal to people out in the world so they
The Changing Role and Relationship to understand what it is that we’re recruiting for. But
other Departments sometimes we bump up against the fact that what
We want to have adaptability for each producer that title means in the interactive space, it’s different
to be able to adapt their style while also allowing to what it means in the visual effects space. Plus,
things to work well for that project and still be able all the Virtual Production studios are developing
to have a project manager easily jump from one their own proprietary workflows, so each studio will
project to another. have its own special sauce, if you will, of how their
solution is a little bit different or better.
Our wider immersive development workflow
is highly creative and collaborative and we do The Skills Demand
always have a lead creative who is the final Unreal is definitely an area where there’s a talent
decision maker. We had a couple of Animation shortage, and Technical Artists for us is the most
Supervisors who said it was a new experience for challenging position to fill. The talent base is
them to be included and consulted on so many growing exponentially which is fantastic.
different creative levels. Everyone has the ability There have been open positions with us at
to contribute ideas and it really is a taskforce to ILMxLAB, and these are R&D roles for VP, that
figure out what is the best narrative here? How are have been open for literally months and months.
we going to tell this story in the best way? What’s These positions require specialist high-end
working? What’s not working? scientists and engineers who are interested in
I think Postproduction will become more of a melding different mindsets, and it is an interesting
priority… VFX is going to really be helping in the skills gap that will be there for a while.
look of a film, working closer with the DoP and The demand for people with real-time capability is
the director and the whole creative and technical absolutely escalating at an incredible pace and it
process. They will be involved at the beginning and crosses many industries. It’s not just competition
have a stronger say in how the overall vision within the immersive storytelling or within the
is delivered. games space, it is competition for people with
real-time capability across industries.
20 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 21
Courtesy of ARRI / CT Group.Figure 3: The VP Skills Mandala, detail: Creative and Communicative, Development phase
SECTION 5
THE SKILLS GAPS
In this section we analyse the responses of our The necessity for early and resolved conversation is
interviewees to the question: “What roles and apparent when the pressures of disciplinary work and
associated skills are in most demand for Virtual scheduling are considered: “10 scenes in two days... It’s
Production?” Within the current internal business almost choreographing the whole thing. It’s got to be
structures there are significant challenges around so regimented” (Shaw, VP Supervisor, Quite Brilliant).
recruitment of skilled people. This is becoming “My not having the vocabulary to describe things made
a barrier to growth. Those who can make the it difficult, especially to talk to the people who control
crossover from games or VFX into VP are in short the software” says Grapes Wasserman, Production
supply and all respondents who had responsibility Designer. To take one example of where communication
for recruitment reported that there are severe skills is paramount, in the Camera Department, says Andreas
shortages. These skills gaps are represented in the Neo, Director of Photography:
detailed images of areas of the VP Mandala that are
“[Usually] we have quite strict dialogue on sets
shown in this section, and which highlight the mix,
on how to communicate with one another…
and overlap, of creative and technical skills currently
there’s a learning curve attached to a completely
in short supply.
new Department being on the set … [but] what’s
Skills in VP may be used across different roles and interesting with a virtual environment is the
Departments, however, here we present the insights communication will be completely different … it’s
gained about skills most in-demand in VP, organised not [with] a Gaffer. I have to understand that it’s
across two broad domains: not as easy as turning a switch on a physical lamp
versus a virtual one.”
Creative and Communicative Skills: which
includes soft skills needed for collaboration and Acquiring the new communication skills for this is
to manage on-set technical skills in VP VFX for important for making the adjustment to a new way of
example (Figures 3, 4). working: “this was slightly different to what the DOP
was used to, [who was] saying, ‘Can you just take the
Technical Skills: including those used by VP
magenta out?’, and the LED chap didn’t have that
Artists, and VP R&D for innovation of new
control, he only had RGB control over things” (Ellie
workflows, as well as VP Colour specialisms
Clement, VP Supervisor).
(Figures 5, 6).
By presenting these in this order we follow the With good communications in place an Unreal
general sequencing of the Mandala. Within each Technical Artist can collaborate effectively with Art
domain we present an ordered list of the key skills Department Physical Standby on crafting the “matte
gaps, as described by our interviewees. Many of the line” (the join between LED screen and real set), to
most in-demand skills are in the Technical domain, say, “Well, actually, it’d be easier for me to quickly just
while other critical skills gaps are in Production put a fake bit of grass, or something, where that joint
Design, for example, within the creative sphere. is and just try and blend it a bit better rather than you
Talent Strategy is dealt with in more detail in the starting to physically move things around” (Russell
ILMxLAB Case Study (see p.20 and Figure 4, for Shaw, VP Supervisor, Quite Brilliant).
Production manager role).
Integration of Film-Games Cultures and Tools
5.1 Creative and Communicative Skills Interviewees noted problems with combining skills
and methods associated with games into a film
Communications
context. Some crew lacked an understanding of how
Communications are central to the creative process games tools can be integrated into the general flow
of VP, says Monica Hinden, Producer at Final Pixel, of the VP shoot day. In the context of a busy working
“giving heads-up when things happen, asking honest crew on a VP stage where time management is key
questions, having open lines of communications, to ‘shooting the schedule’, general problems were
being comfortable to say what you need from people, noted with incompatible working practices and lines
and to tell them when and what was a problem for of communication. Teams were also confused by
you.” The Preproduction process involves an intensive differing nomenclatures and reporting structures.
cycle of collective ‘review and rework’, which must be This often came down to lack of understanding of
planned for.
22 VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK VIRTUAL PRODUCTION | A Global Innovation Opportunity for the UK 23You can also read
