DB Broadcast Delivers Landmark Project for BBC Wales

dB Broadcast Delivers Landmark Project for BBC Wales
New IP Infrastructure Required Extensive System Testing, and was Delivered
Despite Coronavirus

Overview and Background

dB Broadcast has completed a landmark project for BBC Wales – a state of the art broadcast
centre at Central Square in Cardiff. Not only is it the first BBC facility to use IP technology across
both its production and broadcast operations, but it is also one of the largest SMTPE 2110 IP
routing systems for broadcast deployed anywhere.

The vision for the project from the start was to design and implement new technology to support
production and creativity based on a Live IP approach. As a result, it delivers new agile ways of
working through the open plan, collaborative style and flexible production spaces. The system is
central to the broadcaster’s new TV playout and radio operations, supporting BBC1 Wales, BBC2
Wales, Welsh language channel S4C, Radio Wales and Radio Cymru.

 BBC Wales is responsible for content production in Welsh and English across all platforms, and has the
 largest BBC newsroom outside London. In addition to BBC1, BBC2 and S4C TV channels and BBC Radio
  Wales, BBC Radio Cymru & Network radio production, the BBC Wales hub produces flagship content
     for BBC Studios and runs online services for BBC News, BBC Sport, BBC iPlayer and Cymru Fyw.

The broadcast centre is housed in a new 150,000 sq ft building - around half the size of the
previous HQ in Llandaff, which suffered from dated technology and a poor building configuration.
It has space for around 1,200 staff, and the technology deployed can adapt quickly to new video
formats, higher bandwidth and better video resolution. It also offers simpler production for new
platforms, with less time spent converting between different video or audio formats. The new
BBC Wales HQ extends over five floors, with a rooftop garden and public spaces pre-wired as
shooting locations. The new facilities provide the best possible tools with a greater level of
creative freedom for content makers, while audience engagement is enhanced.

dB Broadcast was appointed, through a competitive process, as system integrator in recognition
of its significant prior experience in broadcast IP systems. There was collaboration with the BBC
from a very early stage providing knowledge and expertise on project planning, system design
and architecture. A Proof of Concept (PoC) analysis phase followed at dB Broadcast’s premises in
Ely for interoperability testing for 2110 and AMWA/NMOS networks and equipment compliance.

Broadcast Infrastructure

dB Broadcast worked closely with contract holder Grass Valley, who installed their IP
orchestration and routing system, capable of carrying around 6Tbps of uncompressed live media
traffic throughout the building. IP enabled Sirius routers were installed alongside a fully
redundant Morpheus and ICE Panoplay system, with Morpheus deployed as a completely
virtualised system. The 2110 IP network is based on a spine/leaf architecture utilising Cisco
networking hardware with a Cisco DCNM layer for device authentication and management of
flow permissions. The IP Core was designed with the highest resiliency in mind utilising A & B
networks, enabling dual path resiliency across the network supporting the SMPTE 2022-7
protocol. The Software Defined Network (SDN) was provided by Grass Valley with GV Convergent
which enabled interoperability with commercial off the shelf switches (COTS) and support of
SMPTE/EBU/AMWA standards.

Audio Network integration with the IP Core was paramount with dedicated Audio Over IP (AoIP)
islands for Radio (AES67) and Post-Production (Dante). Tight integration between BNCS, the
principal control platform, and Convergent allows users and operators to utilise the IP core
routing with familiar, easy to use interfaces.

TV Studios

                                                                    dB Broadcast integrated two
                                                                    News Studios and a General
                                                                    Purpose Studio, all Live IP
                                                                    facilities, as well as a further
                                                                    versatile space equipped
                                                                    with a lighting rig so that it
                                                                    can be another shooting
                                                                    space. There are three studio
                                                                    galleries that can be
                                                                    assigned to any of the studio
                                                                    spaces.

                                                                    The largest 3,500 sq ft studio
                                                                    includes a fully motorised
                                                                    lighting grid and is equipped
with Sony IP cameras and a Vizrt AR graphics system. The second 1,600 sq ft news studio is fully
automated and equipped for AR and a third studio is a full green-screen VR space. The News
studios benefit from Robotic cameras and pedestals, augmented and virtual reality (AR/VR)
graphics systems, whilst the larger News Studio boasts a 70m2 video wall, integrated and driven
by gallery automation.

The building has 20 ‘inject points’ where cameras can be connected, and the IP network can
route the signals from them directly to any of the galleries.

Media production

Elsewhere in the building,
there are state of the art Post-
Production facilities, including
27 Video Editing Suites, three
Dubbing Theatres, two
Tracklaying suites, three Voice
Over Booths and a Colour
Grading Suite. All take
advantage of the Live IP Core
and facility wide IHSE KVM
system to access a shared pool
of resources. The Audio Editing
Suites were designed with
Dolby Atmos capability, enabling the BBC to work with the latest and most impressive audio
standard. Avid Interplay was chosen as the Post Production asset management tool, with fast
access Nexus storage. While facility asset management was provided by Dalet, which meant dB
Broadcast and the BBC engaging in workflow analysis and consultancy to optimise the file-based
workflows between sub-systems such as EVS, Avid, VizRT and NRCS systems. dB Broadcast

worked closely with EVS and VizRT so their equipment natively supported 2110 directly onto the
IP Core network.

TV Playout

dB Broadcast delivered automated Playout and Continuity systems for BBC1W, BBC2W and S4C to
viewers in Wales across linear TV (DTT, DSAT) and VOD Platforms, with a mixture of Network and
Local programming, both live and pre-recorded. A flexible approach to de-risk this critical
element of the project included a phased deployment, with dB Broadcast delivering live and POC
systems in the original Llandaff facility.

Radio

                                                                         dB Broadcast delivered
                                                                         five flexible IP radio
                                                                         studios clustered around
                                                                         three control rooms for
                                                                         main services, with one
                                                                         larger studio to
                                                                         accommodate live music
                                                                         guests. A further seven
                                                                         self-op radio studios used
                                                                         for a variety of
                                                                         programming including
                                                                         news were also
                                                                         delivered. A total of nine
                                                                         production spaces had
                                                                         visualisation and lighting
installed, enabling the radio studios to deliver multimedia based services particularly to online
platforms. Further to this, ten Audio Edit Suites based on SADie editing were installed, as well as
a dedicated Drama Studio, Control Room and Edit Suite.

Benefits of Live IP

A key feature of the project has been the focus on a Live IP infrastructure throughout, requiring
liaison with multiple equipment vendors and extensive system testing before handover.

The benefits of this approach are substantial. It is a highly flexible solution, supporting new ways
of working with more dynamic assignment of resources, allowing facilities to be scaled up more
easily.

While IP standards are increasingly being adopted, this was not the case at the start of the
project, and dB Broadcast had to maintain as much flexibility as possible. The result is a system
with a good deal of future proofing, with the ability to adopt new formats as they emerge.

Efficiency has been a key focus. The system supports exceptional utilisation of facilities due to the
adoption of virtualisation, COTS hardware and the ability to move broadcast functions onto a
more generic platform with a distributed routing core.

And the challenges…

Achieving these benefits did not come easily of course: moving site and introducing new
technology at the same time was always going to be a challenge!

“The most significant issue was achieving interoperability along with reliable device control,
because at the time IP standards were very new and evolving rapidly”, says Mike Bryan,
Programme Manager at dB Broadcast. “We have found from the many large scale IP projects that
dB Broadcast has delivered recently, that having access to our own test lab here at dB is hugely
beneficial, not only for our own engineers’ access, but we are able to provide remote access to
the BBC team and to third party manufacturers to a standalone version of the core system. This
allowed extensive testing and promoted collaborative supplier working to prove multi-vendor
interoperability in a controlled way.” Accordingly, a greater level of resource was required to
concentrate efforts on this aspect. dB Broadcast used a formalised approach using IT techniques
(TestRail) to manage regression testing, and to track progress between test runs.

Fault finding can become a huge issue on large ST 2110 systems, and the testing approach itself
can unexpectedly break the system under test. Security is another major challenge with IP
systems as a network can be disrupted e.g. by denial of service attacks. So a lot of time was
spent designing in safeguards such as segregating the network from general office traffic.

With Live IP, equipment refresh cycles are much faster, and don’t necessarily align well with
traditional capital planning models. New skills were required to deliver and support Live IP, both
at dB Broadcast and at the BBC. This implied an increased attention to training.

Delivery 2020

On top of the inherent challenges of delivering the BBC’s first large IP facility with Post-
Production and Radio integration came Covid-19! This of course meant a radical change to ways
of working, with colleagues required to work either remotely - connected to the dB Broadcast
server and administration systems by a secure VPN - or under more controlled conditions with
adequate PPE etc. Everything possible was done to protect employees and the client while
continuing the project, allowing dB Broadcast and its partners to successfully deliver the system
which went on air under lockdown conditions.

Roger Crothers, Head of Technology at BBC Cymru Wales, has commented: “The worlds of
broadcasting and media are changing at an unprecedented rate; our audience wants more
content than ever before across an increasingly diverse range of devices. We need a facility that
can remain up-to-date for many years to come, allowing us to adapt easily to new formats and
services. IP is the only logical choice: it is format agnostic and meets our need for scalability and
agility.”

A final word from Mike Bryan: “With any programme of this scale, especially taking into account
this being based on an IP infrastructure, it had some significantly challenging moments, but
ultimately it was extremely rewarding to see the finished installation come together, and BBC
Wales migrate live services into Central Square. It has been a huge team effort, bringing together
BBC Wales, wider BBC teams, suppliers and ourselves; everyone had a part to play and did so.
The system gives BBC Wales an excellent technology platform to build on, and to produce high
quality content for its audiences for many years to come.”