FELIX: the Detector Interface for the ATLAS Experiment at CERN
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
FELIX: the Detector Interface for
the ATLAS Experiment at CERN
Front End LInk eXchange
Alexander Paramonov1 on behalf of ATLAS TDAQ Collaboration
1) Argonne National Laboratory
19 May 2021
25th International Conference on Computing in High-Energy and Nuclear Physics
ATLAS DAQ for LHC Run3 (2022-2024)
LAr FE FE FE
100kHz Fully synchronous
GBT&FM
L1 trigger Custom
FELIX FELIX ROD ROD Custom Link
40MHz 100kHz Multiple types
GBT : synchronous serial Fixed latency
protocol at 4.8 Gb/s Highly-parallel
software software
FM: 8b/10b RX link at ROD ROD
ROS ROS
9.6 Gb/s (Full Mode)
COTS Custom
ROD : ReadOut Driver, Ethernet Electronics
ROS : ReadOut System, PC
HLT : High Level Trigger HLT HLT HLT HLT
FELIX (Front End LInk eXchange) is a modern detector readout system with fewer
custom parts.
Easier to support and upgrade than the older system
FELIX will readout the new muon detectors, LAr calorimeters, and calorimeter
trigger electronics
Readout and configuration of on-detector electronics
Distribution of trigger information and LHC clock to the on-detector (FE)
systems
25th International Conference on Computing in High-Energy and Nuclear Physics 2
FELIX hardware platform
FELIX Host PC
FELIX PCIe card
DMA
TTC interface Memory CPU MSI-X
buffer Device Driver
Optical Links FPGA PCIe Gen3 buffer
24-48 ch, firmware 126 Gb/s buffer
4.8 or 9.6 Optical Link
Gb/s PCIe Gen3 NIC Dual 25 or 100Gb/s ports
126 Gb/s
PCIe card with FPGA chip + Host PC + NIC
TTC (Time, Trigger and Control) : LHC protocol used to distribute global clock
(40.08MHz) and information from the real-time trigger
25th International Conference on Computing in High-Energy and Nuclear Physics 3
FELIX hardware components
SuperMicro X10SRA-F
Intel® Xeon™ E5-1660 V4
CPU
o FLX712 board
Custom board with Xilinx Kintex Ultrascale XCKU115
48 optical links (MiniPODs) with up to 12 Gb/s per link
Input: 24 GBT links or 12/24 links at 9.6 Gb/s (fullmode)
Output: 24 or 48 GBT links (4.8 Gb/s)
Slot for mezzanine board to allow customization to different
timing and trigger distribution mechanisms Mellanox ConnectX-5; 2x100
The ATLAS TTC signal is sampled with ADN2814 receiver on GbE (25 GbE in some cases)
the mezzanine board
SI5345 and LMK03200 clock jitter cleaners
PCIe Gen3 x16 (via a 2x8 switch)
250+ boards were produced via an external manufacturer
contracted through CERN
25th International Conference on Computing in High-Energy and Nuclear Physics 4
ATLAS sub-detectors currently connected to FELIX
– Liquid Argon Calorimeter
– LTDB (LAr Trigger Digitizer Board) --
propagation of TTC information to front-end via
FELIX cards hosting 48 GBT channels. Control,
configuration and monitoring data arriving from
the front-end.
• LDPB (LAr Digital Processing Blade) -- FELIX in
FULL mode
– Level-1 calorimeter trigger (see the TDR)
• gFEX (Global Feature Extractor): 12 FULL mode
links
• ROD, Hub for eFEX (Electron Feature
Extractor) and jFEX (Jet Feature Extractor)
• TREX (Tile Rear Extension)
– Muon spectrometer
• New Small Wheels (NSW)
• BIS78 (Barrel Inner Small RPC (sector 7/8))
25th International Conference on Computing in High-Energy and Nuclear Physics 5
Highlights from integrations for Run-3
• NSW cosmic tests were performed as a final step of the integration process
– FELIX was used to readout a section of the detector
– As an example, the efficiency map is shown below
• First LAr Trigger Digitiser and Digital Processing Blades integrated into
ATLAS data taking with FELIX during 2020. Commissioning of remaining
modules ongoing
6
Planning and development for Run 4
• System specifications
– Higher trigger rate: 100 kHz 1 MHz
– support new detector systems and data transmission protocols
• lpGBT and 64b/67b (Interlaken) for the optical links
• 64b/66b (Aurora), 6b/8b, and a variety of custom serial link protocols for the
slower electrical links.
– The new TTC system will receive data at 9.6 Gb/s instead of 80 Mb/s
• Prototype hardware platform
– Uses Xilinx Versal Prime FPGA
– Supports PCIe Gen4 x16
– Optional support for optical links with up to 25 Gb/s
– Recent tests of the PCIe gen4 interface are promising
• System integration
– Tile Calorimeter
– Pixel and Strip sensors test setups (for the upcoming ITk Inner
Tracker)
25th International Conference on Computing in High-Energy and Nuclear Physics 7
Selected results for Run-4 integration
RD53A in FNAL
testbeam
FIBER TO FELIX
Tile Cal in testbeam
8
Selected results for Run-4 integration
• The system performance was evaluated for Run-3 operation at
100 kHz trigger rate (left-hand plot).
• The system is stress tested to evaluate its performance for trigger
rates up to 1 MHz (right-hand plot)
9
Summary and prospects
• FELIX is a router between custom serial links and a
commodity network
• Takes advantage of the latest technology to simplify the ATLAS
readout
• In LHC Run-3 (2022-2024) FELIX will be used for
selected detectors and trigger systems.
– FELIX firmware and the software are mature
– All of the boards needed for data taking have been produced
• Ongoing efforts:
– Integration with the ATLAS on-detector (front-end) systems
– Development of FELIX board and firmware for LHC Run-4.
25th International Conference on Computing in High-Energy and Nuclear Physics 10Thank you!
11What is a collider experiment?
Contemporary collider experiments study collisions of particles such as
protons, electrons, and nuclei to look for new physics.
The particles are accelerated in bunches so collisions are spaced in time.
For example, the Large Hadron Collider accelerates protons to about 7 TeV
and collides them every 25 ns. The ATLAS experiment at the LHC examines
products from these collisions.
25 ns
25th International Conference on Computing in High-Energy and Nuclear Physics 1Design of the present Data Acquisition System
• At LHC energies only 10 out of 40,000 proton
collisions can be recorder for further study. All collisions
• The trigger system performs real time
selection efficiently to remove most of
uninteresting data.
• The DAQ system buffers data while the
trigger system is analyzing it. It also Interesting collisions
transports data from on-detector to offline,
while facilitating online data preparation.
ATLAS DAQ CPU-based
Disk
detector Event filter
Trigger system
25th International Conference on Computing in High-Energy and Nuclear Physics 1Challenges of the detector readout
1. Huge number of channels. For example,
the ATLAS experiment has
calorimeter with 120k channels
muon system with 820k fast channels
(RPCs and TGCs)
inner tracker has 80M pixels and 46k
strips
2. Fast sampling rate without dead time
40 Msps for the majority of the ATLAS
experiment
3. High level of synchronization for
digitization across all the systems (better
than 1 ns) for years.
4. High reliability. Beam time is very
expensive.
5. Affordable. Limit how much custom
electronics is designed.
25th International Conference on Computing in High-Energy and Nuclear Physics 14ATLAS DAQ for LHC Run2 (2015-2018)
FE FE FE FE
100kHz Fully synchronous
L1 trigger Custom
40MHz ROD ROD ROD ROD Custom Link Multiple types
Fixed latency
Highly-parallel
ROS ROS ROS ROS
COTS
Ethernet Custom
ROD : ReadOut Driver, Electronics
ROS : ReadOut System,
HLT : High Level Trigger PC
HLT HLT HLT HLT
Custom hardware and link protocols are used for the frontend readout
During Runs 1 and 2, the experience gained suggested that commodity systems are easier to support than the
custom alternatives
Most of data are buffered in the on-detector electronics (FE) while the L1 trigger system is analyzing the
data
Trigger signals and LHC clock are sent to both front-end and ReadOut Driver (ROD)
This architecture was dictated by the capabilities of available electronics in ~2005.
Based on this experience, a new system for Run 3 and 4 has been designed to make better use of
commodity electronics and benefit from technological advancement in the intervening years
25th International Conference on Computing in High-Energy and Nuclear Physics 15FELIX data flow overview Large-scale flexibility of commodity
FE ASIC
up to 320 Mb/s to synchronous serial networking.
E-link TTC
FE ASIC E-link GBTx Calibration
FE ASIC E-link
FPGA network
link up to 100Gb/s DCS
~4.8 Gb/s
TTC FE configuration
FE ASIC E-link
FE ASIC E-link GBTx
E-link Event readout
FE ASIC E-link
FE ASIC
COTS
network Event readout
switch
FE ASIC E-link TTC
FE ASIC E-link GBTx
FE ASIC E-link
FPGA
Event readout
FE ASIC E-link
FE ASIC E-link GBTx
E-link
FE ASIC E-link
FE ASIC
Commodity network. Multiple
asynchronous data streams.
Custom data links. Fully synchronous system.
25th International Conference on Computing in High-Energy and Nuclear Physics 1FLX712 components
25th International Conference on Computing in High-Energy and Nuclear Physics 17FELIX firmware architecture
Output:
24 GBT (4.8G) links
OR
12 or 24 Full mode (9.6G) links
Input:
24 GBT (4.8G) links
OR
48 GBT links for TTC-fan-out
25th International Conference on Computing in High-Energy and Nuclear Physics 18GBT Protocol
• The protocol provides aggregation and de-aggregation of
multiple slower data streams into a single fast serial link
• A slow serial link (aka E-link) is driven by a selected set of bits in the
GBT frame
GBTx IC. Rad-hard
• The GBT frame is fully synchronous with the LHC clock (200 bits in 25 ns)
• GBTX ASIC offers high fidelity clock (40 , 80, 160, or 320 MHz clock)
• Trigger and timing control for the on-detector electronics
• Flexible configuration of the I/O ports Slow serial e.g. 160 Mbps
Fast serial 4.8 Gbps
40 MHz clock
FELIX GBTx IC or FPGA
Fast serial 4.8 Gbps
Slow serial e.g. 320 Mbps
Off-detector
On-detector
25th International Conference on Computing in High-Energy and Nuclear Physics 1FELIX software
FelixStar application is a data processing pipeline between the PCIe DMA buffer
and the NIC
Custom-designed network transport software layer (netio-next) is used for
communication between FELIX hosts and network peers. It abstracts the low
level network implementation. .POSIX and RDMA backends are supported
25th International Conference on Computing in High-Energy and Nuclear Physics 20You can also read
