PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC

Page created by Tom Sandoval
 
CONTINUE READING
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
Number 1 : 2014
                                                A magazine published by the Notur II project

PHYSICS OF EARLY
UNIVERSE WITH
ATLAS AT CERN'S LHC

Microbial Communities
in permafrost-affected
peatlands;               Supercomputing-              Engineering
important players        enabled study                Nanoparticles
in Earth’s carbon        of subcellular               for Oil Field
cycle                    calcium dynamics             Applications

          The Norwegian metacenter for computational science
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
E D I TO R I A L

                                                                                                                                                                                                           Documenting the utility value of the national e-infrastructure is important
                                                                                                                                                                                                           to secure funding for future development. The articles in this edition of
                                                                                                                                                                                                                                                                                         F E ATURE S
                                                                                                                                                                                                           META show that there is a wide diversity in the scientific disciplines
                                                                                                      CON TEN TS                                                                                           utilizing the e-infrastructure. This underlines the importance of e-infra-
                                                                                                                                                                                                           structure as tool for science.
                                                                                                                                                                                                           The article on subcellular calcium dynamics on Tianhe-2 is particularly
                                                                                                     Physics of Early Universe with ATLAS at CERN's LHC                                     p:   4         interesting in terms of the exploitation of a HPC compute hierarchy – from
                                                                                                                                                                                                           the national level to a top international system. In a future computational
                                                                                                     Microbial Communities in permafrost-affected peatlands;                                               infrastructure, a careful and cooperative provision of integrated services
                                                                                                                                                                                            p : 10
                                                                                                     important players in Earth’s carbon cycle                                                             at the national and international level will be of growing importance, both

                                                                                                     Supercomputing-enabled study of subcellular calcium dynamics                           p : 14
                                                                                                                                                                                                           in terms of hardware and software.                                            Are there more  Higgs particles
                                                                                                                                                                                                                                                                                         than the one already observed?
                                                                                                                                                                                                           We also present a short overview of the Notur and NorStore statistics
                                                                                                                                                                                                           which indicates that the demand for resources continues to increase.
                                                                                                     Engineering Nanoparticles for Oil Field Applications                                   p : 18         Also worth mentioning is that statistics from CRIStin (Current Research       Supersymmertry
                                                                                                                                                                                                           Information System in Norway) shows that a considerable number of             predicts five Higgs bosons.
                                                                                                     NorStore launches pilot research data archive                                          p : 22
                                                                                                                                                                                                                                                                                                                                  :4
                                                                                                                                                                                                           publications are referring to the use of Notur and NorStore as a neces-
                                                                                                                                                                                                           sary tool for their work.
                                                                                                     From PRACE-3IP: Exploiting HPC Tools and Techniques                                    p : 24         On this background it is very important and good news that a decision has
                                                                                                                                                                                                           been made to introduce a new model for investment in and operation of
                                                                                                                                                                                                           national high-performance computing and storage solutions for scientific
                                                                                                     Usage Statistics of the national e-infrastructure                                      p : 28
                                                                                                                                                                                                           data. As the current contract between UNINETT Sigma and The Research
                                                                                                                                                                                                           Council is finishing at the end of this year, a new company will be
                                                                                                                                                                                                           established. Still being a subsidiary of UNINETT this company will,
                                                                                                                                                                                                                                                                                         The   sequence comparison is
                                                                                                                                                                                                           compared to UNINETT Sigma, have a broader strategic responsibility and        computationally very demanding, but using a
                                                                                                                                                                                                           stronger operative control in addition to more long-term funding. The
                                                                                                                                                                                                           overall aim is to make a foundation for better services to the scientific     high performance cluster
                                                                                                                                                                                                           disciplines. An interim board has been formed and work has been started
                                                                                                                                                                                                           to establish the new company which will be going live on 1. January 2015.     such as Stallo, millions of sequences

                                                                                                                                                                                                           Based on a broad community-driven request we announced in March
                                                                                                                                                                                                                                                                                         can be compared within days.

    Cover picture: © CERN
                                                                                                                                                                                                           a new service for archiving and sharing large-scale research data. The
                                                                                                                                                                                                           Archive provides functions to deposit, describe and manage datasets from
                                                                                                                                                                                                           the user community. You can read more about this new service both in
                                                                                                                                                                                                                                                                                                                            :10
                                                                                                                                                                                                           this edition of META and by visiting our web sites.
                                                                      Number 1 : 2014                 Number 1 : 2014
                                                                                                      A magazine published by the Notur II project – The Norwegian metacenter for computational science
                                                      A magazine published by the Notur II project

                                                                                                      Editorial Director:    Arild Halsetrønning
                                                                                                                                                                                                                                                                                         Instead of passively waiting for the arrival
      PHYSICS OF EARLY                                                                                Contributing Editor:   Vigdis Guldseth                                                                                                                                             of exascale supercomputers,
      UNIVERSE WITH
      ATLAS AT CERN'S LHC
                                                                                                      Subscription:          An electronic version is available on www.notur.no. Here you can download a
                                                                                                                             PDF-file of the magazine and subscribe or unsubscribe to the magazine.                                                 Arild Halsetrønning                  we believe that progresses can be made in the
                                                                                                      ISSN:                  1890-1956                                                                                                              Managing Director
                                                                                                                                                                                                                                                    UNINETT Sigma AS                     current  numerical scheme to
                                                                                                                                                                                                                                                                                         facilitate faster simulations.
      Microbial Communities
      in permafrost-affected
      peatlands;               Supercomputing-              Engineering
                                                                                                      Reproduction in whole or in part without written permission is strictly prohibited.
                                                                                                      e-mail address: sigma@uninett.no. Phone: +47 73 55 79 00. Fax: +47 73 55 79 01.
      important players        enabled study                Nanoparticles
      in Earth’s carbon        of subcellular               for Oil Field

                                                                                                                                                                                                                                                                                                                            :14
      cycle                    calcium dynamics             Applications

                The Norwegian metacenter for computational science
                                                                                                      Print: Skipnes kommunikasjon AS

    Disclaimer: UNINETT Sigma is not responsible for errors, opinions or views expressed by the authors or the contents of external information referred to in this magazine.

2                                                                                                                                                                                                                                                                                                                             3
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
PHYSICS OF EARLY
                                                                                    UNIVERSE WITH
                                                                                                                                                                                                                                                                                                 Figure 1. Standard Model particles. Fermions (quarks
Copyright 2013 CERN. Photo: Dominiguez, Daniel.

                                                                                                                                                                                                                                                                                                 and leptons) as well as the W and Z weak bosons ac-
                                                                                                                                                                                                                                                                                                 quire their masses by interacting with the Higgs field.
                                                                                                                                                                                                                                                                                                 The photon and gluon do not interact directly with the

                                                                                    THE ATLAS AT THE
                                                                                                                                                                                                                                                                                                 Higgs field, and thus stay massless. The Higgs particle
                                                                                                                                                                                                                                                                                                 observed by ATLAS and CMS at LHC is the manifesta-
                                                                                                                                                                                                                                                                                                 tion of this Higgs field.

                                                                                    CERN’S LARGE                                                                                         experiment at the South Pole. This discovery
                                                                                                                                                                                         is a further brilliant confirmation of the theory
                                                                                                                                                                                         of general relativity of Einstein and may be
                                                                                                                                                                                                                                             decayed to lighter particles as the Universe
                                                                                                                                                                                                                                             cooled down. Today they are produced in high-
                                                                                                                                                                                                                                             energy particle collisions and some are found
                                                                                                                                                                                                                                                                                                 theory, which happens by introducing particle
                                                                                                                                                                                                                                                                                                 masses per hand, a scalar field is predicted to
                                                                                                                                                                                                                                                                                                 populate the vacuum. The whole Universe

                                                                                    HADRON COLLIDER                                                                                      interpreted as a first hint towards the existence
                                                                                                                                                                                         of the graviton, the hypothetical mediator of
                                                                                                                                                                                         gravity at the elementary particles scale. At the
                                                                                                                                                                                                                                             in cosmic rays.

                                                                                                                                                                                                                                             A POWERFUL THEORY MISCALLED
                                                                                                                                                                                                                                                                                                 swims in an invisible, cosmic field, Higgs-field,
                                                                                                                                                                                                                                                                                                 which acts on particles and provide them with
                                                                                                                                                                                                                                                                                                 what is called mass. As all fields have associ-
                                                                                                                                                                                         level of quarks and electron, gravity is some       “STANDARD MODEL”                                    ated bosons (the photon is associated to the
                                                                 AUTHOR             High Energy Particle Physics has recently extended the frontiers of physics                          forty orders of magnitude weaker than electro-      The Standard Model (SM) is the theoretical          electromagnetic filed), the Higgs-field has its
                                                               Farid Ould-Saada                                                                                                          magnetism, itself a factor of ten weaker than       framework that elegantly describes nearly all       Higgs-boson. It is that particle that the ATLAS
                                                              Physics Professor     knowledge using the advanced technology of the world's largest and
                                                                                                                                                                                         the strong force. Electrons and quarks feel a       existing particle physics data with astonishingly   and CMS experiments at the LHC observed in
                                                              University of Oslo,
                                                                                    highest energy particle accelerator, the Large Hadron Collider (LHC) at                              fourth fundamental force of Nature: the weak        good precision. The resulting quantum field         2012.
                                                      Leader, Norwegian CERN-
                                                                                                                                                                                         interaction, mediated by three force particles      theory relies on quantum mechanics and
                                                           related HEPP project     the CERN laboratory in Geneva, and the world's largest and most
                                                                                                                                                                                         W+, W- and Z0, responsible for phenomena such       special relativity unifying space and time into     Englert and Higgs, two of the people behind the
                                                                                    sophisticated particle detectors, one of them being the ATLAS experiment,                            as nuclear radioactivity and burning of stars.      a four-dimensional space-time. To each              so-called BEH-mechanism of spontaneous
                                                                                    by discovering the last missing building block of the Standard Model (SM)                            To this force is associated the neutrino, an        particle is associated a field describing its       breaking the electroweak symmetry, thus
                                                                                                                                                                                         essential ingredient of nuclear ß-decay, where      probability of presence at some space-time          generating masses of the W, Z and fermions,
                                                                                    of elementary particles, the Higgs boson. The coming LHC running at                                  a neutron (d-quark) decays into a proton            point. Forces in nature originate from some         were awarded the 2014 Nobel price.
                                                                                    higher energy and higher collision rates is crucial to settle the properties                         (u-quark), an electron and an anti-neutrino,        symmetry required to hold at each space-time        How important is this? It is well known that
                                                                                                                                                                                         and of other phenomena involving transfor-          point. These symmetries are related to definite     more than 99% of the mass the atom, i.e. of
                                                                                    of the Higgs particle and to continue the exciting “travel” towards “new                             mation of matter from one state to another. The     conservation laws, such as the conservation of      protons and neutrons, comes from the kinetic
                                                                                    physics” that may be expected in a previously unexplored energy regime.                              neutrino and electron belong to the family of       electric charge.                                    energy of quarks and gluons. However, without
                                                                                                                                                                                         leptons. Matter particles, quarks and leptons,                                                          the relatively tiny mass of the electron (~2000
                                                                                                                                                                                         as well as the anti-matter counter parts,           The Standard Model describes interactions           times lighter than the proton) the atom would
                                                                                    THE PARTICLE CONTENT OF THE                       the photon, syntesizes nuclei and electrons into   positrons and anti-quarks, are named                between elementary particles grouped in             not form as the electron would be traveling
                                                                                    UNIVERSE                                          atoms, atoms into molecules, and molecules         fermions, whereas force particles are called        3 families of quarks and leptons (Figure 1).        undisturbed at the speed of light, and life would
                                                                                    A few elementary particles play a role in the     into matter stars and we are made of. Gravity      bosons. Fermions interact by exchanging             At high energies, it unifies electromagnetism,      not be. Idem, if the weak boson were massless,
                                                                                    world we leave in. Two quarks dubbed up (u)       acts on matter at larger scales of macroscopic     bosons.                                             a long-range, macroscopic force, carried by         the weak force would be as strong as electro-
                                                                                    and down (d) combine to make the proton (uud)     objects, planets, stars, galaxies and black                                                            the massless photon, and the weak force,            magnetism and radioactivity would be rather
                                                                                    and the neutron (udd) under the influence         holes.                                             The weak force is responsible for the fact that     a short-range, microscopic force carried            the rule then the exception. Finally the
                                                                                    of the strong force or interaction, which is                                                         the content of the Universe evolves with energy,    by heavy W and Z bosons. The resulting              existence of scalar particles could be the
                                                                                    mediated by a force carrier named gluon. The      The observation of gravitational waves that set    temperature and distance scale. We know that        electroweak symmetry must be broken at low          source of inflation, the incredible acceleration
                                                                                    same force is responsible for the nuclear force   on the space-time infl ation of the Universe       two additional families of heavier fermions         energies in order to give the weak bosons           of the expansion of the Universe just after the
                                                                                    holding protons and neutrons together to form     some 13.8 billion years ago was recently           existed just after the big bang, although they      (W, Z), as well as all matter particles, masses.    big bang, in connection with the gravitational
                                                                                    nuclei. The electromagnetic force, carried by     announced by the radio telescope Bicep2            do not form the matter surrounding us. They         In order not to spoil the predictability of the     waves mentioned above.

                                                  4                                                                                                                                                                                                                                                                                                    5
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
TO SEE WITH THE LHC
The Large Hadron Collider (LHC) at the CERN
laboratory in Geneva is the first particle
accelerator to directly explore the Tera-
electron-Volt (TeV)1 scale, a new energy
frontier. By colliding beams of protons or lead
nuclei, the LHC probes deeper into matter than
ever before, reproducing conditions in the first
picoseconds in the life of the Universe.

                                                                                                                                                                                                                                                                                                                                      Copyright 2013 CERN. Photo: Dominiguez, Daniel.
Several experiments at accelerators and
colliders were built with the goal to, among
others, discover the Higgs particle. LHC and
the accompanying multipurpose experiments
were planned such as not to miss the Higgs, if
it exists.

The higher the beam energy and collision rates
are, the greater the discovery potential. How
come? To produce particles of large mass re-
quires a lot of energy, according to the known
Einstein relation E = Mc2, stating the equiva-      Figure 2. ATLAS event display showing a candidate Higgs boson of mass 125 GeV decaying into two Z bosons each
                                                    of which subsequently decays into two muons (red lines). The inset on the right-hand side shows a zoom into the
lence of mass M and energy E, related to each
                                                    tracking detector. The inset on top left shows a zoom into the vertex region, indicating that the 4 muons originate
other by the speed of light c. A mass of a proton   from the same primary vertex. This picture of a particle collision, demonstrates that the ATLAS detector performs                                                                                                                       Figure 3. Invariant mass of 4 leptons
corresponds to ca. 1 GeV. Two protons at rest       very well. The Norwegian scientists made a strong contribution to the semi-conductor tracker, one of the central                                                                                                                        featuring the Higgs signal (in red)
                                                    detectors with the aim of identifying charged particles, such as electrons and muons. The tracker has been                                                                                                                              consistent with the prediction of the
have energy equivalent to ca. 2 GeV. The total      instrumental in dealing with the harsh conditions of extremely high collision rates with up to 40 proton-proton col-                                                                                                                    Standard Model. The rest of the spec-
LHC energy of 8 TeV, equivalent to 8000 proton      lisions reconstructed per event. Other detector types – calorimeters – measure the energy of electrons, photons                                                                                                                         trum (blue) is due to well-understood
masses, is due to the kinetic energy of the         and hadrons (protons, neutrons, pions). All other particles have a too short lifetime to be observed in detectors.                                                                                                                      electroweak processes.
                                                    Those are inferred from their decay products with the help of conservation laws, such as energy and momentum.
accelerated protons. It is this kinetic energy      This is the case of the Higgs and Z bosons, leading to muons in the picture above.                                                                                                                                                      Figure 4. IInvariant mass of 2 photons
that allows the creation of heavy particles such                                                                                                                                                                                                                                            featuring the Higgs signal, A Gaussian
as the known top quark and Higgs boson. The                                                                                                                                                                                                                                                 peak, on top of well-understood back-
                                                                                                                                                                                                                                                                                            ground processes due to the strong
higher the particle mass, the harder it is to                                                                                                                                                                                                                                               interaction. The signal, shown in the
produce it. This is where luminosity enters.                                                                                                                                                                                                                                                lower plot after background subtrac-
                                                                                                                                                                                                                                                                                            tion, is consistent with the prediction
Luminosity gives a measure of how many              SHAKING THE VACUUM                                          Approximately 500 Higgs events were observed
                                                                                                                                                                                                                                                                                            of the Standard Model.
collisions are happening in a particle collider:    Billions of events were recorded in 2011 and                through fi ve decay channels. Figure 3 and
the higher the luminosity, the more particles       2012 that lead to the “re-discovery” and study              Figure 4 show the Higgs signal through its
are likely to collide. When looking for rare        of all known elementary particles, such as                  decay into two Z particles leading to 4 leptons
processes, this is important. The Higgs             the mediators of weak interactions (W and Z                 and two photons, respectively, both featuring              The discovery of the Higgs particle has been a    the world on the worldwide LHC Computing         Other computing facilities in universities and
particle would be produced very rarely, so          particles) and the top quark, the heaviest of               a particle at a mass of ~125 GeV. Decays to pairs          “global effort leading to a global success”, to   Grid (WLCG). The mission of the WLCG (Figure     laboratories take part in LHC data analysis as
for a conclusive discovery or disproof of its       all. LHC is a top quark and weak boson                      of W bosons and tau leptons were also                      repeat the words used by the CERN DG, Rolf        5) is to build and maintain data storage and     Tier-3 facilities, allowing scientists to access
existence, a large amount of data was               factory. Millions of events have been re-                   observed. It has been proven that the observed             Heuer. “Results today only possible due to        analysis infrastructure for the entire High      Tier-1 and Tier-2 facilities. With the rapid
required. Peak luminosity close to the LHC          constructed and studied through their decays                particle is a boson of spin 0, a scalar compat-            extraordinary performance of accelerators,        Energy Physics (HEP) community around            increase of data produced by LHC, Norway
design luminosity was reached in 2012. The          into leptons: 108 W, 107 Z, and 5x105 top events.           ible with the missing ingredient of the Standard           experiments and Grid computing”. So, it would     the LHC. Raw data emerging from the              unfortunately missed the opportunity to host
total number of proton collisions amounted          The ATLAS and CMS collaborations, in a                      Model, coupling to both bosons and fermions.               have been impossible to release so many           experiments are recorded and initially           the extension of the Tier-0 outside CERN. The
to 1.80 X 1015.                                     heroic “dugnad” of thousands of physicists,                 The decays to pairs of bottom quarks and other             results and discover the Higgs particle so        processed, reconstructed and stored at the       Wigner centre in Budapest is related to CERN
An LHC collision leads to hundreds of particles     students and engineers, made the first major                rare decays still need confirmation. A future              quickly without the outstanding performance       Tier-0 centre at CERN, before a copy is          by a 100 Gbit/s connection.
measured by detectors, such as ATLAS, which         discovery of a new boson at the LHC in 2012,                goal is to study the coupling of the Higgs                 of the Grid.                                      distributed across 11 Tier-1s, which are large
can be thought of as a sort of digital cameras      announced on July 4.                                        particle to its self, and confirm that the Higgs                                                             computer centres with sufficient storage
with millions of readout channels instead of                                                                    boson acquires its mass through the BEH                    A WORLDWIDE DISTRIBUTED                           capacity and with 24/7 operation. The role of
pixels. Figure 2 shows an example of a Higgs                                                                    mechanism.                                                 COMPUTING SYSTEM                                  Tier-2s is mainly to provide computational
candidate event observed in ATLAS.                                                                                                                                         The real data collected by the LHC experi-        capacity and appropriate storage services for
                                                                                                                                                                           ments have been distributed smoothly around       event simulation and for end-user analysis.

                                                       1
                                                           Tera-electron-Volt. 1 TeV = 1000 GeV is equivalent to the energy of ca. 1000 proton masses.
6                                                                                                                                                                                                                                                                                                                                7
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
WLCG BEYOND MOORE’S LAW
                                                                                                                             The WLCG has handled hundreds of peta-
                                                                                                                             bytes of data at rates far beyond the design
                                                                                                                             goals. The size of the data is close to the
                                                                                                                             content uploaded to the social media. With                                                                                                  Figure 7. At any given time,
                                                                                                                                                                                                                                                                         there are more than 160,000
                                                                                                                             the advent of higher LHC luminosities,                                                                                                      concurrent ATLAS jobs running
                                                                                                                             larger data sets and more complicated                                                                                                       on the Grid.
                                                                                                                             events to be recorded and analysed, an
                                                                                                                             upgrade of the ATLAS computing system,
                                                                                                                             and of WLCG in general, is foreseen. Until
                                                                                                                             2017 the LHC numbers in terms of CPU and
                                                                                                                             disk are probably sustainable with close to
                                                                                                                             flat computing budgets. The progress             predicts five Higgs bosons. Super-           col le c te d b y th e dete c to r s . Tw o
                                                                                                                             achieved due to Moore’s law increase in          symmetry proposes a good candidate for       hundred institutes from 41 countries
                                                                                                                             computing power is being halted by               Dark Matter observed in the Universe.        par ticipated in the 2014 program.
                                                                                                                             limitations on power and cooling. Multi and      Why is gravity much weaker than other        Exciting measurements are proposed to            Further reading
                                                                                                                             many-core processors and the use of              Nature’s fundamental forces? Are there       the students to extract properties of
                                                                                                                             co - p r o ce s s o r s such a s G r ap hic al   more space dimensions than the usual         known particles, such as the W and Z             1. CERN, Accelerating Science,
                                                                                                                             Processor Units (GPUs) are possible ways         known 3, which would allow gravity-          bosons, mediators of the weak force, and             home.web.cern.ch
                                                                                                                             forward. Any efficient use of the new hard-      related phenomena, such as microscopic       various hadrons, particles made of
                                                                                                                             ware architectures must be followed by the       black holes and gravitons, be produced,      quarks, as well as searching for new force       2. ATLAS Experiment,
                                                                                                                             challenging modification or rewriting of the     to occur at the LHC energy era?              particles linked to new interactions.                atlas.ch
                                                                                                                             current application code. The adaptation of      If accessible, LHC can produce such new      T he educ ational mater ial , p ar tly
                                                                                                                             the code has started in order to make use        par ticles and phenomena could be            developed by the University of Oslo, is          3. A Particle Consistent with the Higgs Boson Observed
                                                                                                                             of opportunistic resources: volunteer            produced at the LHC and observed and         tuned and expanded to follow LHC                     with the ATLAS Detector at the Large Hadron
                                                                                                                             computing (Boinc) and supercomputers             studied by ATLAS.                            “heartbeats”.                                        Collider, Science 338 (6114), 2012, 1576-1582.
                                                                                                                             (High Performance Computing) for special,        Despite enormous gains in mass reach                                                              [DOI:10.1126/science.1232005]
                                                                                                                             CPU intensive, short jobs. NorduGrid/ARC         since the previous experiments, there is     LHC SOON BACK TO EXPLORE A NEW
                                                                                                                             is critical part of both these activities.       as yet no direct evidence for new physics.   TERRITORY                                        4. "The Nobel Prize in Physics 1979",
                                                                                                                                                                              However, we have collected only a few %      LHC´s physics research programme has                 nobelprize.org/nobel_prizes/physics/laureates/1979/
                                                                                                                             THE ROAD TO “NEW PHYSICS”                        of the data planned for the full LHC         definitively started. The first running
                                                                                                                             Despite the success of the Standard Model        program and already in 2015 the doubling     period ended on a high tone. ATLAS, to           5. "The Nobel Prize in Physics 2013",
Figure 5. The success of the Worldwide LHC Computing Grid relies on a high-volume performance of the world                   it is not the ultimate theory of Nature.         of the collision energy could yield some     name only one experiment, made three                 nobelprize.org/nobel_prizes/physics/laureates/2013/
network, allowing high and efficient data transfers between data centres: Tier-0 (data processing), 11 Tier 1s (simu-
                                                                                                                             It does not account for dark matter or dark      surprises.                                   discoveries: “jet quenching”, probably a
lation, reprocessing), 140 Tier 2s (simulation, user analysis) and many Tier 3s (user analysis). In total there are, as of
April 2014, 270 PB disk and 201 PB tape storage space, and 500,000 processors available on WLCG. The NDGF Tier 1             energy, and does not incorporate gravity.                                                     sign of the primordial quark-gluon               6. “LHC research started … WLCG busy”,
(Norwegian UNINETT Sigma Tier 2) has 6.4 PB (320 TB) storage and 42,000 (1,480) processors available.                        Various extensions have been proposed            SHARING ATLAS DATA AND DISCOVERIES           plasma, in 2010 (META 2/2011), a new                 notur.no/sites/notur.no/files/publications/pdf/
                                                                                                                             incorporating new symmetries, additional         WITH HIGH-SCHOOL STUDENTS                    B-meson state in 2011 and a scalar particle          meta_2011_2.pdf
                                                                                                                             space dimensions, substructure of the            The ambition to bring to the “classrooms”    compatible with the long-sought Standard
                                                                                                                             fermions and bosons, to name only those.         any important LHC discovery is fulfilled     Model Higgs boson in 2012. The LHC               7. The Worldwide LHC Computing Grid,
NORDIC E-INFRASTRUCTURE                                        ATLAS distributed computing infrastructure.                   ATLAS looked for a large spectrum of new         with the recent discovery of the Higgs bo-   machine and its detectors are in the                 wlcg.web.cern.ch
The Nordic countries participate in the                        The operation of the Nordic Tier-1 and                        phenomena, leading to new heavy particles        son. Approximately 10% the 8-TeV ATLAS       middle of a two-year shutdown period.
Nordic Tier-1 component of the WLCG                            Norwegian Tier-2 is done in collaboration                     or new interactions from the unification of      discovery data are made available for        Starting from spring 2015 the LHC will           8. The Nordic e-infrastructure Collaboration,
collaboration, through the Nordic Data Grid                    with USIT (University of Oslo IT department)                  the electroweak and strong forces, or even       students to search for the Higgs boson,      deliver 13-14 TeV energies and ATLAS is              neic.nordforsk.org
Facility (NDGF), now part of the Nordic                        and UNINETT Sigma, within NDGF. ATLAS                         including gravity. Down to 10 -20 meter the      the particle responsible for spontaneous     being consolidated and tuned to cope with
e-infrastructure Collaboration (NeIC). NGDF                    data - more than 150 petabytes (Figure 6) -                   fundamental constituents of matter seem          breaking of the electroweak symmetry,        the much higher luminosities and data            9. NorduGrid Advanced Ressuorce Connector,
is the only distributed Tier-1, with 2 of its                  are distributed, processed and analysed at                    without structure. Following the discovery       through which all elementary particles of    rates expected. The WLCG has as a                    nordugrid.org
7 centres located at universities of Oslo                      more than 130 sites across the world,                         of the Higgs boson, further data will allow      the Universe acquire mass. In the future     challenge to incorporate new ideas and
and Bergen. A key contribution is made in                      including Oslo. At any given time, there are                  in-depth investigation of the boson's            the hope is to bring new discoveries to      sustain a computing infrastructure for
developing and deploying Grid middleware                       more than 160,000 concurrent jobs running                     properties and thereby of the origin of          high school and university students.         another period of 20 years, allowing
– NorduGrid Advanced Resource Connector                        (Figure 7) and more than a million jobs are                   mass. Do more Higgs particles exist than         The international masterclasses in the       physicists all over the world to explore a
(ARC) – and software and setting up a the                      submitted on a daily basis.                                   the one already observed? Supersymmetry          LHC era feature the use of real fresh ex-    new territory full of promises. Stay tuned!
                                                                                                                                                                              perimental data as they are produced and

8                                                                                                                                                                                                                                                                                                                                 9
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
MICROBIAL COMMUNITIES IN PERMAFROST-AFFECTED PEATLANDS;

                                IMPORTANT PLAYERS IN
                                EARTH’S CARBON CYCLE

                                                                                                                                                                                                                                                                                                                             Photo: Mette M. Svenning
                                                                                                                                                                                                 Photo: Christiane Graef
                                                                                                                                              Figure 1. Peter Frenzel is sampling methane gas and soil from the Arctic peat, Ny-Ålesund, Svalbard.

                                                                                                                                              ARCTIC SOILS, MAJOR SOURCES                                                  are still poorly understood. Advanced            T, G, and C in DNA and A, U, G, and C in RNA)
                                                                                                                                              FOR GREENHOUSE GASES                                                         molecular methods combined with high             in millions of gene and transcript frag-
                                                                                                                                              Recent estimates point out that 277 Pg                                       throughput computer analyses are                 ments. Sequences from cultivated model
                                                                                                                                              (1 petagram (Pg) = 1000 Terragrams (Tg) =                                    necessary tools to address the underlying        microbes are available from public data-
                                                                                                                                              1 billion metric tons) of SOC are stored in                                  complex microbial processes in these high        bases. With the computational resources
                                                                                                                                              Arctic peatlands (1), which is approximately                                 organic soils. We have applied metage-           available at Stallo (Notur), we have
                                                                                                                                              1/3 of the current greenhouse gas content                                    nomics and metatranscriptomics to identify       compared the metagenomes and meta-
                                                                                                                                              in the atmosphere. These peatlands have                                      the microbial genetic diversity and activity     transriptomes with the sequence entries in
                                                                                                                                              acted as a carbon sink since the Holocene                                    in high-Arctic peatlands from Svalbard. One      the public databases. If the similarity of two
                                                                                                                                              (2, 3). In contrast, they are substantial                                    of four field sites in Ny-Ålesund, Svalbard,     sequences is below a threshold, we can
                                                                                                                                              sources of methane (CH4), releasing ap-                                      is shown in Figure 1. The taxonomic and          assume that the two sequences stem from
                                                                                                                                              proximately 35Tg per year, which corre-                                      metabolic diversity of microorganisms in         phylogenetically related organisms, and
                                                                                                                                              sponds to 6% of the global CH4 emissions                                     soil ecosystems is immense. A challenge          share the same or a very similar function.
                                                                                                                                              (4). CH4 is a much more potent greenhouse                                    in the study of microbial ecology is there-      The sequence comparison is computation-
                                                                                                                                              gas than carbon dioxide (CO 2 ), being                                       fore to identify the function of the different   ally very demanding, but using a high per-
                                                                                                                                              26 times more efficient (on a mol to mol                                     microorganisms in the soil, and how              formance cluster such as Stallo, millions
                                                                                                                                              basis) in absorbing the infrared radiation                                   changes in the microbial community               of sequences can be compared within days.
                                                                                                                                              from Earth (5). Arctic and especially high-                                  structure and activities affect the rates of
                                A substantial part of the Earths' soil organic carbon (SOC) is stored in arctic peatlands, ecosystems         Arctic regions are already exposed to, and                                   biogeochemical transformations.                  FROM GENE TO ECOSYSTEM
                                that are sensitive to global warming. These peatlands represent large sources for emissions of the            predicted to experience an even stronger                                     Metagenomics, the study of all DNA               Plant litter is the major component of SOC
                                                                                                                                              temperature increase until the end of the                                    fragments in an environmental sample is a        in Arctic peatlands. The major players in
                                greenhouse gases methane (CH4) and carbon dioxide (CO2), end products of microbial soil organic               century (1-5°C for summer and 2.5-11°C                                       powerful genomic technique, yielding             the SOC degradation are microorganisms
                                carbon degradation. Global warming might promote microbial processes, increasing CH 4 and CO2                 for winter, surface temperatures) (6), which                                 know-ledge about the genetic potential of        of all three domains of life including
                                                                                                                                              is expected to lead to expanded frost free                                   the entire microbial community directly in       Bacteria, Archaea and Eukarya. They
                                emissions from the Arctic. To predict future CH4 and CO2 emissions, an in depth understanding of              vegetation periods and increased active lay-                                 their natural environment. Using this            participate in a cascade of degradation
                                                                                                                                              er depths in permafrost affected soils (1).                                  method one can also identify functional          steps, eventually resulting in the emission
                                microorganisms and microbial metabolic networks in Arctic soils is required.
                                                                                                                                              Microorganisms play a key role for degra-                                    aspects of the microorganisms. However,          of CH4 and CO2. The initial step of degrada-
                                                                                                                                              dation of the stored carbon and are respon-                                  DNA based studies do not provide informa-        tion is the hydrolysis of large polymeric
                                                                                                                                              sible for climate gas production. To under-                                  tion about the activities of microorganisms.     plant compounds such as cellulose and
                                                                                                                                              stand their role and how their activity is                                   For this purpose, metatranscriptomics, the       hemicellulose, catalysed by extracellular
                                                                                                                                              affected by climate changes is therefore                                     study of expressed genes (transcripts) of a      enzymes from microorganisms. Further
                                                                                                                                              important as a basis both for modelling of                                   community, can complement metage-                steps in the degradation are carried out
                                                                       AUTHOR                                              AUTHOR             permafrost ecosystems and predictions                                        nomics and provide information on the            through anaerobic respiration and fermen-
                                                             Mette Marianne Svenning                                           Tim Urich      about future CH4 and CO2 emissions.                                          presumably active community members (7).         tative and methanogenic metabolic path-
                                               Department of Arctic and Marine Biology    Department of Ecogenomics and System Biology                                                                                     We have applied these direct high-through-       ways leading to production of CH4. Before
                                                   UiT The Arctic University of Norway                      University of Vienna, Austria     METAGENOMICS AND                                                             put methods on the active soil layer of          reaching the atmosphere, CH4 can be con-
Photo: Mette M. Svenning

                                                                                                                                              METATRANSCRIPTOMICS                                                          Arctic permafrost ecosystems.                    sumed by CH4 oxidizing bacteria that re-
                                                                                                                                              The microorganisms responsible for the                                       Using high-throughput 454 pyrosequencing         quire oxygen. They are the biological filters
                                                                       AUTHOR                                              AUTHOR             breakdown and mineralization of soil                                         or Illumina sequencing, we identify the base     for CH4 in the environment. Their activity
                                                                Alexander Tøsdal Tveit                                        Peter Frenzel   organic carbon (SOC) in permafrost soils                                     composition (combination of the bases A,         will depend on the water level in the soil.
                                               Department of Arctic and Marine Biology    Max Planck Institute for Terrestrial Microbiology
                                                   UiT The Arctic University of Norway                                  Marburg, Germany
                           10                                                                                                                                                                                                                                                                                          11
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
one single species of Bacteria, Methylobacter tundri-                              Acknowledgements
                                                                                                                                                                          paludum. Therefore, this special bacterium represents                              NOTUR High Performance Computing, is acknowledged for very
                                                                                                                                                                          a key organism as CH4 filter in the Arctic soil (11). The                          professional service and access to the resources on Stallo. The
                                                                                                                                                                          emission of CH4 from this ecosystem might thus be                                  sequencing service was provided by the Norwegian Sequencing
                                                                                                                                                                          particularly vulnerable to climatic changes, depend-                               Centre (www.sequencing.uio.no). Our research in Arctic microbial
                                                                                                                                                                          ing on the response of these key microorganisms.                                   ecology has been funded through The Research Council of Norway
                                                                                                                                                                                                                                                             Grant 191696/V49.

                                                                                                                                                                          COMPUTATIONAL RESOURCES: STALLO                                                    1. Tarnocai C, Canadell JG, Schuur EAG, et al. 2009. Soil organic
                                                                                                                                                                          Our research depends on the high performance                                       carbon pools in the northern circumpolar permafrost region. Glob.
                                                                                                                                                                          cluster facilities at UiT, The Arctic University of                                Biogeochem. Cycle 23.
                                                                                                                                                                          Norway (Stallo). Our main task is the screening of                                 2. Post WM, Emanuel WR, Zinke PJ, et al. 1982. Soil Carbon Pools
                                                                                                                                                                          large datasets containing genetic information using                                and World Life Zones. Nature 298:156-159.
                                                                                                                                                                          the BLAST (12) and HMMER (13) algorithms. In future
                                                                                                                                                                                                                                                             3. Harden JW, Mark RK, Sundquist ET, et al. 1992. Dynamics of
                                                                                                                                                                          work, it is expected that this will be broadened to
                                                                                                                                                                                                                                                             Soil Carbon During Deglaciation of the Laurentide Ice Sheet.
                                                                                                                                                                          include the modelling of climate change effects on
                                                                                                                                                                                                                                                             Science 258:1921-1924.
                                                                                                                                                                          Arctic microbial networks.
                                                                                                                                                                                                                                                             4. Cao M, Marshall S, Gregson K. 1996. Global carbon exchange
Figure 2. The microbial community in the Arctic                                                                                                                                                                                                              and methane emissions from natural wetlands: Application of a
peatlands of Svalbard based on the relative abundance                                                                                                                                                                                                        process-based model. J. Geophys. Res., 101:399-414.
of small subunit rRNA molecules (8). The size of the
squares corresponds to the biomass of the different                                                                                                                                                                                                          5. Lelieveld J, Crutzen PJ, Bruhl C. 1993. Climate effects of
groups. Bacteria are divided into the major sub-
                                                                                                                                                                                                                                                             atmospheric methane. Chemosphere 26:739-768.
groups called phyla. Metazoa, Protists and Fungi, all
eukaryotic microorganisms with larger cell struc-                                                                                                                                                                                                            6. van Oldenborgh GJ, Collins M, Arblaster J, et al. 2013. IPCC:
tures, are also divided into their respective phyla.
                                                                                                                                                                                                                                                             Annex I: Atlas of Global and Regional Climate Projections.

                                                                                                                                                                                                                                      Photo: Peter Frenzel
                                                        Figure 3. A model showing the major processes carried out by microorganisms in Arctic peatlands (8). The stored                                                                                      7. Carvalhais LC, Dennis PG, Tyson GW, et al. 2012. Application
If the soil is water saturated and thus oxygen          SOC are in the form of plant compounds such as cellulose, hemicellulose, pectin, lignin and polyphenols.                                                                                             of metatranscriptomics to soil environments. J Microbiol Methods
                                                        These compounds are degraded to the finale products CH4 and CO2 emitted to the atmosphere. Amount of CH4                                                                                             91:246-251.
is absent, CH4 will be released directly to the         emitted is dependent on the activity of Methylobacter tundripaludum. Each step of the degradation is performed
atmosphere.                                             by specific groups of microorganisms, shown in yellow. These names correspond to those shown in figure 2. The                                                                                        8. Tveit A, Schwacke R, Svenning MM, et al. 2013. Organic carbon
                                                        degradation is mainly anaerobic (without oxygen).
Our combined study of the genes (meta-                                                                                                                                                                                                                       transformations in high-Arctic peat soils: key functions and
genomics) and the transcripts (meta-                                                                                                                                      CONCLUSIONS AND OUTLOOK                                                            microorganisms. Isme J 7:299-311.
transcriptomics) in the Arctic soil has given           the microbial community, but may still have              such as increased temperatures will affect the           Our studies have revealed the complex structure of                                 9. Bradford MA. 2013. Thermal adaptation of decomposer
us a detailed insight into this “invisible”,            very important roles in this ecosystem.                  composition of microorganisms (9).                       microbial communities in Arctic peatland ecosystems,                               communities in warming soils. Front Microbiol. 2013 Nov 12;4:333.
but very important microbial world. Differ-                                                                      It can also be expected that such changes may            comprising individual taxa or populations having a                                 eCollection 2013.
ences in the genetic code were used to                  The genes and transcripts encode enzymes                 affect the metabolism of microbial communi-              distinct role in the degradation of organic matter.
                                                                                                                                                                                                                                                             10. Hall EK, Neuhauser C, Cotner JB. 2008. Toward a mechanistic
distinguish the different groups of micro-              that catalyse different reactions in the micro-          ties (10). Thus, identifying the presence and            These microbes are sensitive to changes, and there-
                                                                                                                                                                                                                                                             understanding of how natural bacterial communities respond to
organisms. From the number of genes or                  organisms. From the similarity to genes and              activity of the key microorganisms related to            fore will affect the emissions of CH4 and CO2 from the                             changes in temperature in aquatic ecosystems. Isme J 2:471-481.
transcripts corresponding to the different              transcripts in the databases, we identified the          climate change is of outmost importance. Our             soil to the atmosphere. Further studies will
microorganisms we have constructed a                    microorganisms responsible for the                       results show that some microbial groups have             address I; how this community act when the temper-                                 11. Svenning MM, Hestnes AG, Wartiainen I, et al. 2011. Genome
schematic overview of the microbial com-                different biochemical reactions needed to pro-           very specific roles, while other groups have             ature increases and II; if changes in activity and                                 Sequence of the Arctic Methanotroph Methylobacter tundripaludum
                                                                                                                                                                                                                                                             SV96. J. Bacteriol. 193:6418-6419.
munity of the Svalbard peatlands (Figure                d u ce a n d ox i d i z e C H 4 . We f u r t h e r       many different functions in the Arctic                   interactions influence the amount of climate gas
2). In these ecosystems, the microbial com-             constructed a complete model of SOC degra-               peatlands. Actinobacteria has a very broad               emissions. These cold Arctic environments may also                                 12. Altschul SF, Madden TL, Schaffer AA, et al. 1997. Gapped
munity is dominated by small single celled              dation in Arctic peat, including the complex             metabolic range, while Firmicutes and Proteo-            select for microbial adaptation to low temperatures                                BLAST and PSI-BLAST: a new generation of protein database
microorganisms within the domain Bacte-                 network of anaerobic respiratory and                     bacteria are restricted to specific metabolisms.         and thereby causing population shifts related to                                   search programs. Nucleic Acids Res. 25:3389-3402.
ria. Most of the larger eukaryotic microbes,            fermentative pathways, CH4 production and                The last steps of decomposition leading to CH4           increased temperature. How this will affect the                                    13. Eddy SR. 2011. Accelerated Profile HMM Searches. PLoS
many of which feed on the smaller Bacteria              CH4 oxidation. This metabolic network together           and CO2 production are carried out by only               microbial communities and their metabolic proper-                                  Comput Biol 7:e1002195.
and Archaea, are found within two major                 with the active soil microorganisms are illus-           three groups of Archaea,Methanomicrobiales,              ties need to be further investigated.
groups called Protists and Metazoa. Fungi               trated in Figure 3.                                      Methano-bacteriales and Methanosarcinales.
and Archaea make up smaller fractions of                It is recognized that environmental changes              The oxidation of CH4 to CO2 is carried out by

12                                                                                                                                                                                                                                                                                                                               13
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
Figure 1: A schematic overview of a sarcomere (left) and a calcium release unit (right)
Photo: Shutterstock/www.phys.org

                                                                                                                                                                                           making nanometer-resolution simulations of                  co-processor, which adopts the new many-         total. This means that employing a sizable

                                        SUPERCOMPUTING-ENABLED STUDY                                                                                                                       sub-cellular calcium dynamics
                                                                                                                                                                                           extremely time consuming.
                                                                                                                                                                                                                                                       integrated-core (MIC) architecture. Each
                                                                                                                                                                                                                                                       Xeon Phi contains between 57 and 61 cores,
                                                                                                                                                                                                                                                                                                        portion of Tianhe-2 requires performance
                                                                                                                                                                                                                                                                                                        scalability on thousands of nodes. Although

                                        OF SUBCELLULAR CALCIUM DYNAMICS
                                                                                                                                                                                                                                                       where each core can spawn four hardware          this is a classical problem with using large-
                                                                                                                                                                                           The arrival of petascale supercomputers                     threads and has a private 32KB L1 data           scale clusters, an extra difficulty associated
                                                                                                                                                                                           has ignited new thinking about the feas-                    cache, together with a 512KB L2 cache that       with Tianhe-2 is due to its three-coproces-
                                                                                                                                                                                           ibility of detailed subcellular simulations.                is kept fully coherent with L2 caches on the     sor configuration per node, meaning that
                                                                                                                                                                                           Although these parallel systems are still                   other cores. The vector processing unit on       hiding the overhead of data movement
                                                                                                                                                                                           not powerful enough for 3D simulations                      Xeon Phi uses a 512-bit SIMD instruction         needs to consider more levels.
                                                                                                                                                                                           with 1-nm resolution, experiences thus                      set, theoretically capable of simultaneously
                                                                       Calcium dynamics is of paramount importance for the heart. Every                                                    obtained will guide the preparation of                      producing 8 double-precision values per          HANDLING THE CHALLENGES
                                                   AUTHORS                                                                                                                                 extreme simulations that target future                      clock cycle. In comparison with Intel's multi-   We have adopted a straightforward numeri-
                                                            Xing Cai   heartbeat is triggered by an increase in the intracellular calcium                                                  exascale systems.                                           core architecture, it is considerably more       cal scheme, based on explicit time integra-
                                        Simula Research Laboratory.
                                                                                                                                                                                                                                                       difficult to achieve a decent fraction of        tion and 3D finite differences. The target
                                          Department of Informatics,   concentration, for which calcium channels open in the cell membrane and
                                                                                                                                                                                           TIANHE-2                                                    Xeon Phi's more than 1 Tflop/s peak              mathematical model consists of five
                                                  University of Oslo
                                                                       allow calcium ions to flow inward. This calcium inflow triggers a further                                           Thanks to an ongoing collaboration                          performance.                                     coupled 3D reaction-diffusion equations
                                                                                                                                                                                           between Simula Research Laboratory and                                                                       and two ordinary differential equations in
                                                                       release from the internal calcium storage, which is called the sarcoplasmic                                         National University of Defense Technology                   The second challenge arises from the             each grid node. Moreover, at every time
                                                                       reticulum. Each heart cell has about 20,000 calcium release units (CRUs).                                           in China, we were given the opportunity to                  unique hardware configuration of                 step, each ryanodine receptor, which is the
                                                                                                                                                                                           test simulations of subcellular calcium                     Tianhe-2's compute nodes, each having            calcium release channel, is subject to a sto-
                                                                       Contrary to being stable and synchronous in healthy heart cells, calcium                                            dynamics on Tianhe-2: the world's current                   namely three Xeon Phi coprocessors that          chastic Markov-chain computation deciding
                                                       Johan Hake      release during heart failure is less reliable, slower and dyssynchronous.                                           No.1 supercomputer. Challenges on three                     are connected to a CPU host through dis-         whether calcium release happens or not.
                                        Simula Research Laboratory                                                                                                                         levels, however, had to be overcome in                      tinct PCIe buses. In comparison with the
                                                                       The resulting random Ca releases can trigger whole-cell calcium waves,                                              order to effectively use Tianhe-2 to run the                common design of one Xeon Phi per node,          The key to achieving high performance on
                                                                       which are arrhythmogenic and potentially lethal under unfortunate                                                   simulations with an unprecedented                           Tianhe-2's node configuration requires the       Tianhe-2 lies in an efficient execution of
                                                                                                                                                                                           resolution.                                                 additional MIC-MIC collaboration, while          all the involved triply nested for-loops. First
                                                                       circumstances. Intricate details that happen at the subcellular level are                                                                                                       putting more pressure on the CPU host.           of all, the entire 3D uniform mesh of
                                                  Glenn Terje Lines
                                                                       thus vital for understanding the changes in calcium handling during                                                 First of all, the basic hardware building                   The host has to offload three pieces of          computational voxels is evenly partitioned
                                        Simula Research Laboratory.
                                                                                                                                                                                           block of Tianhe-2 is Intel's Xeon Phi                       computation to the three coprocessors, in        into a set of subdomains, matching the
                                          Department of Informatics,
                                                  University of Oslo
                                                                       different pathologies                                                                                                                                                           addition to orchestrating MIC-MIC and            number of Tianhe-2 nodes used. The
                                                                                                                                                                                                                                                       MIC-host data movements and carrying out         computations per subdomain are further
                                                                       One of the main obstacles to accurately        s a rc o m e re , w h i c h o c c u p i e s a 3 D                                                                                inter-node MPI communication. The actual         divided between the CPU host and the three
                                                                       simulating calcium releases, for both          volume of 10x10x2 µm3, it would require in                                                                                       programming thus becomes rather                  coprocessors, such that each coprocessor
                                                                       healthy and pathological situations, is the    total 2x1011 computational voxels, each of one                                                                                   complex.                                         is responsible for a 3D subpartition of the
                                                                                                                                                                          www-top500.org

                                                                       enormous computational requirement.            cubic nanometer in volume. For example, a                                                                                                                                         subdomain. During each time step, the CPU
                                                                       Ideally, 1-nm resolution is needed to simu-    straightforward numerical algorithm is esti-                                                                                     The last challenge for effective usage of        host invokes the three coprocessors to each
                                                                       late the details of the CRUs. If such an ex-   mated to require 1019 floating point operations                                                                                  Tianhe-2 is associated with its large            compute within its respective subpartition.
                                                                       treme resolution is used throughout a single   for simulating one sarcomere over 1ms, thus                                                                                      number of nodes, amounting to 16,000 in          In addition to computing the remaining 3D
                                                                                                                                                                                           Tianhe-2

                                   14                                                                                                                                                                                                                                                                                                               15
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
1.0                                                             1.0                                                            1.0
                              c at t=1 ms                                                     c at t=8 ms                                                    c at t=24 ms
             3.0                                               0.9           3.0                                               0.9          3.0                                               0.9

                                                               0.8                                                             0.8                                                            0.8
             2.5                                                             2.5                                                            2.5
                                                               0.7                                                             0.7                                                            0.7   For the single-node performance on                performance obtained on a single Xeon Phi          or GPUs of a supercomputer. Challenges will
             2.0                                                             2.0                                                            2.0
                                                                                                                                                                                                    Tianhe-2, we have focused on carefully            coprocessor came, though, at the cost of           lie in mesh generation, load balancing and the
                                                               0.6                                                             0.6                                                            0.6
                                                                                                                                                                                                    orchestrating the different tasks that need       meticulous code optimizations. Luckily, these      numerical treatment of interfaces that are
      [µm]

                                                                      [µm]

                                                                                                                                     [µm]
             1.5                                                             1.5                                                            1.5
                                                               0.5                                                             0.5                                                            0.5   to be handled by the CPU host, with help of       optimizations are not restricted to simulations    between two mesh patches of different
             1.0
                                                               0.4
                                                                             1.0
                                                                                                                               0.4
                                                                                                                                            1.0
                                                                                                                                                                                              0.4
                                                                                                                                                                                                    OpenMP threads. In particular, care was           of subcellular calcium dynamics, but generi-       resolutions.
                                                                                                                                                                                                    taken to make data movements happen at            cally reusable for many other similar com-
             0.5                                                             0.5                                                            0.5
                                                               0.3                                                             0.3                                                            0.3
                                                                                                                                                                                                    the same time while computations were             putations. Scalable performance was                With respect to programming, progress can
             0.0                                               0.2           0.0                                               0.2          0.0                                               0.2   being carried out on the coprocessors             achieved on more than 12,000 Xeon Phi co-          be made to generalize the various code
               0.0    0.5    1.0   1.5  2.0     2.5      3.0                   0.0    0.5    1.0   1.5  2.0     2.5      3.0                  0.0    0.5    1.0   1.5  2.0     2.5      3.0
                                   [µm]                                                            [µm]                                                           [µm]                              and the CPU host. The overhead associated         processors, by carefully applying pipelining       optimizations as reusable software libraries,
                                                               0.1                                                             0.1                                                            0.1
             Limits c: [0.14, 14.1] µM; mean 0.21 µM                         Limits c: [0.14, 14.7] µM; mean 0.62 µM                        Limits c: [0.14, 8.6] µM; mean 0.75 µM                  with various data movements was thus              strategies to hide data movements on the           or even automated source code generators.
                                                                                                                                                                                                    effectively hidden, resulting in a very good      various hardware levels.                           Such programming innovations will not only
                                                               1.4                                                             1.4                                                            1.4   performance scaling from using one co-                                                               simplify the coding effort of similar compu-
                             csr at t=1 ms                                                   csr at t=8 ms                                                  csr at t=24 ms                          processor to three coprocessors per node.         WHAT NEXT?                                         tations, but also ease the transition to new
                                                               1.3                                                             1.3                                                            1.3
             3.0                                                             3.0                                                            3.0                                                                                                       Our experiences mean that petascale super-         hardware architectures in future.
             2.5
                                                               1.2
                                                                             2.5
                                                                                                                               1.2
                                                                                                                                            2.5
                                                                                                                                                                                              1.2   For using a large number of Tianhe-2              computers can easily simulate a single             Progress should also be made to improve
                                                               1.1                                                             1.1                                                            1.1
                                                                                                                                                                                                    nodes, similar care was also taken to hide        sarcomere over 100ms with 3-nanometer              the current mathematical model, based on
             2.0                                                             2.0                                                            2.0                                                     the overhead associated with data move-           spatial resolution. Allocating the entire          observations from the high-resolution
                                                               1.0                                                             1.0                                                            1.0
                                                                                                                                                                                                    ments, both intra-node and inter-node. By         Tianhe-2 system for several days, if ever          simulations. Effort should be made to
      [µm]

                                                                      [µm]

                                                                                                                                     [µm]
             1.5                                                             1.5                                                            1.5
                                                               0.9                                                             0.9                                                            0.9   pipelining the various tasks on both the CPU      possible, should enable a simulation of the        couple calcium dynamics on different
             1.0                                                             1.0                                                            1.0                                                     hosts and the coprocessors, we managed            single sarcomere with the ultimate 1-nm            levels: subcellular, cellular and tissue. The
                                                               0.8                                                             0.8                                                            0.8
                                                                                                                                                                                                    to obtain satisfactory scalability on a large     spatial resolution. However, simulations of        latter progress is important for discovering
             0.5                                               0.7           0.5                                               0.7          0.5                                               0.7   number of nodes. In one particular perfor-        one cell, which has 50 sarcomeres connected        the medical implications of abnormal
             0.0                                               0.6           0.0                                               0.6          0.0                                               0.6
                                                                                                                                                                                                    mance test, we achieved an aggregate per-         together, are still beyond the capability of the   calcium dynamics.
               0.0    0.5    1.0   1.5
                                   [µm]
                                        2.0     2.5      3.0                   0.0    0.5    1.0   1.5
                                                                                                   [µm]
                                                                                                        2.0     2.5      3.0                  0.0    0.5    1.0   1.5
                                                                                                                                                                  [µm]
                                                                                                                                                                       2.0     2.5      3.0
                                                                                                                                                                                                    formance of 1.27 Pflop/s by utilizing 4096        world's No.1 supercomputer.
             Limits csr : [0.01, 1.3] mM; mean 1.29 mM
                                                               0.5
                                                                             Limits csr : [0.01, 1.3] mM; mean 1.22 mM
                                                                                                                               0.5
                                                                                                                                            Limits csr : [0.01, 1.3] mM; mean 0.88 mM
                                                                                                                                                                                              0.5   nodes on Tianhe-2.                                                                                   NOTUR
                                                                                                                                                                                                                                                      Instead of passively waiting for the arrival       Last but not least, a remark about the
                                                               20                                                              20                                                             20    FINDINGS                                          of exascale supercomputers, we believe             NOTUR facilities is in order here. The Abel
                            cCSQN at t=1 ms                                                 cCSQN at t=8 ms                                                cCSQN at t=24 ms                         We used Tianhe-2 to carry out several sim-        that progresses can be made in the current         system, in particular, has played a central
                                                               19                                                              19                                                             19
             3.0                                                             3.0                                                            3.0                                                     ulations of calcium dynamics within one           numerical scheme to facilitate faster simu-        role in developing our software code. Most
                                                               18                                                              18                                                             18    sarcomere, with 3-nm resolution. Among            lations. While the current scheme uses a           of the initial programming work was done
             2.5                                                             2.5                                                            2.5
                                                               17                                                              17                                                             17    other things, we simulated the evolution of       unified small time step that is dictated by        and tested on Abel, before further porting
             2.0                                               16            2.0                                               16           2.0                                               16
                                                                                                                                                                                                    a calcium wave that started with an initial       the strongest diffusion term, an obvious           to Tianhe-2. The same was true for numer-
                                                                                                                                                                                                    increase in the calcium concentration due         improvement lies in allowing individually          ous coarse-resolution simulations that
      [µm]

                                                                      [µm]

                                                                                                                                     [µm]

                                                               15                                                              15                                                             15
             1.5                                                             1.5                                                            1.5                                                     to a spontaneous activity at two CRUs. The        determined time steps for the different            were used to determine the many para-
             1.0
                                                               14
                                                                             1.0
                                                                                                                               14
                                                                                                                                            1.0
                                                                                                                                                                                              14    simulation then confirmed a subsequent            diffusion and reaction terms. The conse-           meter values. As long as access to top-
                                                               13                                                              13                                                             13    spread of the activity. With an increased         quence is that the computing time used on          range supercomputers remains indirect
             0.5                                                             0.5                                                            0.5                                                     calcium concentration, the simulation also        the reaction terms will become negligible          and/or restricted, it is vital to have easy
                                                               12                                                              12                                                             12
                                                                                                                                                                                                    revealed an increased probability in              in comparison with the diffusion terms. An-        access to middle-range computing facili-
             0.0                                                             0.0                                                            0.0
               0.0    0.5    1.0   1.5  2.0     2.5      3.0   11              0.0    0.5    1.0   1.5  2.0     2.5      3.0   11             0.0    0.5    1.0   1.5  2.0     2.5      3.0   11
                                                                                                                                                                                                    triggering neighboring CRUs, which is the         other more advanced improvement is to              ties, such as provided by NOTUR.
                                   [µm]                                                            [µm]                                                           [µm]
                                                               10                                                              10                                                             10    mechanism behind calcium wave propaga-            avoid using a uniform global mesh over the
             Limits cCSQN : [6.07, 20.1] mM; mean 20.04 mM                   Limits cCSQN : [6.80, 20.1] mM; mean 19.18 mM                  Limits cCSQN : [4.88, 18.8] mM; mean 16.37 mM
                                                                                                                                                                                                    tion. Although such preliminary numerical         entire 3D solution domain. Instead, nano-
Figure 2: Snapshots of a 2D slice from a 3D simulation of a calcium wave, first published in Chai et al. (2013)                                                                                     experiments were only a proof-of-concept,         meter resolution should only be applied in         Reference
                                                                                                                                                                                                    nanometer-resolution simulations have             certain small regions, such as within and
volume exterior to the three subpartitions,                          A total number of 4x56=224 threads are                             single-coprocessor implementation                           demonstrated the potential of becoming an         around the CRUs. In the remaining vast             J. Chai, J. Hake, N. Wu, M. Wen, X. Cai, G. T. Lines,
the CPU host is also responsible for con-                            spawned per Xeon Phi 31S1P coprocessor                             achieved 138 GB/s, which amounted to                        indispensable research tool.                      areas a much coarser mesh resolution               J. Yang, H. Su, C. Zhang & X. Liao (2013). Towards
trolling all the necessary intra-node and                            on Tianhe-2. To achieve good data reuse in                         more than 90% of Xeon Phi's realistically                                                                     should suffice. On the other hand, this            simulation of subcellular calcium dynamics at
inter-node data movements.                                           the caches, we have adopted loop fusion                            achievable memory bandwidth. In terms of                    With respect to high-performance comput-          approach should not adopt completely               nanometre resolution. The international Journal
                                                                     and hierarchical blocking. Moreover, we                            number of floating-point operations per                     ing, our experiences with Tianhe-2 show           unstructured meshes, because these are             of High Performance Computing Applications.
The computations offloaded to each co-                               have enforced explicit reuse of Xeon Phi's                         second, the achieved performance was 118                    that it is possible to effectively use the cut-   inherently slow when used on super-                DOI: 10.1177/1094342013514465
processor are also triply nested for-loops.                          vector registers, in addition to using the                         Gflop/s, amounting to 11.8% of the theoreti-                ting-edge MIC architecture for numerically        computers. A good compromise is to use a
OpenMP threads have therefore been used                              512-bit SIMD instructions. Measured in the                         cal peak double-precision capa-bility of the                solving a coupled system of several reac-         patch-wise uniform mesh, where the differ-
for parallelization within each coprocessor.                         actually obtained memory bandwidth, our                            coprocessor.                                                tion-diffusion equations. The good                ent patches are assigned to the coprocessors

16                                                                                                                                                                                                                                                                                                                                                          17
PHYSICS OF EARLY UNIVERSE WITH ATLAS AT CERN'S LHC
INTRODUCTION                                        exploration, drilling and completion, production                nanoparticle multiphase system. Nano-
Photo: Shutterstock

                                                                                                                      Energy is the number one in the humanity’s top      and especially enhanced oil recovery. This                      particles are placed in middle of both oil and
                                                                                                                      ten problems for the next 50 years. Currently,      article presents some insights on the effect of                 water aqueous. After a simulation time of
                                                                                                                      the main source of energy comes from the            nanoparticle on oil/water binary mixture                        4 nanoseconds, nanoparticles move towards
                                                                                                                      easily accessible energy such as oil, gas, and      transport through a nano-sized channel by MD                    and stick to the clay wall due to both van der
                                                                                                                      coal. And the oil and gas resource will still       simulation [1]. In addition, we also discuss the                Waals and coulombic force between nano-
                                                                                                                      dominate the energy mix in near future,             study of nanoparticle-based smart cement for                    particles and clay. This implies that the pres-
                                                                                                                      although renewable energy such as solar, wind       oil and gas production by using large scale                     ence of nanoparticles changes the interface
                                                                                                                      power, bio-mass/-fuel, and geothermal               computational resource.                                         properties between aqueous and clay due to a
                                                                                                                      energy, will experience fast growth. With the                                                                       new surface formation by nanoparticle. The
                                                                                                                      passing of the era of easy oil and the in-          NANOPARTICLES FOR                                               same phenomenon is observed for the cases
                                                                                                                      creasing difficulty of finding new economic         ENHANCED-OIL-RECOVERY                                           of nanoparticle placed only in oil or in water.
                                                                                                                      resources, much attention of the traditional oil    Up to now, oil recovery has experienced its
                                                                                                                      and gas industry has been directed to extract       primary and secondary stages, some bottle-                      To reveal the effect of nanoparticles on the oil-
                                                                                                                      more resources from existing mature oilfields       necks are met in the conventional methods.                      water flow through the nanochannel, three
                                                                                                                      and from the fields exposed to extremely harsh      One of the most promising potentials is to                      cases of one, two and four nanoparticles placed
                                                                                                                      environments by exploring new technologies          utilize the nanoparticles to increase the                       in water phase of left reservoir were investi-
                                                                                                                      and solutions.                                      recovery efficiency of oil in the reservoirs. It                gated, respectively. Figure 2 displays a typically
                                                                                                                                                                          mainly reflects in theses aspects: changing the                 perspective snapshot of oil/water fluid with
                                                                                                                      Nanoparticles defined between 1 and 100 nano-       properties of the fluid, wettability alternation                four nanoparticles in a clay nanochannel. It is
                                                                                                                      meters in size hold unexpected properties that      of rocks, advanced drag reduction, strength-                    observed that the flow pressure of oil-water
                                                                                                                      differ remarkably from those observed in bulk       ening sand consolidation, reducing the inter-                   mixture with nanoparticle through the confined

                      ENGINEERING NANOPARTICLES                                                                       materials primarily originating from the large
                                                                                                                      surface-area-to-volume ratio. Structurally, the
                                                                                                                                                                          facial tension and increasing the mobility of
                                                                                                                                                                          the capillary-trapped oil. MD simulation is a
                                                                                                                                                                                                                                          nanochannel is found to be strongly channel
                                                                                                                                                                                                                                          size dependent. The presence of nanoparticles

                      FOR OIL FIELD APPLICATIONS
                                                                                                                      decreasing of particle size can directly result     powerful tool to understand the deformation                     not only alters the dynamical and structural
                                                                                                                      in changes in spacing between atoms. This           and flow behavior of nanoparticles and its link                 properties of oil-water-clay systems, but also
                                                                                                                      effect is related to the compressive strains        to the atomistic structures. A fundamental                      enhances the oil-water flow through the nano-
                                                                                                                      induced by the compression of surface atoms         understanding of the role of nanoparticles on                   channel at a small nanoparticle concentration,
                                                                                                                      as a result of reduction of surface energy.         the oil-water binary mixture in a confined                      which implies that nanoparticles can be poten-
                                                                                                                      Small nanoparticles therefore adopt a               nanochannel was studied by using MD simula-                     tially utilized for enhanced oil recovery. While
                                                                                                                      different crystal structure than their bulky        tions based on CLAY force-field. A set of                       the nanoparticle-free oil-water flow behaves
                                                                                                                      counterpart. Once the compressive strain            computational experiments in which hydro-                       in a laminar flow manner, with the increase
                                                                                                                      becomes critical, phase transformation              philic silica nanoparticles mixed with an                       concentration of nanoparticle the flow tends to
                                                                                                                      spontaneously occurs. From the chemical point       oil-water system confined in kaolinite clay                     change from a laminar to turbulent type.
                                                                                                                      of view, size reduction changes the chemical        nanochannels were performed by MD                               The nanoparticle first slides along the clay wall
                           Nanoparticles have already contributed to the technological advances in a variety of       reaction ability due to the increase in surface     simulations.                                                    in a local low velocity with the process of break-
                                                                                                                      area to volume ratio. For example, the                                                                              ing and forming hydrogen bonds, whereas it
                           industries, such as medicine, electronics, biomaterials and renewable energy production,   utilization of nanoscale particle catalysis can     We first examine the effect of nanoparticle on                  rotates forwards in a local high velocity
                                                                                                                      significantly enhance the rate, selectivity and     a clay-oil-water system without a driving pres-                 aqueous environment, and escapes from the
                           etc., over the last decade. Recently, a new interest springs up with the application of    efficiency of chemical reactions, and simulta-      sure. Figure 1 presents initial and equilibrium                 attractive clay wall ultimately to enter into the
                                                                                                                      neously result in reduction of waste and            configuration snapshots of a clay-oil-water-                    nanochannel.
                           nanoparticle-based technology in the upstream petroleum industry. Large-scale molecular    pollution. Moreover, a substance can dissolve
                                                                                                                      easily at nanoscale although may not be
                           dynamics simulations could provide fundamental understanding of the role of smart          soluble in water at micro scale. Mechanically,
                                                                                                                      NTNU nanomechanical lab demonstrated by
                           nanoparticles on the upstream oil production, such as enhancing oil transport in micro/    both experiment and molecular dynamics (MD)
                                                                                                                      simulation that there exists a strong size effect
                           nano-sized channel and the prevention of pressure-induced crack of cement sheath behind    in polymeric particles with diameters at
                                                                                                                      nanometer length scale. The source for the
                           the casing.                                                                                increases in modulus is the increase in
                                                                                                                      relative surface energy for decreasing particle
                                                                                                                      sizes. Because of the unique properties of
                                                                                                                      nanoparticles, it is believed that engineered
                                                                                                                      nanoparticle can play a significant role in an
                                                                                                                      upstream oil and gas industry, including

                                                                                                                                                                             Figure 1 Representative side-views of a clay-oil-water-nanoparticle multiphase system simulation box. (a) Initial configu-
                                                        AUTHORS                                  Jianying He                                                                 ration, one nanoparticle in oil and one in water. (b) Equilibrium configuration, two nanoparticles stick to the clay wall and
                                                       Jianyang Wu, PhD                  Associate Professor,                                                                stand in a water-surrounded environment. Green: water; red: oil; and blue: clay and silica nanoparticle
                                                     Nanomechanical Lab                  Nanomechanical Lab
                      18                                         NTNU                                  NTNU
                                                                                                                                                                                                                                                                                                    19
You can also read