EPIC operations status 2021 - ESA ESAC Pedro Calderón Riaño 05/05/2021

EPIC operations status 2021

                                               Pedro Calderón Riaño

                                                        ESA ESAC

                                                        05/05/2021
ESA UNCLASSIFIED – For ESA Official Use Only                      1

General Status

•   The instruments continue working fine.

•   The routine operations continue being automatic from SOC Mission Planning products.

•   The out of nominal operations are now divided between MOC and SOC
    •   MOC take care of immediate reaction to problems.
    •   SOC continue giving expert support.

                                                                                          2

Incidents since the last meeting
1.   MOS 1 EMCR sw crash, auto reboot to ROM version on 22 May 2019.
         The information about the FW position was lost at the same time (sw issue, no movement). That cause some confusion with the SCP
          procedures, but in the end the instrument was recovered.

2.   PN EPCE warm reset on 16 Sep 2019.
         The EPCE operational sw version is the ROM one, so the reset had no impact. First case in more than 3 years.

3.   MOS 1 +18V, +28V and +32V lines drop to 0 on 11 Oct 2019.
         This time the alarm at ground was disabled by the anti ADC corruption sw, so it was not discovered until the reduction on the CCD
          temperature (the internal heater was left without power) triggered a not disabled alarm. The ground sw was improved for avoid
          repetitions. It is a reoccurrence of NCR#126 (previous case in 2015).

4.   MOS 2 -13V line drop to 0 on 27 Jan 2020.
         A group of HK TLM was fix on they maximum value (raw 255) and this caused the trigger of the ground protection switching OFF the instrument
          (with substitution heaters on).
         By bad luck it happened when both SOC Instruments Experts (RP and myself) were fighting to ESOC for training ESOC SOE on instruments, so
          nobody was available to correct this for hours, and the heaters elevated the temperature of the CCDs of MOS 1&2 (and in less way PN) to about -
          80 degC (thermal excursion of +40 degC) , more on MOS1 (internal heater power can not decrease to 0).
         Another case of NCR#126.

                                                                                                                                                            3

Incidents since the last meeting
5.   MOS 2 +28V line drop to 0 on 8 Aug 2020.
         The recovery was left to be started on the next morning (Sunday). The maximum CCD temperatures were PN -80.0 degC,
          MOS1 -69.23 degC and MOS2 -76.92 degC.

6.   MOS 1 on-board sw crash on 6 Dec 2020.
         The instrument stop sending TLM. Solved via power OFF / ON and reconfiguration.

7.   MOS 1 message indicating Annealing circuit Current Limiter Activation on 8 March 2021.
         It was not real at all (the circuit was OFF at the time). Some kind of SEU in the current limiter, or on the reporting hw & sw.

8.   MOS 2 lines +18V, +28V and +32V drop to 0 on 17 Mar 2021.
         Another occurrence of NCR#126. Quickly solved via power OFF/ON and reconfiguration.

                                                                                                                                            4

NCR#126, volts & temps

                         5

Thermal impact of NCR#126 plus late recovery

                                               6

Comparison with an eclipse
15 hours long, temperatures from 0 to -120, -130 degC, eclipse of more than 1 hour, plus Earth albedo

                                                                                                        7

Eclipses

Since March 2019 (last meeting)…

       • There were 4 eclipse seasons (58 eclipses in total)
       • But there were little incidents (mostly not caused by the instruments).

The orbit is evolving to eclipses easier to by operated, with no more ground stations gaps and with
enough time for the automated operations windows.

MOC on-call SOE now does regular training on how to recover eclipse contingencies.
       • No automatized recoveries yet (MOIS), but may be some day.

                                                                                                      8

Instruments CCD temperatures, all mission

                                            9

Long term monitoring

  Nothing      new to report.

    The Trend Analysis shows stable H/K parameters.
    No clear indications of degradation in any particular
     point.

    For a more complete view of indications of degradation, please see
     the results presented in the Calibration meeting ;-)
                                                                          10

Long term monitoring (examples)

   MOS 1       MOS 2         PN

                                  11

Changes in operations: updates and fixes

• 6 ODBs and 7 RFCs were deployed in the period.

• But no big changes.
    Mostly fixes and little updates.
    And even less related to the EPICs
         Correction of MOS1 LW Super ED EE7912
         DP E/KD006 now have values for 'Power OFF' and 'BSW?' , for avoid confusion by the IXG Spacons
         Corrected the time of MOS1 Filter wheel SEQ "to Close", that previously was 5 seconds longer than the others.
         Update to Merged with Gaia conditions the Safe Blocks of the EPICs (E/K/FE8000)
         ED028 and KD028 "+28v OoL confirmation" for avoid confusions with the EMCR A/D corruption protection DPs
         Added external validation to MOS 1&2 ADC corruption check (E/K1078/80R).
         RADVAL switch for avoid automated action on false radiation.

                                                                                                                          12

Summary :

      The instruments continue working fine.

                                                13

Questions ?

              14

15

Statistics

             Pre- and Post- merge with Gaia (incidents in dark blue)

                                                                       16

Trend analysis rare cases

                            17