Air Safety Through Investigation - Journal of the International Society of Air Safety Investigators
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Air Safety Through Investigation APRIL-JUNE 2021 Journal of the International Society of Air Safety Investigators ESASI and Other Societies Are Using Virtual Communication Tools: Page 28
CONTENTS Air Safety Through Investigation
Journal of the International Society of Air Safety Investigators
FEATURES Volume 54, Number 2
Publisher Frank Del Gandio
4 Analysis of Aviation Accident Videos at NTSB Editorial Advisor Richard B. Stone
By Dan T. Horak, Mechanical Engineer, National Transportation Safety Board, Washing- Editor J. Gary DiNunno
ton, D.C.—The author discusses NTSB efforts to analyze 30 aviation accident videos and Design Editor Jesica Ferry
observes that the number of videos requiring examination is increasing from year to year Associate Editor Susan Fager
with the growth rate for more security cameras and public use of phones to record aviation
events. ISASI Forum (ISSN 1088-8128) is published quar-
terly by the International Society of Air Safety
12 Breaking Airlines Flight Data Monitor Barriers Investigators. Opinions expressed by authors do
By Capt. Bertrand de Courville, Safety Risk Management Consultant and Retired Air not necessarily represent official ISASI position
France Captain, and Corporate Safety Manager—The author provides a history of Flight or policy.
Data Monitors and recognizes initial airline pilot’ concerns. He raises the question of who
owns the Quick Access Recorder (QAR) data and suggests and pilots throughout the world Editorial Offices: Park Center, 107 East Holly Ave-
should have access to this information to improve their flight safety performance. nue, Suite 11, Sterling, VA 20164-5405. Telephone
703-430-9668. Fax 703-430-4970. E-mail address,
16 Reducing Risks for Canadian Air-Taxi Operations isasi@erols.com; for editor, jgdassociates@
starpower.net. Internet website: www.isasi.org.
By Glen Whitney, Regional Senior Investigator (Ontario), Transportation Safety Board
ISASI Forum is not responsible for unsolicited
of Canada; Heather Parker, Senior Human Factors Investigator, Transportation Safety manuscripts, photographs, or other materials.
Board of Canada; Doug McEwen, Regional Senior Investigator (Atlantic), Transporta- Unsolicited materials will be returned only if
tion Safety Board of Canada; and Pierre Gavillet, Regional Senior Investigator (Québec), submitted with a self-addressed, stamped enve-
Transportation Safety Board of Canada—The authors summarize results of a safety issues lope. ISASI Forum reserves the right to reject,
investigation that the Transportation Safety Board of Canada conducted. The investigation delete, summarize, or edit for space con-
examined in detail the state of safety in the air-taxi sector and identified where the safety siderations any submitted article. To facilitate
bar can be raised to reduce the number of accidents, particularly fatal ones. editorial production processes, American Eng-
lish spelling of words is used.
24 A New Safety Investigator Profile
By Daniel Barafani, Latin American Society of Air Safety Investigators (LASASI) Pres- Copyright © 2021—International Society of Air
ident, ICAO AIG Panel Expert, Former Air Force Pilot, Safety and Accident Advisor, Safety Investigators, all rights reserved. Publica-
Junta de Sequridad del Transporte (JST-Transportation Safety Board), Argentina; and tion in any form is prohibited without permis-
Enriqueta Zambonini, LASASI Vice President, Pilot, Air Accident Investigator, Junta de sion. ISASI Forum registered U.S. Patent and
Sequridad del Transporte—The authors briefly examine the evolving air safety investi- T.M. Office. Opinions expressed by authors do
not necessarily represent official ISASI position
gation process and paradigms with a focus on the investigator´s profile as part of that
or policy. Permission to reprint is available upon
historical process. application to the editorial offices.
Publisher’s Editorial Profile: ISASI Forum is print-
ed in the United States and published for profes-
sional air safety investigators who are members
DEPARTMENTS of the International Society of Air Safety Inves-
tigators. Editorial content emphasizes accident
investigation findings, investigative techniques
2 Contents and experiences, regulatory issues, industry ac-
3 President’s View cident prevention developments, and ISASI and
28 News Roundup member involvement and information.
30 ISASI Information
Subscriptions: A subscription to members is pro-
32 ISASI 2021: “Staying Safe: Moving Forward” vided as a portion of dues. Rate for nonmem-
bers (domestic and Canada) is US$28; Rate for
nonmember international is US$30. Rate for all
libraries and schools is US$24. For subscription
information, call 703-430-9668. Additional or
ABOUT THE COVER replacement ISASI Forum issues: Domestic and
Canada US$4; international member US$4; do-
mestic and Canada nonmember US$6; interna-
Due to travel and meeting restrictions brought about by the global pandemic, tional nonmember US$8.
ISASI societies and the International Council have begun to meet virtually to
conduct business, provide training opportunities, and communicate with one
another. ESASI, for example, recently began a regular virtual presentation titled
“FocusOn…” to discuss issues and subjects that require communication be-
tween annual meetings (see page 28). This year, ISASI 2021 will be conducted in
the same fashion—a virtual gathering on a global scale—rather than in person. INCORPORATED AUGUST 31, 1964
2 • April-June 2021 ISASI ForumPRESIDENT’S VIEW
Please continue to adhere to all air safety policies and rules as we emerge
from the pandemic--everything by the book.
INTERNATIONAL COUNCIL MEETS ONLINE
O
n Feb. 25, 2021, ISASI held a virtual International Coun- More information will be forthcoming as the trial proceeds.
cil meeting attended by 20 participants representing During the past year, I made a few appointments to ISASI
most societies. committees. Mid-Atlantic Regional Chapter President Frank
The focus of the meeting was the financial state of Hilldrup is joining the Membership Committee. Roger Cox is
ISASI. Treasurer Robert MacIntosh gave an excellent overview serving as the Audit Committee chair. David King is a member
of the revenues and expenses. As to be expected during the of the Kapustin Scholarship Committee.
ongoing global pandemic, individual and corporate mem- I’m pleased to report that $2,450 has been donated to the
bership renewal is down. However, we continue to gain new Rudolph Kapustin Scholarship Fund in memory of Toby Car-
members, including four new corporate members. He laid out roll. As there were few applicants during 2020 and the seminar
a solid budget projection for this year. The bottom line shows was postponed, we decided to roll the annual scholarships
a projected loss of $28,000. Keep in mind that this projection over to this year. The scholarship recipients’ essays will be
is based on historical data and a number of sound estimated included as part of ISASI 2021 presentations and will be pub-
expectations. The final profit and loss numbers for 2021 will lished in the Forum.
mainly be based on the financial success of the ISASI virtual The ISASI 2021 virtual seminar is scheduled for Aug. 31–
seminar scheduled for August–September 2021. Registration Sept. 2, 2021. The theme of the seminar is “Staying Safe: Mov-
for this seminar is now open on the ISASI website or through ing Forward.” The 2020 seminar in Montréal, Québec, Canada,
was canceled due to the pandemic. The virtual seminar will be
a link provided in an e-mail sent to all members. Information
a departure in format and procedure from our typical semi-
can also be found on page 32 of this issue.
nar but necessary due to the pandemic. Even with efforts to
ISASI Forum is one area in which members can help the Soci-
develop and distribute vaccines, many nations still have travel
ety improve its financial status. For the January-March issue,
restrictions in place and reduced in-person gatherings that ad-
ISASI mailed 929 hard copies, and 393 individual and corpo-
versely affect work life, business meetings, and social events—
rate members received electronic copies. Please send the office and rightly so. The lower registration fee for the seminar will
an e-mail (ann.schull@isasi.org ) if you’d like to switch to an enable members who usually cannot attend the international
electronic issue. You’ll save ISASI money and receive the Forum seminar to participate. The proposed broadcast schedule will
weeks before a mailed copy arrives. In addition, in areas where take multiple time zones into
mail service has degraded or is unreliable, you can ensure that consideration so that most
you get your magazine. participants can log in without
Australian Society President John Guselli and the Applica- having to attend in the middle
tion Process Working Group in cooperation with SRCA, our of the night. More information
website hosting company, is designing and has submitted to is available on ISASI’s website.
the International Council a proposal for an electronic applica- Please continue to adhere to
tion process. The International Council authorized the team all air safety policies and rules
to proceed with a 30-day trial of suggested software with the as we emerge from the pandem-
cooperation of the ISASI office manager and the SRCA con- ic—everything by the book. In
tract analyst. A key element of this proposal requires the active addition, take every possible
participation of the ISASI Membership Committee. National precaution to remain healthy
societies that process their own membership applications and and safe both at home and on Frank Del Gandio
dues collection shouldn’t be adversely affected by this change. the job. ISASI President
April-June 2021 ISASI Forum • 3Analysis of Aviation Accident Videos at NTSB
By Dan T. Horak, Mechanical Engineer, National Transportation Safety Board, Washington, D.C.
Video Analysis Background Video Analysis Cases Classified by extrapolation category.
Over the last several years, videos Complexity The main problem facing the analyst
became a major source of information In some runway accidents, the airplane of extrapolation cases is that small an-
for aviation accident investigations. passes by reference points seen in videos gular errors of the camera optics model
To date, the NTSB analyzed more than recorded by airport cameras. Speed parameters, i.e., the camera model yaw,
30 aviation accident videos. In these estimates in such cases can be derived pitch, roll, and horizontal field of view
accidents, videos were one of the main by dividing the traveled distance, usually angles, result in large trajectory, alti-
sources—and often the only source—of along a runway, by the time it took to tude, and speed errors of the airplane
information for estimating trajectories, travel that distance. Analysis of such that is far away. These small angular
altitudes, speeds, and orientation angles low-complexity cases is not discussed in errors are not detectable during camera
of crashing airplanes. These videos this paper. model calibration if all the reference
were recorded by cameras mounted The more than 30 cases mentioned points used for calibration are near the
on airport structures, on commercial earlier are in the medium-complexity or camera. In other words, the model can
buildings, on private homes, on crashing high-complexity categories. They involve handle accurately the reference points
airplanes, and on airplanes that record- airborne airplanes that do not pass or airplanes located near the reference
ed crashing airplanes. We also analyzed by reference points seen in the videos. points, but it cannot accurately handle
automobile dashboard camera videos Analysis of such accidents requires the airplanes that are far from the refer-
and traffic camera videos that recorded use of mathematical models of camera ence points that were used for camera
crashing airplanes. An increasing source optics. Such models are calibrated using calibration.
of aviation accident videos we analyze reference points on the ground. Once
are cameras and smartphones hand held calibrated, the models can project points Fixed Camera vs. Moving Camera
by bystanders on the ground and even in the 3-D field of view of a camera onto The scenarios in which the trajectory,
by a passenger in a crashing airplane. video frames acquired with that camera. altitude, and speed of an airplane are
The number of analyzed accident The calibration and use of such camera being estimated can also be classified
videos has been increasing from year optics models will be described in detail according to the location of the camera
to year, primarily due to the increasing later in this paper. Video analysis cases that recorded the video. The simplest
number of installed high-resolution and can be classified by type and complexity cases are those in which the camera
high frame–rate security cameras and based on the four criteria described next. location is fixed, typically because it is
the increasing number of bystanders mounted on a building. A higher level
who record accident events with phone Interpolation vs. Extrapolation of complexity involves smartphones
cameras. Since 2008, the NTSB has been If the ground reference points used for and cameras that are hand held by
developing methodology, algorithms, camera model calibration are surround- videographers on the ground. While
and software for analyzing aviation ac- ing the airplane or are close to it, the the camera location is approximately
cident videos. We have reached a point analysis can be viewed as interpolation. constant, the camera orientation is
where accurate estimates of trajectories, Even if the calibrated parameters of changing because the camera is being
altitudes, speeds, and orientation angles the camera optics model are somewhat rotated to keep the airplane in its field
can be derived quickly and successfully inaccurate, if the camera optics model of view. Smartphones and cameras
handling the increasing rate at which can accurately project the calibration allow zoom adjustment while a video is
aviation accidents requiring video anal- reference points onto video frames, it being recorded. Analysis of videos with
ysis occur. will also accurately project points on the changing zoom requires recalibration
Aviation accidents that require video airplane onto the video frames. of the field of view angle in addition to
analysis come in many flavors. Many The extrapolation cases involve availa- recalibration of the camera orientation
aviation accident videos require only a ble calibration reference points that are angles for each analyzed video frame.
basic factual summary report that does all near the camera, such as 50 meters Video recorded by a camera mounted
not involve estimation of numerical from it or closer, and the estimation in an airplane can also be used for es-
quantities. This paper does not discuss of trajectory, altitude, and speed of an timating the trajectory, altitude, speed,
such cases, and the more than 30 cases airborne airplane that can be 500 meters and orientation angles of that airplane.
previously mentioned do not include or farther away from the camera. Most of Analysis of such videos requires the use
such straightforward video analyses. the 30 cases mentioned earlier are in the of a large number of ground reference
4 • April-June 2021 ISASI Forumpoints along the ground track of the
airplane because as the airplane moves,
the reference points located in the cam-
era field of view change. The analysis can
be further complicated if the camera is
hand held by a passenger in the airplane
and is changing its orientation with
respect to the airplane. Analysis of such
videos requires the recalibration of the
camera orientation with respect to the
airplane for each analyzed video frame,
followed by the estimation of the air-
plane location and orientation with re-
spect to ground reference points visible
in that video frame. The NTSB has the
methodology and the tools for analyzing
videos recorded by fixed or movable
cameras whether they are on the ground
or inside flying airplanes.
Airplane Details Visible vs. Airplane
Details Not Visible (Adapted with permission from the
When the image of the airplane in video author’s technical paper Analysis of
frames is sufficiently large so that its Aviation Videos at NTSB presented dur-
details such as nose, fuselage, tail, and ing ISASI 2019, Sept. 3–5, 2019 in The
wings are visible, analysis can be based Hague, the Netherlands. The theme
for ISASI 2019 was “Future Safety:
on wireframe model alignment. A 3-D
Has the Past Become Irrelevant?”
wireframe model of the airplane is The full presentation can be found on
constructed and projected onto a frame the ISASI website at www.isasi.org
Figure 1. Classification of video analyses by from the video using the camera optics in the Library tab under Technical
complexity.
model. The wireframe model is then Presentations.—Editor)
Dan T. Horak
Figure 2. Projection from 3-D field of view onto a 2-D video frame.
April-June 2021 ISASI Forum • 5moved and rotated until the projected image matches the image of the airplane in the video frame. Once an optimal match is achieved, the location and ori- entation angles of the wireframe model are the optimal estimates of the location and orientation angles of the airplane at the time the analyzed video frame was recorded. The movement and rotation of the projected wireframe model is managed by an algorithm that uses the mathematical model of the optics of the camera. This model is described in de- tail later in this paper, and an example later in this paper uses the wireframe model method. When the airplane is far from the camera, its image in a video frame can Figure 3. Frame from the parking garage camera video. be as small as one or several pixels. In such cases, the wireframe model approach cannot be used, and the orientation angles of the airplane can- not be estimated based on the video. Estimation of the location and altitude of the airplane is usually possible, but it requires some additional information, such as the ground track of the airplane. An example later in this paper uses radar-based ground track to supplement the information in a video that does not show airplane details. Many Reference Points Available for Camera Calibration vs. Few Refer- ence Points Video analysis is based on the math- ematical model of camera optics. As described later in this paper, the model Figure 4. Barrel-distorted frame from the airport tower camera video. requires seven parameters that must be estimated in a calibration process that is based on ground references. When there are many available reference points that are distributed throughout the field of view of the camera, calibration is rela- tively simple and the resulting calibrated camera optics model is accurate. In many cases, however, there are few reference points, and they may not be distributed throughout the field of view of the camera. Calibration in such cases is time consuming, and it results in camera optics models that may have lower accuracy. Based on the criteria described above, it is possible to classify video analysis cases by their overall complexity. Figure 1 illustrates this classification. On the bottom, in blue, is the simplest case in Figure 5. Light poles P1-P5 used for calibration of parking garage camera video. 6 • April-June 2021 ISASI Forum
which all four complexity criteria point
to a low-complexity case. On top, in grey,
is the most complex case in which all
four criteria point to a high-complex-
ity case. Most cases are in the medi-
um-high–complexity range, where two
or three criteria point to a high-com-
plexity case.
Mathematical Model of Camera Op-
tics and Its Use for Analysis
Video analysis aimed at estimating tra-
jectories, altitudes, speeds, and orienta-
tion angles of airplanes is based on the
use of mathematical models of camera
optics. The strategy behind the use of
such models is quite simple. Assume
Figure 6. Parking space marking lines used for calibration of garage camera video. that a 3-D model of the airplane, with its
dimensions specified in units of distance
such as meters, is placed and oriented
by an analysis program at a 3-D location
in the field of view of a camera. The 3-D
location is specified in meters, and the
airplane orientation is specified by its
Euler yaw, pitch, and roll angles. The
analysis program is then used to project
points on the airplane model onto
frames from the analyzed video using
the mathematical model of camera op-
tics. These points can be located on the
airplane nose, tail, and wingtips and on
the fuselage and the wings, depending
on the visibility of airplane details in the
video.
Figure 2 illustrates the computational
process of projecting points in the 3-D
field of view of a camera onto 2-D video
Figure 7. Frame from the parking garage camera video with marked reference points. frames, simulating the process cameras
use to record video frames. The camera
sensor in the figure is at the location of
the camera that recorded the video. The
airplane model is located and oriented
in the 3-D field of view of the camera.
The image plane is placed in front of the
camera sensor and is oriented according
to the orientation of the camera. A point
on the airplane model is projected onto
a point in the video frame that is at the
intersection of the image plane with a
line from the camera to that point on
the airplane model.
If the projected airplane model points
are accurately placed on the images
of these points on the real airplane as re-
corded in the analyzed video frame, then
the airplane model 3-D location and its
Euler angles used by the analysis pro-
Figure 8. Block diagram of the calibration of the parking garage camera model. gram are the accurate estimates of the
April-June 2021 ISASI Forum • 7real airplane location and orientation
angles. The process of aligning the pro-
jected points with their images has two
stages. First, the mathematical model of
camera optics must be calibrated. The
model has seven parameters. Three are
the X, Y, and Z location coordinates of
Figure 9. Camera model projection capability from 3-D to 2-D. the camera. Three are the yaw, pitch, and
roll orientation angles of the camera.
The seventh parameter is the horizontal
field of view angle of the camera.
The seven camera model parameters
are estimated in an iterative calibration
process where they are varied until
reference points on the ground, pro-
jected onto a video frame, are optimally
aligned with their images in the video
frame. At that time, the values of the
seven camera parameters are their op-
timal estimates. The references used for
calibration typically include points on
buildings, roads, runways, and taxiways.
These points must be visible in the video
frame, and their ground coordinates
must be known from aerial images or
from an optical survey of the area. The
resolution of Google Earth aerial images
has become sufficiently high in recent
Figure 10. Video frame recorded when the airplane first became visible. years so that optical surveying is needed
track
ground
Figure 11. Accident area with superimposed ground track and azimuth direction from the parking garage camera to the airplane.
8 • April-June 2021 ISASI Forumonly infrequently.
Once the camera is calibrated, the
location and orientation of the airplane
is estimated in the second stage of the
analysis process. The location and the
orientation angles of a 3-D wireframe
airplane model are varied in an iterative
process until the points on it, project-
ed onto a video frame, are optimally
aligned with their images in the video
frame. At that time, the three location
coordinates and the three Euler an-
gles are the optimal estimates of these
parameters of the airplane at the time
the analyzed video frame was recorded.
This airplane location and orientation
estimation process is repeated for each
analyzed video frame.
In many cases, the details of the Figure 12. Aerial view of the airport with marked reference points.
airplane are not visible in a video frame.
The wireframe model of the airplane in
such situations is just a point. While it is
not possible to estimate the orientation
angles of the airplane based on an image
that is just a point, partial information
on the location of the airplane can be
derived and fused with information from
other sources to derive an estimate of
the location of the airplane.
The calibration and the use of math-
ematical models of camera optics is
illustrated next using the analysis of a
recent accident. It involves both the use
of the wireframe model approach and
the fusion of information from a video
that does not show airplane details with
radar information.
Figure 11. Accident area with superimposed ground track and azimuth direction from the
parking garage camera to the airplane.
Description of the Analyzed Accident
NTSB accident number DCA17FA109 is other camera was on the airport control the descending airplane as it emerged
used to demonstrate the video analy- tower. It displayed new video frames at from the cloud cover. The estimated
sis process. A Shorts SD3-30 airplane the rate of 2.857 frames per second. altitude of the airplane when it became
crashed during landing on May 5, 2017, Figure 3 shows a frame from the park- visible in the video for the first time was
on Runway 5 at Charleston Yeager Inter- ing garage camera video. It was taken considered an estimate of the overcast
national Airport (CRW) in Charleston, before the airplane became visible. ceiling. This video-based estimate was
West Virginia. The airplane was de- Figure 4 shows a top segment of a frame used to determine whether the reported
stroyed and the two pilots suffered fatal from the airport tower camera video. It 500-foot (152 meters) overcast ceiling
injuries. The flight was a scheduled was taken before the airplane became was accurate.
cargo flight from Louisville, Kentucky. visible. The airport tower camera video The airport tower video recorded
At the time of the accident, weather was frame shows severe barrel distortion the airplane as it impacted the ground
reported as an overcast ceiling at 500 caused by the wide field of view angle of on the runway. Analysis of this video
feet (152 meters). Two cameras record- the camera. provided estimates of the airplane speed
ed the airplane as it was approaching and orientation angles at the time of
the runway. One camera was on the top Accident Analysis ground impact. The analyses of the two
floor of a parking garage building in the The two videos recorded information videos are described next.
city of Charleston, about 2 miles (3.2 that was extracted and analyzed to pro-
kilometers) from the airport runway. Its vide insight into two aspects of this acci- Camera Calibration
frame rate was 6 frames per second. The dent. The parking garage video recorded The analysis of this accident required
April-June 2021 ISASI Forum • 9a calibrated mathematical model of Figure 9 illustrates the capability of airplane as it was approaching the
the camera optics of each camera. The the mathematical model of camera op- airport.
mathematical model of camera optics tics to project points from the 3-D space Figure 11 shows an aerial view of the
requires seven parameters. Three are the in the field of view of a camera onto a accident area. The ground track of the
X, Y, and Z camera location coordinates. 2-D video frame. It shows that the model airplane derived based on radar data
Three are the yaw, pitch, and roll camera can project from the 3-D space that in- is shown with a solid line in the figure.
orientation angles, and the seventh pa- cludes large amount of information onto The dashed line is the azimuth direction
rameter is the camera horizontal field of a 2-D video frame that includes much from the parking garage camera to the
view angle. The X and Y location coordi- less information. airplane that was estimated with the
nates of both cameras in this accident However, this is not what is needed camera optics model based on Figure
could be measured in Google Earth. The for analysis of accident videos. We need 8, as described above. The video anal-
other five parameters of each camera information to flow in a direction oppo- ysis estimated the azimuth angle and
had to be estimated. site to what is shown in Figure 9, from the elevation angle from the camera to
The estimation was based on refer- the small amount in a 2-D frame to the the airplane but not the location of the
ence points that were visible both in 3-D space where three coordinates are airplane along that direction. Fusing
video frames and in aerial images. The needed to specify a location. This may the video information and the radar
references used for the parking ga- initially look like an impossible task. information made it possible to estimate
rage camera calibration included five However, it becomes possible when the the ground coordinates of that location.
highway light poles and parking space 3-D to 2-D projection capability of the That location is at the intersection of
markings. The light poles were located camera model is combined with addi- the radar-based ground track solid line
between 575 feet (175 meters) and 1,200 tional information. That information can and the video-based azimuth dashed
feet (366 meters) from the camera. They come from sources such as wireframe line seen in Figure 11.With the ground
are marked on the Google Earth aerial model alignment or a known ground coordinates of the airplane location
image in Figure 5. The parking space track or a second camera. The example estimated, the altitude of the airplane
markings were located 130 feet (40 me- below illustrates the use of such addi- could be estimated by multiplying the
ters) or less from the camera. They are tional information sources. ground distance from the camera to the
shown in the Google Earth aerial image airplane by the tangent of the eleva-
in Figure 6. tion angle. The estimated altitude was
Analysis of Parking Garage Camera
Figure 7 shows a frame from the park- 683±60 feet above the airport runway.
Video
ing garage video with marked reference This estimate is based on cloud cover
Once the parking garage camera model
points that were used for calibration. at a location about 3,800 feet (1,158
was calibrated, it could be used for
The two points on each light pole were meters) west of the landing spot on the
analysis of the video. Figure 10 shows
placed at fixed heights above ground airport runway.
the first frame from the video in which
because the heights of the light poles the descending airplane could be seen.
were not known. Analysis of Airport Tower Camera
It is marked by the yellow circle. Be-
Camera optics model calibration of Video
cause of the distance from the camera,
each camera was performed as follows. Figure 12 shows an aerial image of the
no airplane details are visible. The
A computer program that simulates airport with marked reference points
estimation of the distance of an airplane
camera optics was used to project the that were used for airport tower camera
from a camera is ideally based on the
reference points onto frames from the calibration. Figure 13 shows the frame
dimensions of the airplane image in a
video in an iterative process in which from Figure 3 after the barrel distortion
video frame using the wireframe model
the five unknown camera parameters was mathematically corrected. When
approach. However, since the airplane
were varied so as to align the project- compared to the distorted video frame
image in this case was only a dot in the
ed references with their images. When in Figure 3, the pixels near the corners
video frame, the distance could not
the projected references were aligned of the frame in the corrected frame are
be estimated this way, and without a
optimally with their images in the frame, located farther away from the center
distance estimate the altitude of the
values of the five parameters were their of the frame. Marked on the frame are
airplane could not be estimated either.
optimal estimates. At that point, the the reference points that were used for
Estimating the altitude was the goal be-
mathematical model of the camera op- airport tower camera calibration. These
cause it was an estimate of the overcast
tics was calibrated. points correspond to the reference
ceiling. The only quantities that could
Figure 8 shows a block diagram of the points marked in Figure 12. The calibra-
be estimated without any additional
calibration process of the parking garage tion process was similar to the calibra-
information were the azimuth direction
camera where a frame from the video tion of the parking garage camera, i.e.,
and the elevation angle from the camera
and an aerial view of the scene covered using the block diagram shown in Figure
to the airplane.
by the camera are analyzed to generate 8 with Figure 13 and Figure 12 being the
The additional information that made
a mathematical model of the parking video frame and the aerial view, respec-
estimating the altitude of the airplane
garage camera optics. The calibration of tively.
possible as it emerged from the cloud
the airport tower camera followed the The airport tower camera video
cover was radar data. Analysis of radar
same logic and will be described later. was used for estimating the speed of
data provided the ground track of the
10 • April-June 2021 ISASI ForumFigure 14. Frame from the airport tower camera showing the airplane and its wireframe model shortly before ground impact.
ground impact and the orientation of The calibrated camera model was then 42 degrees at the time of ground impact
the airplane as it impacted ground. The used to project the wireframe model and the nose-down pitch angle of the
airplane was visible in seven frames in onto frames from the video. The model fuselage was estimated as 14 degrees.
the video over approximately 2.4 sec- automatically projected the 3-D wire-
onds. Only the last three frames showed frame model dimensioned in units of Conclusion
the fuselage and both wings. In earlier distance into its 2-D image in a video This paper described the aviation
frames, part of the airplane was not in frame, dimensioned in pixels. The cam- accident video analysis activities at the
the field of view of the camera. era model was then used to iteratively NTSB. The analyses were classified
Analysis of the airport tower camera move and rotate the wireframe model based on their type and complexity. The
video was based on a wireframe model until its projection coincided optimally core component of the tools used for
of the airplane. Such models can consist with the image of the airplane in a video video analysis, the mathematical model
of points on the fuselage, the tail, and frame. At that time, the location and of camera optics, was introduced and
the wings. The points can optional- orientation of the wireframe model were explained. The analysis of a recent case
ly be interconnected with lines. The the optimal estimates of the location was then described in detail. The
wireframe models are dimensioned in and orientation of the accident airplane accident involved an airplane that
units of distance, such as meters or feet, at the time the analyzed video frame crashed at an airport while attempting
corresponding to the actual dimensions was recorded. to land. Videos from two cameras were
of the analyzed airplane. used for estimating the overcast ceiling
In this case, because of the distance Figure 14 shows the last video frame at the time of the accident, the speed of
from the camera, only points on the before the left wing of the airplane con- ground impact, and the orientation of
nose, tail, and wingtips could be pin- tacted the ground and broke. It shows the airplane at the time of ground
pointed in the video. Consequently, only the wireframe model optimally superim- impact. The analysis required calibrated
these points were used in the wireframe posed on the image of the airplane. The mathematical models of the optics of
model. The model nose was marked in previous and the next video frames were the two cameras and used fusion of
grey, the tail in black, the left wingtip analyzed in a similar process. The three video and radar information for extract-
in white, and the right wingtip in blue. estimated locations of the airplane were ing airplane altitude data from one of
The nose and tail markers were inter- then used to estimate the magnitude the videos.
connected with a light blue line, and the of the velocity vector of the airplane. It
wingtips were interconnected with a was estimated as 92±4 knots. The left-
dark grey line. wing–down roll angle was estimated as
April-June 2021 ISASI Forum • 11Breaking Airlines’ Flight
(Adapted with permission from the author’s technical paper Breaking Airlines Flight Data
Monitoring Barriers: A Pilot’s Perspective presented during ISASI 2019, Sept. 3–5, 2019, in The
Hague, the Netherlands. The theme for ISASI 2019 was “Future Safety: Has the Past Become
Irrelevant?” The full presentation can be found on the ISASI website at www.isasi.org in the Library
tab under Technical Presentations.—Editor)
Some Flight Data Monitoring Pioneers necessarily be the result of a bad pilot’s
If someone tells you that fog in France decisions and actions. Such a tool could
and the UK helped to develop the first have been very efficient to detect, judge,
flight data monitoring (FDM) program in and discipline those bad pilots—the
the early 1970s, you’d probably not take perfect way to prevent future accidents.
that very seriously. Yet this is part of the These perception and fears, which were a
real story. bit caricatural, were nevertheless widely
Until the mid-1960s, flight data re- shared, and, therefore, pilots’ fears were
corders (FDRs) were exclusively used for not totally unfounded.
accident investigations. This started to Using recorded data for the benefit of
change when data recorders began to be CAT III operations was not an issue, but
used for aircrafts airworthiness aspects doing so to monitor all phases of flight
related to autopilot performance. New from all flights meant something different
separate digital recorders apart from for the pilot community. Any flight (i.e.,
crash recorders were developed together any pilot) could be monitored by anyone
with software on the ground to automati- who had access to the records, even by
cally monitor and detect specific param- those who had little or no flight opera-
eters or flight path exceedance. This use tions expertise.
was soon extended to certifying the first FDM programs had great value and
CAT III autoland operations for the Cara- deserved to be implemented, but this had
velle in France and the Trident in the UK. to be done with great precaution. This
Thanks to this data monitoring program, was discussed within the IATA Safety
passengers could be flown into foggy Advisory Committee and more particu-
airports on days when manual landings larly among British Airways, Air France,
would have been impossible. and TAP representatives who were all
The use of FDRs soon went much from airlines involved in CAT III autoland
further, as it appeared that the nature and certification programs at that time.
the amount of data made available could Eventually, people of goodwill from
be used to monitor much more types of pilots’ organizations and airline man-
exceedances. Therefore, why not use it to agements were able to overcome these
detect near accidents of many types? This very real obstacles. They were deeply
was an excellent idea, but in the early convinced that, if properly used, record-
1970s, human factors knowledge as well ed data could contribute efficiently to
as “just culture” were not as mature as prevent accidents.
they are today. At Air France, an FDM agreement was
The notion of airline pilots having signed in April 1974. To demonstrate to
the “right stuff ” was dominant. Events all pilots the value of the program, an
detected through these programs had to FDM bulletin was created and published
12 • April-June 2021 ISASI ForumData Monitoring Barriers
By Capt. Bertrand de Courville, ISASI Member, Retired Captain,
and Corporate Safety Manager, Air France, Safety Risk Management
Consultant
four month later, in August 1974, to share garding FDM information sharing and in-
the most interesting events. But with- novations. FDM rules in place within the
out input from the pilots involved, the agreements were so efficient and so well
lessons were limited and sometimes only respected that no one dared to change
hypothesis. anything. Many initiatives to make FDM
In 1984, the FDM agreement was more flexible for the benefit of safety or to
modified in order to have access to pilots’ introduce some innovation were always
feedback. A nonpilot trusted person—at feared because they might undermine
this time there was no notion of gate- the precious agreement. It took 10 years
keeper—was assigned to contact pilots for Air France, one of the FDM pioneer
and was responsible for keeping their airlines, to be able to communicate FDM
anonymity. events to the interested crews and get
At British Airways, a very similar pro- in return, anonymously, their valuable
gram and solutions were developed—still feedback.
in use today—known as SESMA (Special Having in mind that any misuse could
Event Search and Master Analysis). An jeopardize the whole program, what had
agreement was made with the BALPA un- been put in place was most often consid-
ion stipulating that only the fleet-specific ered as enough and interesting sugges-
BALPA SESMA representative was able to tions were dismissed. Although these
identify the pilot. In fact, an independent FDM programs worked well at some air-
trusted flight ops manager had the ability lines, they were nearly frozen and stayed
to find out which crew was involved in a resistant to any change.
SESMA event and pass that name to the In the meantime, digitalization was
union rep. transforming all aspects of flight op-
This was a major innovation for flight erations, including many other safety
safety. Nevertheless, it remained limited processes. As a result, during more than
to a small number of airlines, mostly 40 years of the Air France, British Air-
European, for decades before beginning ways, and TAP initiative, individual pilots
to be adopted slowly across the world. were kept from their flight data. Only
This process required onboard equipment when something wrong had happened,
as well as software and human resources and an FDM event had been identified,
on the ground. And above all, it need- pilots could receive their data. It is no
ed strong internal rules to maintain a surprise that pilots’ and airlines’ attitude
minimum level of trust. FDM worked, but regarding everything related to flight data
most pilots still perceived it as a threat. monitoring remained defensive to the
While pilots understood the need for point that FDM potential stayed largely
recorded flight data analysis in case of underexploited.
accidents, they were very cautious about
its use for daily flights. But beecause “con- Who Owns the QAR Data?
fidentiality” and “anonymity” barriers During the recent EASA/FAA conference
were robust enough, they simply trusted in Cologne, Germany, a participant to a
the program. conference session dedicated to safety
As necessary as they were, these bar-
riers had serious drawbacks mainly re-
intelligence asked, “Who owns the QAR Capt. Bertrand de Courville
data?” The question is simple, but the
April-June 2021 ISASI Forum • 13answer is not. The session
Photo: Courtesy CEFA Aviation
panelists did not really an-
swer it. However, it deserves
more attention, at least in
the context of this article.
One pragmatic approach is
to say that recorded flight
data belongs to those who
use it. So let’s have a look
at these users. First, we
have experts from airline
safety departments who
may process the data either
directly through their own Figure 1: QAR data processing for both mobile service and FDM
program.
program or through FDM
subcontractors. We also to data, some may also say that pilots may ask their airline
have experts from large or- whenever they want to look at part of their data. This is true
ganizations such as the FAA for only a limited number of airlines, and these airlines have no
through its ASAP where its resource to respond if requests become too numerous. When
subcontractor is processing this happens, pilots are often given nonintuitive lists of figures
large amounts of aggregated and curves with little explanations. This does not encourage
data provided by airlines. pilots to ask again.
Experts from manufacturers
In some cases, pilots could visit the department and have a
are also collecting and pro-
look at the 3-D animation tool used by the safety department,
cessing operational record-
when the safety managers have time to do so. Yet pilots are
ed data as well as experts
also hesitant about asking for recorded data coming from their
from research teams work-
own flights, even when there is no deviation, because they fear
ing closely with airlines on
being suspected of having made some errors and not reporting
very specific safety topics.
them. Again, the same defensive attitude regarding recorded
There are also consulting
companies specializing in data is dominant.
fuel cost reduction that have The truth is that well into the digital age, more than 40 years
contracts and agreements after the first FDM programs, individual airline pilots still do
to use aggregated data. All not have a direct access to their own flight data. Today, even
these programs and con- general aviation pilots can look at a record of their flight path
tracts are formally defined on their tablets.
to guarantee the anonymity
of data. Individual pilots are Breaking the Barriers
never identified, and airline In 2017, All Nippon Airways broke the barriers by giving its
anonymity is also part of pilots a direct access to their flight data on their profession-
the agreements in programs al tablets. This was made possible thanks to a powerful and
such as ASAP in the U.S. and well-thought-out combination of Internet and data processing
more recently EASA’s Data tools. Flight data is translated and displayed under an intuitive
for Safety (D4S). All experts format consisting of detailed video animations of the cockpit
involved in these activities instruments simultaneously with the aircraft flight path. Thus,
can be considered as users each of the 3,000 All Nippon Airways pilots could freely replay
and as detaining a part of a key animated periods of their own flights. Since then, a daily
shared data ownership, but average of 200 video animations are reviewed.
not the pilots. In the future, in addition to the very first users (All
Nippon Airways and Ryanair), other airlines will use similar
Pilots of These Flights Are programs. This can be considered as a major and long-await-
Not Users ed step, comparable to the birth of FDM in the early 1970s in
Some may say that pilots Europe.
benefit from this flight data First, how does it work? After each flight or series of flights,
indirectly and collectively QAR data is sent as usual to the airline FDM server through
through training programs a wireless communication channel. But in addition, this QAR
oriented through or safety data is now encrypted and uploaded into a dedicated, secured
awareness bulletin based on server in which high-performance video animation software is
deidentified data analysis. placed. When a pilot needs to review their take off or approach
Regarding individual access and landing, the pilot just opens the list of previous flight
14 • April-June 2021 ISASI Forumnumbers displayed on a tablet through the impossible to be memorized correctly. Even
application and clicks on the specific flight. experienced flight instructors are missing
The software then creates the animation in significant aspects of a pilot’s performance.
the cloud and makes it accessible (see Figure As an example, autopilot mismanagements
1). scenarios are often very dynamic and com-
Each commercial aircraft type is availa- plex. By having access to their own data,
ble in this mobile service in order to make pilots can review and better understand a
images as realistic as possible, particularly sequence of actions, not only by memory but
regarding cockpit controls and instruments from what has really been recorded. Doing
displays. so while the crew is still together will make
Various security measures have been de- debriefing possible and much more efficient.
veloped. Pilots have access only to their own We are not far from evidence-based training
flights on a personal account. No one else can principles, brought to an individual level. In
see it. The animations must be reviewed on a a way, such a tool can be compared to videos
streaming mode and cannot be downloaded used in sports to make progress. Any better
on a tablet. Data is deidentified. The airline understanding of things that went well or not
FDM program remains unchanged: FDM help an individual to improve and to build
events are monitored and addressed by the the right level of self-confidence sooner.
safety team the same way. The same confi- This “digital-age approach” is going to be
dentiality and anonymity rules that made beneficial to pilot performance at the front
possible the very first FDM programs at Air line. Because it is much less visible and
France, British Airways, and TAP in 1974 difficult to measure, we may tend to under-
are still in place. Individual pilots are still estimate it, but we must recognize it as a key
protected from any misuse. More protections component of airline safety performance.
can be applied depending on each airline On the other hand, at the safety manage-
policy. ment level and as regards to possible benefits
How is such an innovation justified? Pilots to reporting programs, we may use the same
could say that hundreds of experts from var- example. Automation mismanagement is
ious organizations are looking at “their data” hardly reported on when there are no visible
so why not us? If it is technically feasible effects on a flight either because they are de-
today, at a reasonable cost while complying tected and corrected early enough or because
with anonymity and confidentiality rules, the circumstances at the time prevented
pilots’ access to their own flight data will be them from having any consequence. Being
perceived as a “right” and does not need to be able to replay a sequence of events will give
justified further. But there is more than this. factual evidence and encourage pilots to
A premonitory article published in 1966 by report.
Flight International suggested that more use
should be made of FDRs in normal service to What About the Future?
“monitor pilot approach performance” and
Hideo Morioka, All Nippon Airways senior
that airline managements should be persuad-
director of safety promotion and flight data
ed that “flight recorders are not just crash re-
analysis, said, “It has revolutionized the
corders…they are pilot training aids.” Today,
company culture regarding the debriefings
we could write it this way: “QARs tool are
not only there to detect FDM events, they are and the use of flight data…. It has freed pilots’
pilot training aids.” This is exactly the point. speech.”
From their very first flight hours, pilots are Empowering pilots around the world to
mentally reviewing each of their flights to learn from their own flight data, daily, and
understand what went well and what did not. not only after a visible incident or a deviation
When it relates to a first solo flight, it is not detected through FDM/FOQA programs, has
exaggerated to say that it is a very personal the potential to change pilots’ attitude
question of life or death. Pilot culture is made regarding recorded flight data. From defen-
of that. Airmanship, experience building, and sive, it could become more constructive. Not
learning processes are based on this capacity only because they are now sharing part of the
to question one’s own performance and iden- flight data “ownership,” but also because they
tify potential ways to improve. After a flight, are benefiting directly from it. In addition,
this process is essentially based on a pilot’s new generation of pilots are more familiar
memory, and when it comes to flight training and positive about new IT tools, which make
it is based on the flight instructor’s memory. them more autonomous. This could unlock
In modern and complex aircraft, many historical barriers and make possible future
situations happen in such a way that they are developments beneficial for airline SMS.
April-June 2021 ISASI Forum • 15RAISING THE BAR ON SAFETY:
REDUCING RISKS FOR CANADIAN AIR-TAXI OPERATIONS
By Glen Whitney, Regional Senior Investigator (Ontario), Transportation Safety Board of Canada; Heather
Parker, Senior Human Factors Investigator, Transportation Safety Board of Canada; Doug McEwen, Regional
Senior Investigator (Atlantic), Transportation Safety Board of Canada; and Pierre Gavillet, Regional Senior
Investigator (Québec), Transportation Safety Board of Canada
Introduction from the large dataset. All results were same regulations: Subpart 703 of the
Safety in the air-taxi sector has been the further analyzed within a model called Canadian aviation regulations (CARs).
subject of concern, as well as numer- the safe operating envelope. Recommen- The CARs were drafted to recognize the
ous studies and reviews, for more than dations to address the safety risks in the differences among segments of the in-
27 years. The air-taxi sector, in simple air-taxi sector are presented. dustry, with smaller aircraft (defined by
terms, is comprised of smaller aircraft, certified seating capacity) being subject
both helicopters and airplanes, having Background to less-stringent regulation. The techni-
nine or fewer passenger seats providing The air-taxi sector provides a diverse cal definition of air-taxi operations in the
scheduled and nonscheduled service. CARs is the operation by a Canadian air
array of air services to Canadians. These
It is a challenging sector of commercial operator, in an air transport service or in
include helicopters to transport injured
aerial work involving sightseeing opera-
aviation, given its complex operating or ill patients to hospitals; floatplanes to
tions, of any of the following aircraft:
context, and it experiences a high num- take commuters from harbor to harbor
(a) a single-engined aircraft;
ber of accidents, especially fatal ones. In in coastal cities; and airplanes to bring
the 15 years between 2000 and 2014, the workers to remote areas, provide search (b) a multiengined aircraft, other
Canadian air-taxi sector experienced 716 and rescue, or deliver food, equipment, than a turbojet-powered airplane,
accidents resulting in 227 deaths. These and passengers to communities. These that has a MCTOW (maximum certi-
accidents represent 56% of all commer- vital air links have helped build Canada fied takeoff weight) of 8,618 kilo-
cial aviation accidents in Canada and and sustain its people. In 2015, approx- grams (19,000 pounds) or less and
account for 64% of commercial aviation imately 550 companies in Canada held a seating configuration, excluding
fatalities. an air operator certificate for air-taxi pilot seats, of nine or less;
This paper summarizes the results operations. (b.1) a multiengined helicopter cer-
of a recent safety issues investigation Although air-taxi operations are tified for operation by one pilot and
conducted by the Transportation Safety diverse, they are all covered under the operated under VFR (visual flight
Board (TSB) of Canada. The investigation
examined in detail the state of safety in
the air-taxi sector and identified where
the safety bar can be raised to reduce the
number of accidents, particularly fatal
ones. Data was collected, analyzed, and
combined from accident summaries,
investigation reports, and interviews
with industry participants. The highest
number of fatalities in both airplane and
helicopter air-taxi accidents resulted
from flights that started in visual mete-
orological conditions and continued to a
point where the pilot lost visual refer-
ence with the ground.
There were two key categories of
underlying factors: the acceptance of
unsafe practices and the inadequate
management of operational hazards.
Figure 1. Total number of accidents and fatalities involving Canadian-registered aircraft by
Nineteen safety themes were identified operator type, 2000 to 2017. (Source: TSB)
16 • April-June 2021 ISASI Forum(Adapted with permission from the authors’ technical paper Raising the Bar on Safety: Reducing the Risks Associated with Air-Taxi Operations in
Canada submitted for ISASI 2020 in Montréal, Québec, Canada. ISASI 2020 was postponed until 2021 due to COVID-19 restrictions. The full technical
paper can be found on the ISASI website at www.isasi.org in the Library tab under Technical Presentations.—Editor)
Glen Whitney Heather Parker Doug McEwen Pierre Gavillet
Figure 2. Average flight time for pilots involved in airplane air-taxi accidents compared to the total number of airplane accidents and fatal
accidents during the study period 2000–2014.
April-June 2021 ISASI Forum • 17You can also read
