Presentation Abstracts for the - 2022 NASA Cost and Schedule Symposium April 26- 28, 2022 Hosted near Kennedy Space Center Cocoa Beach, FL
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
National Aeronautics and
Space Administration
Presentation Abstracts
for the
2022 NASA
Cost and Schedule
Symposium
April 26- 28, 2022
Hosted near Kennedy Space Center
Cocoa Beach, FL2022 Abstract Collection The Strategic Investments Division would like to welcome you to the 2022 NASA Cost and Schedule Symposium. This document contains the names of the authors and abstracts for the presentations that will be given this year. In the Symposium Agenda you will notice that there is a unique ID number mapped to each presentation. These same ID numbers can be used, within this document, to find the presentation abstract that you are interested in. This year, with a cadre of excellent presentations and an awards banquet full of worthy nominations, the NASA Cost and Schedule Symposium will be a full and eventful three days!
Contents 01_ Dealing with Missing Data – The Art and Science of Imputation .......................................................... 4 03_ NASA Programmatic Performance........................................................................................................ 4 04_Chief Program Management Officer Overview ..................................................................................... 4 05_Earned Value Management, Year in Review.......................................................................................... 5 06_ COMPACT KNN V2: Analogy-Based Cost Estimation Model for CubeSats ............................................ 5 07_ Schedule Analysis for Dummies ............................................................................................................ 5 08_ A Deep look into Optimistic Cognitive Bias based on a NASA’s STEM 4th Grade Activity .................... 6 10_ Do Firm-Fixed Price Contracts Curb Cost Growth? ............................................................................... 6 11_ Early Project Formulation and Development Success: Does NASA Sink Projects Even Before They Start? ........................................................................................................................................................... 7 14_ Integration of NASA Cost Tools to Estimate Mission Concept Costs .................................................... 8 15_TruePlanning Space Missions Catalog ................................................................................................... 8 16_ Schedule Confidence and Acceleration using Deltek Acumen Fuse, Risk, and 360 .............................. 9 17_ Imaging X-ray Polarimetry Explorer (IXPE) Cost Trace .......................................................................... 9 18_ Integration of Model-Based Systems Engineering and Programmatic Analysis Tools ........................ 10 19_ NNSA’s Early-Stage Cost Estimating Tools for Strategic Planning ....................................................... 10 20_ Alternative Risk Measures for Determining Program Contingency .................................................... 11 21_ Discrete Event Simulation as a Tool for Cost Estimating .................................................................... 12 22_ Programming Analysis & Evaluation (PA&E) Analysis of Alternatives (AoA) Methodology ................ 12 23_ NASA’s EVMS Surveillance .................................................................................................................. 12 25_ The NASA SMEX Myth......................................................................................................................... 13 26_ Class D Missions PM/SE/MA ............................................................................................................... 13 27_ NASA PCEC Updates for 2022 ............................................................................................................. 14 28_ PP&C Improvement at MSFC .............................................................................................................. 14 30_ Post Launch Testing Sequence of Events. A day-to-day project schedule .......................................... 15
31_ CADRe 2022 ........................................................................................................................................ 15 32_ Math is Hard ....................................................................................................................................... 15 33_ Using Automation to Lighten your Load ............................................................................................. 15 34_ Aerospace Viewer of NASA Project Staffing Data (aView): A Practical Tool for Analyzing Staffing Levels and Cost Across Missions ................................................................................................................ 16 35_ Integrating Architecture, Programmatic, and Affordability Viewpoints: The Programmatic Cost Tool (PCT) .......................................................................................................................................................... 17 36_NICM Version 10 .................................................................................................................................. 18 37_ Mission Operations Cost Estimation Tool (MOCET) Research and Status Update ............................. 18 38_ Electrified Aircraft Propulsion Economics........................................................................................... 18 39_ Beyond the Box: Utilization of Underlying JACS Simulation Results for Advanced SRA and JCL analysis ...................................................................................................................................................... 19 40_ Human Spaceflight Schedule Study & Database ................................................................................. 20 41_ ARES Schedule Integration Tool - Integrating Artemis Schedules ..................................................... 21 42_ Applying Schedule Uncertainty in JCLs – Problems and Solutions...................................................... 22 43_ ASCoT/ONSET ..................................................................................................................................... 22 44_ Bayesian Rules of Thumb .................................................................................................................... 23 45_ ONCE Database and Data Protection .................................................................................................. 24 46_ Aerospace CubeSat Cost Estimation Tool (ACCET) ............................................................................. 24 47_ Outstanding in Your MS Project Fields ............................................................................................... 25 49_ The Integrated Baseline Review (IBR) – Why Are They Needed ......................................................... 25 50_ Applying Data to Improve Schedule Analysis...................................................................................... 26 51_ Earned Value Management and Schedule Management ................................................................... 26
imputation and the best situations for its
01_ Dealing with Missing
application.
Data – The Art and Science of
Imputation 03_ NASA Programmatic
Authors: Kimberly Roye, Dustin Hilton, Christian Performance
Smart Authors: Kevin Gilligan
Presenters: Christian Smart Presenters: Kevin Gilligan
Abstract: Missing data is a common Abstract: NASA is better than ever before in
phenomenon most analysts have experienced. cost and schedule estimation, yet challenges
Even when a dataset includes a significant remain in holding costs and schedules to our
number of data points, many of the variables of external agency baseline commitments. This
interest will have missing values. The most presentation will provide an overview of cost
prevalent method for dealing with such data and schedule variance against baselines (and re-
points is to leave them out of analysis. This baselines) for the NASA major project portfolio,
method is not ideal for multiple reasons. One is summarize NASA’s continued presence on the
that unless the data are missing completely at Government Accountability Office’s High Risk
random, leaving out data points with missing List, and describe some of the efforts NASA is
values will bias the results of analysis. A second undertaking to improve its acquisition
is that it leads to smaller data sets used for management practices.
analysis. Deleting data points has been
demonstrated through numerous empirical 04_Chief Program
studies to be one of the worst methods for
dealing with missing data. Management Officer
Overview
Most of the time in defense and aerospace
applications datasets are already small. Not Authors: David Mitchell
using all the available data means that analyses
Presenters: David Mitchell
are based on even smaller datasets, which
reduces the power of the analysis and makes Abstract: NASA created the Chief Program
them more prone to overfitting. In this paper, Management Officer in January 2022. This role
we discuss the use of imputation to overcome is responsible for strengthening the agency’s
the issue of missing data. Imputation is a oversight, management, and implementation of
proven statistical technique to fill in missing program management policies, processes, and
data points, allowing analysts to use all the best practices. This presentation will provide a
available data. We discuss two current best brief overview of the role.
practice techniques – Expectation Maximization
(EM) and Multiple Imputation via Chained
Equations (MICE) – and discuss the pros and
cons of each. We discuss the limitations of05_Earned Value released, the COMPACT team has normalized
17 new missions to be added to the model in
Management, Year in Review COMPACT V2. COMPACT V2 also features
Authors: Jon Fleming changes to the KNN tool algorithm including the
introduction of Principal Component Analysis
Presenters: Jon Fleming
(PCA) to the model development process and
Abstract: This presentation will highlight some changes to the input parameters which have
of the great progressions and accomplishments made the analogy results more intuitive and
made in the past year within the NASA Earned have improved model performance. This
Value Management community. This includes presentation will describe the current
NPR 7120.5F EVM Language Changes, COMPACT KNN dataset, improvements made to
Integrated Baseline Reviews (IBRs), NASA-led the model in COMPACT V2, an assessment of
EVMS Compliance Review Update, other Major current model performance, and a forward look
Accomplishments, as well as updates to the at COMPACT's planned future enhancements.
NASA EVM Website.
07_ Schedule Analysis for
06_ COMPACT KNN V2: Dummies
Analogy-Based Cost Authors: Glenn Butts, John Dotson
Estimation Model for Presenters: Glenn Butts
CubeSats Abstract: NASA has numerous skilled schedule
Authors: Melissa Hooke analysts at the agency, and extremely powerful
analysis tools like Acumen FUSE. However,
Presenters: Melissa Hooke
unfortunately management routinely doesn’t
Abstract: The CubeSat Or Microsat Probabilistic understand what these analysts are telling
and Analogies Cost Tool, or COMPACT, is a them. Without understanding what a health
NASA Headquarters funded effort to fill the gap check is, what BEI or HMI is and why they
in cost estimating capabilities for CubeSats, as should care are key sources project schedule
well as other microsat spacecraft. The failures.
COMPACT team has focused mainly on
“I see all this data and I see all these reports, and I
CubeSats to date, and has collected technical,
hear you, but I don’t understand our status.” Mark S.
programmatic and cost data on dozens of flown Geyer, the Orion project manager
CubeSats missions led by NASA, research labs,
and universities. In late 2019, the team released Management just wants to know if the project
the first tool prototype which uses a non- is on track or not. It is helpful if you can show
parametric regression technique, k-Nearest them some intuitive analysis that makes sense,
Neighbors (KNN), on actual data from historical it helps more if it is easy to do. We will show
CubeSat missions to produce early ballpark some quick checks that can be done by non-
analogy-based cost estimates for new CubeSat schedule experts that is helpful to analysts and
concepts. Since the KNN prototype was first management alike.08_ A Deep look into expenses of the project which includes the
planned original cost and any additional labor
Optimistic Cognitive Bias or material costs incurred to complete the
based on a NASA’s STEM 4th work. The CP contract mechanism is ideally
implemented when NASA’s requirements are
Grade Activity not well-defined, and the likelihood of a
Authors: Steve Sterk modification to the scope of the project is high.
When FFP contracts are used, NASA agrees to
Presenters: Steve Sterk
cover a non-variable cost and will not pay for
Abstract: This paper is a direct result based on any additional labor or material costs incurred
independent research being conducted by Steve to complete the work. The FFP contract
A. Sterk who works in the Program Planning and mechanism is ideally implemented when
Control (PPC) Branch at NASA Armstrong Flight NASA’s requirements are well-defined, costs
Research Center. It is a believed optimistic can be predicted, and the contractor has
cognitive bias, is the number one problem while experience in manufacturing a product that
developing cost and schedule estimates and fulfills the requirements. This study investigates
thus digging into human behavioral, a heuristic historical cost growth of spacecraft for a variety
technique in completing the task at hand. This of NASA science missions launched over the last
presentation will include 1.) How people think, 20 years, by comparing historical cost growth of
2.) the start of an Optimistic Cognitive Bias data CP and FFP spacecraft from contract start to
base, will lightly discuss; machine learning, delivery. Additionally, this study will also
iBOTs, Power-Bi, Fit Bit Apps, artificial examine the potential causes of cost growth on
intelligence for future cost and schedule FFP contracts, including schedule delays,
estimates 3.) a Roadmap which will lead the requirement changes, the addition of new
Audience, how to prevent “why” Projects and scope after the contract was signed, and
Programs overrun their baseline cost and mistakenly formulating a basis of estimate by
schedule estimates, thus thinking like a assuming high heritage to a previous spacecraft.
neuroscientist. The results of this study will provide guidelines
and lessons learned to help NASA and other
10_ Do Firm-Fixed Price government agencies determine when an FFP
contract can be used effectively and efficiently.
Contracts Curb Cost Growth?
Authors: Leah Sobel, Elliot Tibor
Presenters: Leah Sobel, Elliot Tibor
Abstract: There are many types of contracts
used by NASA for science missions to procure
spacecraft, this study focuses on two main
categories: Cost-Plus (CP) contracts and Firm-
Fixed Price (FFP) contracts. When CP contracts
are used, NASA agrees to cover the actual11_ Early Project early formulation, it tracks costs from the
earliest proposal that gets incorporated into a
Formulation and budget request, rather than from the ABC’s.
Development Success: Does
Note to selection committee: The primary
NASA Sink Projects Even analysis is “Early Project Formulation and
Before They Start? Development Success: Does NASA Sink Projects
Even Before They Start?”, from SID Insights
Authors: Ron Ray
Volume 10. Other Insights summarized include
Presenters: Ron Ray “Contribution of Individual Project Element Cost
Increases to Total Project Cost Increases”, also
Abstract: Multiple NASA efforts have and from Volume 10, Vendor Performance from
continue to investigate why some projects are upcoming Volume 11, and “Lack of
development cost successes and others Deterministic Factors Influencing Project
overrun, but the results have been less than Development Performance” from Volume 3,
satisfying. SID studies have examined how and I expect to highlight some other individual
various factors are associated with factor analyses from other analysts.
development success, both in isolation and
aggregate. The consensus result is that while
some factors or combinations of factors show
minor correlations, there is a tremendous range
of variance that has yet to be explained.
This presentation briefly summarizes “Factor”
or “Characteristic” studies, then focuses on how
the very early formulation of a project as it is
proposed for incorporation into NASA budget
requests as a new start may affect eventual
project cost performance. The time period of
interest predates Agency Baseline
Commitments (ABCs) and Management
Agreements for schedule and cost, and often
predates formal Phase A effort on a possible
project. This analysis looks at whether early
formulation decisions, leading up to NASA
budget submissions, build a budgetary “box”
around a proposed project that the project
cannot escape from. Most SID studies on cost
performance use ABC’s as the starting point for
cost consistent with external reporting practices
to Congress and the Government Accountability
Office. Since this study looks at the effects ofthere is room for improvement. This paper will
14_ Integration of NASA Cost
discuss two of the latest and best methods for
Tools to Estimate Mission obtaining accurate cost estimates using best-of-
Concept Costs breed model-based cost engineering
techniques.
Authors: Natalie J. Weckesser
This paper / presentation will address two
Presenters: Natalie J. Weckesser
relatively new methods to improve the accuracy
Abstract: Compass, a concurrent engineering of space missions cost estimates: TruePlanning
team at NASA Glenn Research Center, designed Hardware Equipment Types and a relatively
a mission in which an orbiter and several new Space Missions Catalog, with emphasis on
landers use a trajectory to Venus and carry out the later. Both methods include a variety (up to
a number of years of primary science 119) space specific equipment types, and the
operations. Compass desired mission life-cycle Space Missions catalog also includes novel
cost estimates as part of the study’s findings. specific models for electric propulsion, ion
Several unique hardware elements needed to thrusters, lasers, parachutes, radar altimeters,
be estimated, in addition to element-level and thermal protection. This paper /
wraps, mission-level wraps, software costs, and presentation will include two case studies (one
Phase E mission operations and science costs. earth orbiting and one planetary mission)
To build a comprehensive estimate, many featuring many of the above equipment types
estimating tools were required, and their and unique cost models. A validation study of
outputs integrated into a cohesive model. This the results of these case studies will also be
paper will detail the estimating process, included.
including the inputs solicited from the design
team, the NASA tools used and their
application, the approach for creating a
cohesive model, and the greatest barriers to
doing so.
15_TruePlanning Space
Missions Catalog
Authors: Christopher Price, Mark Jacobs,
Shawn Hayes
Presenters: Christopher Price
Abstract: NASA continually strives to improve
cost estimation for the highly advanced
technology flown on planetary as well as earth
orbiting space missions. Over the years it has
been proven that parametric cost models are a
desired way to obtain accurate estimates. Still4) How 360 outputs and generated
16_ Schedule Confidence and
scenarios are examined to review
Acceleration using Deltek driving activities
Acumen Fuse, Risk, and 360 5) discuss how the outputs of the 360
Authors: David Rose, Philip Ashtianie analysis assisted in gaining stakeholder
alignment to re-examine task durations,
Presenters: David Rose, Philip Ashtianie
look for areas of efficiency and synergy,
Abstract: Stakeholders often challenge project and pull in the schedule to help meet
teams to verify the quality of their schedules, program goals
brief the schedule confidence level, and
accelerate the schedule. Project teams need 17_ Imaging X-ray
robust tools and techniques that support the Polarimetry Explorer (IXPE)
responses to the stakeholder requests and help
the project managers examine and identify Cost Trace
schedule areas that may have opportunities for Authors: Billy Carson
acceleration. Recently, Cobec Consulting
Presenters: Billy Carson
acquired Deltek Acumen licenses after
witnessing the tangible benefits NASA realized Abstract: The MSFC Engineering Cost Office
using Fuse by participating in the regular NASA performed cost estimates and analyses for the
SCoPe meetings. This paper describes how Imaging X-ray Polarimetry Explorer (IXPE)
Cobec Consulting is leveraging Deltek Acumen, proposal and Concept Study Report (CSR). The
Fuse, Risk, and 360 to present high quality, high ECO also provided support to IXPE for various
confidence schedules to our FAA stakeholders milestone reviews. Now that IXPE is
and leverage 360 to explore opportunities for successfully in orbit, the ECO wanted to trace
project plan acceleration. our costs estimates through the project lifecycle
The authors of this paper will illustrate: and compare those estimates to the projects
costs via information found in the CADRes and
1) How an FAA Joint Resource Council obtain from the project office.
Final Investment Decision schedule is
prepared for analysis using Deltek Information discussed in the paper:
Acumen. • Brief history of IXPE design changes
over time,
2) How the scheduling team uses Fuse
• Comparison of estimates over time,
to prepare the schedules for handoff to
• Project costs over time as recorded in
the FAA Investment Planning and
CADRe,
Analysis Team for a schedule quality
• Discussion of the delta cost from early
assessment.
estimate to costs at launch.
3) How 360 is used to identify and
process key “challenge” milestones
through 36018_ Integration of Model- 19_ NNSA’s Early-Stage Cost
Based Systems Engineering Estimating Tools for Strategic
and Programmatic Analysis Planning
Tools Authors: Charles Loelius, William Todd
Authors: Louis Fussell Presenters: Charles Loelius, William Todd
Presenters: Louis Fussell Abstract: The Office of Program Analysis and
Evaluation (PA&E) within the National Nuclear
Abstract: Model-based systems engineering
Security Administration's (NNSA) is charged
(MBSE) is an alternative to the traditional
with leading programmatic cost estimating and
document-based systems engineering
associated analytical support for nuclear
approach. NASA is a leading proponent of
weapon-related activities and capital
MBSE, and the approach was applied to several
acquisition projects throughout the federal
NASA projects. In order to eliminate errors and
budgeting process. PA&E has developed a suite
inefficiencies from replicating design
of models which support early-stage cost
information, the MBSE community is developing
estimating and planning within the NNSA. The
interfaces between MBSE tools and
estimates generated with this methodology are
configuration management tools, computer
updated and published annually in NNSA’s 25-
aided design tools, spreadsheets, and other
year strategic planning document called the
discipline-specific analysis tools. This holds true
Stockpile Stewardship and Management Plan
for programmatic analysis tools as well. The
(SSMP). In particular, PA&E publishes cost
paper provides an overview of the application
estimates for upcoming major modernizations
of MBSE at NASA and efforts to integrate MBSE
of nuclear weapon systems and new major
tools with programmatic analysis tools. A
infrastructure acquisition which account for a
technical demonstration of this utility is
significant portion of the NNSA’s budget. PA&E
provided which integrates Vitech’s GENESYS
uses two suites of tools to estimate these
MBSE tool with NASA’s tools for parametric cost
projects, the Major Modernization Model and
and schedule estimation and joint confidence
the Cost, Schedule, and Phasing Estimating
level analysis. The intent is to bring awareness
Relationships for Construction (CSPER-C) model.
of this emerging capability to NASA’s
programmatic community. NNSA’s PA&E develops time-phased planning
estimates for modernization of the United
States’ nuclear stockpile systems over the next
twenty-five years using the Major
Modernization Model. Over the past decade,
the United States has begun planning and
executing the refurbishment of existing systems
and the development of new weapon systems.
PA&E estimates both the cost of research,
development testing and evaluation (RDT&E)for the systems and their production. execution of NNSA projects and allows for the
Development costs are estimated using development of execution and budget profiles.
complexity factors derived from comparisons Like the Major Modernization model, the key to
with prior modernizations, along with a these relationships is that the inputs are high-
modified Rayleigh distribution informed by level and so can estimate projects even at a
historic cost actuals. Production costs are based very low level of definition.
on anticipated production schedules and
quantities, along with a variation of the 20_ Alternative Risk
Crawford model for learning curves. The Major
Measures for Determining
Modernization Model produces s-curves
developed by using Monte-Carlo methods to Program Contingency
account for the distribution of complexity Authors: Louis Fussell
factors. Critically, these early-stage estimates
require only high-level inputs that can be easily Presenters: Louis Fussell
tuned to perform what-if analyses.
Abstract: In 2005, NASA began requiring
NNSA’s PA&E uses the CSPER-C model to projects to statistically sum the cost of project
produce defensible early-stage cost estimates components and the duration of their schedules
for infrastructure across its eight laboratories, to determine confidence levels for total project
plants, and sites to ensure that adequate cost and duration jointly. This sum is typically
resources are available to complete its accomplished by means of a Monte Carlo
recapitalization plans. The nuclear security simulation executed with a cost-loaded
enterprise has inherited its infrastructure from schedule model augmented with probabilistic
the Manhattan project and the cold war, so that distributions assigned to costs and durations
many of its facilities need replacement. PA&E based on project risks and estimate
uses a parametric cost estimating relationship uncertainties. NASA policy requires that project
developed from historic NNSA data to estimate managers reserve budget equal to a 50% joint
the costs of those replacement facilities. The confidence level and that the managing
CSPER-C model uses the gross square footage, directorate hold reserve to a 70% confidence
facility hazard classification, and complexity of level, with some exception. The 50% and 70%
programmatic equipment as parameters. joint confidence levels are quartile risk
CSPER-C accounts for technical uncertainty in measures referred to as Value-at-Risk (VaR) in
the range of project inputs and cost uncertainty risk management literature. This paper will
derived from historic cost actuals. Like the discuss the drawbacks of VaR risk measures and
Major Modernization Model, CSPER-C uses propose the use of alternative risk measures,
Monte Carlo simulation to produce an “s-curve” namely the mean and expected shortfall, for
distribution for each project. The schedule determining project reserve levels. Monte
estimating relationship is derived from historic Carlo simulations for several NASA projects
schedule durations and is driven by the cost of were executed and a comparison of 50% and
the project and whether or not the project 70% VaR, mean, and expected shortfall risk
involves nuclear hazards. The phasing measures is presented.
estimating relationship is derived from historic21_ Discrete Event solutions to meet mission need and future
requirements. The goal of an AoA is to better
Simulation as a Tool for Cost define and understand the solution space to
Estimating address mission gaps and allow for an
understanding of the cost, schedule, risk, and
Authors: Zachary Matheson, Thomas Cook,
effectiveness trade-offs between alternatives
Gabe Sandler, Charles Loelius, William Todd
with the goal of allowing leadership to make a
Presenters: Julie Anderson, Zach Matheson data-driven, unbiased, and defensible down-
select decision. In many government
Abstract: The Office of Program Analysis and organizations, AoAs are conducted by various
Evaluation (PA&E) within the National Nuclear support offices with specialties in cost
Security Administration (NNSA) is charged with estimating, schedule estimating, and systems
leading programmatic cost estimating and engineering/requirements development. This
providing associated analytical support often leads to stove-piped execution of cost,
throughout the federal budgeting process. schedule, risk, and effectiveness analysis which
Several NNSA capabilities require the can lead to differing baseline assumptions that
construction of production facilities to may drive inconsistencies in analysis results.
manufacture components that are not
commercially available. To prepare early-stage The Department of Energy’s National Nuclear
cost estimates for budget planning, PA&E Security Administration (DOE NNSA) conducts
developed discrete event simulation models of AoAs for their major capital acquisition projects
production processes to estimate equipment (>$20M) and has consolidated the functions of
required to manufacture components at leading and conducting analysis for all AoAs for
specified production rates. This analysis has major capital acquisition projects under the
been used to estimate equipment, facility size, Office of Management And Budget’s
and staffing requirements for a given Programming, Analysis, and Evaluation (PA&E)
production capability, informing a defensible group. The consolidation of responsibilities for
cost estimate. AoAs under the analytics group within the NNSA
has allowed for more integrated and
22_ Programming Analysis & informative analysis which has improved both
the resultant analysis and the final alternative
Evaluation (PA&E) Analysis down selections for major capital projects. This
of Alternatives (AoA) presentation is focused on highlighting the
Methodology management processes, analytical approaches,
and lessons learned experienced by PA&E in
Authors: Christopher Massey, Charles Loelius
order to help improve AoA policy, approaches,
Presenters: Cash Fitzpatrick and Christine Suhr and results for other organizations.
Abstract: Analysis of Alternatives (AoA) are 23_ NASA’s EVMS
important and complex studies that are
conducted throughout the federal government
Surveillance
in order to evaluate material and non-material Authors: Briannah Smith, Kristen KehrerPresenters: Briannah Smith, Kristen Kehrer nearly impossible for awarded projects to
deliver to their proposed cost and schedule,
Abstract: An EVM system is the management
especially with increased requirements once
control system that integrates a program’s work
awarded. The myth of a SMEX program recalls
scope, schedule, and cost parameters for the old days of the failed philosophy “faster,
optimum program planning and control. To better, cheaper.” Our research will examine the
make sure the EVM system is working properly
science, cost, and schedule performance of
and producing data that is valid, accurate and NASA’s series of SMEX missions.
timely, NASA implemented routine
surveillance.
26_ Class D Missions
Begun as a product of a GAO recommendation, PM/SE/MA
NASA funded EVMS surveillance in 2019 and
Authors: TBD
hasn’t looked back since. Leveraging the
surveillance process and practices of the Presenters: Ben Clare, Sally Whitley
Defense Contract Management Agency (DCMA),
NASA began by deploying surveillance at Abstract: NASA Class D missions are becoming
suppliers, APL, JPL and SwRI, and then increasingly prevalent, with the promises of
expanded in-house at GSFC, JSC, KSC and lower mission costs being a big motivator. But
MSFC. As of FY21Q4, 689 individual tests have how are these missions so low cost? The answer
been run on 19 projects. lies in the amount of risk these missions are
willing to take on. NASA’s Class D tailoring and
This presentation will describe how the EVMS streamlining memorandum is held up as the
surveillance approach was developed and guideline for how to modify program
implemented. It will describe the various management, systems engineering and mission
metrics utilized to test system health, the data assurance (PM/SE/MA) requirements to meet
collected to enable analysis, and the various Class D standards. It is a common cost
tools used to support surveillance. The estimating heuristic to use a wrap factor on
presentation will also discuss how projects are hardware costs to estimate PM/SE/MA costs as
selected for surveillance and the various it’s understood that these costs will then
surveillance outcomes. High level surveillance fluctuate with the size of the mission. However,
results and trends will be shared that as well as with Class D missions, this method needs to be
some of the successes achieved. reevaluated since the wrap factors typically
used include many functions that are not
25_ The NASA SMEX Myth applicable to Class D missions, such as EVM. In
Authors: Kathy Kha, Salley Whitley this presentation, we will go through
reexamining Class D mission data available in
Presenters: Kathy Kha CADRe and developing an appropriate
PM/SE/MA wrap factor.
Abstract: With a cost cap not to exceed
~$150M, NASA’s SMEX program aims to deliver
frequent world-class science opportunities at a
low cost. While this is theoretically enticing, it is27_ NASA PCEC Updates for Based on feedback from PCEC users and more
experience using PCEC v2.3 for robotic science
2022 missions, two areas for improvement are under
investigation. These areas include improving
Authors: Brian Alford, Shawn Hayes, Mark
estimating performance for flagship missions
Jacobs, Richard Webb
and for System Integration & Test. Multiple
options are being assessed to improve
Presenters: Brian Alford, Shawn Hayes, Mark
estimates for NASA’s flagship missions. These
Jacobs options include derivation of tuning factors that
could be used with the PCEC v2.3 CERs or the
Abstract: The release of the Project Cost
development of new CERs tailored for flagship
Estimating Capability (PCEC) v2.3 last year
missions. Higher-than-expected System
brought about the inclusion of data from
Integration & Test estimates have been noted
several recently launched missions and a
by multiple model users. Data used to support
completely new set of CERs, particularly for
this CER is under further assessment and
estimating robotic science missions. This year’s
potential CER replacement candidates are in
presentation will primarily focus on two topics
development. We will share results from
currently under investigation by the PCEC team:
preliminary analyses of alternative approaches
challenges normalizing the mission data to
for flagship missions and System Integration &
isolate COVID-19 impacts and additional
Test.
enhancements to the Robotic Spacecraft CER
development process.
In addition to these two topics, our
presentation will also provide a summary status
Impacts from COVID-19 have been experienced
update on the tool, updated capabilities that
by all NASA robotic science missions in
are under development for the next major
development between March 2020 and March
release, and the status of the development of a
2022. The level of impact can vary depending
formal PCEC training course.
on where each project was in its development
cycle, use of contractors/subcontractors,
international contributions, launch date 28_ PP&C Improvement at
flexibility, and many organization-specific MSFC
constraints experienced during this 2-year time-
period. Although data from 25 projects shows Authors: Andy Prince
significant variability, the impact appears to be
greatest for missions scheduled to launch in Presenters: Andy Prince
2023. This data can be used to identify any
correlations between COVID-related cost Abstract: MSFC has established a dedicated
growth and various project characteristics. To PP&C organization within the Office of Strategic
better understand how to treat these cost Analysis and Communications (OSAC). This new
increases for future mission cost modeling has organization brings together cost, schedule,
multiple challenges to address. These include EVM, and PP&C generalists for the purpose of
inconsistencies in project tracking of COVID advancing PP&C at MSFC. The new PP&C Office
impacts, data availability limitations, and other
combines legacy functions, such as training,
project-specific limitations that severely
affected performance. We will present high- schedule analysis, and cost estimating, with the
level data and correlation analyses and a plan mandate to lead PP&C stewardship. As the
for accounting for COVID impacts for PCEC PP&C stewards we are focused on creating a
robotic science mission cost data normalization. greater awareness of the importance of PP&Cto program and project success, as well as 31_ CADRe 2022
developing future PP&C leaders who are well-
Authors: Eric Plumer
rounded, knowledgeable, and experienced.
Presenters: Eric Plumer
The presentation will focus on the changes in
organization and on the initiatives underway to Abstract: Overview of current CADRe
improve PP&C at MSFC. initiatives.
30_ Post Launch Testing 32_ Math is Hard
Sequence of Events. A day-to- Authors: Rachel Sholder, Sally Whitley
day project schedule Presenters: Rachel Sholder
Authors: Denis Pinha, Rodolfo Lavaque
Abstract: The NASA cost community relies on
Presenters: Denis Pinha, Rodolfo Lavaque risk analyses to estimate confidence in a
project’s budget. At PDR, convention requires
Abstract: Missions at NASA follow best practice
that baseline cost/schedule confidence should
of project management and systems
be around the 50th percentile and cost plus
engineering.
reserves should be around the 70th percentile of
A key component of project management at the joint distribution of total cost and schedule.
NASA deals with scheduling of tasks which must But how can we test whether our approach to
fulfill conceptual, technical and mission determining 50th and 70th percentiles for
requirements. missions going into PDR is reliable? According to
cost actuals from historical NASA missions,
The challenge of scheduling tasks can vary there is an 88% chance that a mission will
during the mission-life cycle depending upon overrun its planned budget. There is a 50%
the level of detailed required, time frame, and probability that a mission will outspend its
how often re-planning is needed. Long term budget by 16% or more. At the empirical 70th
scheduling assumes a relative more stable percentile, NASA missions are spending their
system which priorities, and other constraints full budgets plus 30%. Our research will
do not change often during the scheduling examine the NASA cost community’s approach
period. However, scheduling tasks that focuses to reserve postures. Using the empirical dataset
on short term, that is, scheduling on an hourly, as our guide, how can projects approaching PDR
daily, or weekly basis assumes much less stable provide cost and schedule analysis that
system. This work deals with short term supports the goal of achieving 70% confidence
scheduling that requires a more often re- in the budget at the portfolio level?
planning activities where priorities, and other
constraints change over time. 33_ Using Automation to
A case study and lessons learned on how to Lighten your Load
design and plan the Post-Lauch Testing Authors: Erin Wood & Danelle Fogle
Sequence of Events (PLT SOE) for GOES-R series
satellite are described to illustrate this problem. Presenters: Erin WoodAbstract: Over the course of the past 2 years, points before they become your future
various methods have been implemented at headache.
Glenn Research Center in efforts to identify and
In addition to learning the technique, also learn
automate touch points within our projects &
programs, within other centers and other quantitative information on implementation.
departments within our center, and with our How much time does it take to set up in a real-
world environment, what buy-in challenges
outside contractor schedules. Touch points can
be identified as the hand-offs, or external have we experienced thus far, and how much
time will it save on a regular basis. Is the small
predecessor/successor relationships between
projects necessary to get the job done. amount of headache and set up work worth it?
Successful coordination within high priority We’ll give you all the information to decide for
yourself.
projects in which lab and resource availability is
competitive is often vital.
34_ Aerospace Viewer of
Benefits of this technique include:
NASA Project Staffing Data
*High customer service by providing both the (aView): A Practical Tool for
information and the impact to the project and
program management instantly
Analyzing Staffing Levels and
Cost Across Missions
*a substantial reduction in manual data mining
for summary graphics and integrated reporting Authors: Sarah Lang, Justin McNeill, Jr.,
Tommy Tran, Alexander Zarate Garcia, C. Jason
*the ability to isolate metrics within Acumen Zhang
Fuse to more easily identify ‘trouble spots’
Presenters: Sarah Lang
*Greater ability to provide proactive data-
driven decision-making information to stay Abstract: The Aerospace Viewer of NASA
ahead of the fire instead of spending time Project Staffing Data (aView) team will present
putting it out a summary of updates made to aView since
2020. Built upon the FTE Tool first released in
*Reducing required communication between 2011, aView is a repository of programmatic
departments. Automated integration means data used for comparative analysis of staffing
independent schedules can ‘talk’ to each other profiles of NASA science missions. It provides
without being disruptive to the daily flow, high-level views of the historical data of NASA
instead of that one more email, phone call, and planetary missions for development Phases C
meeting to coordinate the exchange of and D as well as the operations Phase E of Full-
information time Equivalents and Work Year Equivalents
(FTE/WYE). It can be beneficial using aView
It cannot work without the right programming when reviewing and evaluating the basis of
and human touch though. This presentation estimate for mission phases to understand how
will explain exactly how-to, mistakes that we’ve
the labor basis of estimate compares with past
made so far, other strategies that didn’t work as NASA missions. aView was developed by The
well and why, and how to identify these touch- Aerospace Corporation for the benefit of theNASA Planetary Missions Program Office. It is a downstream to make certain GUIs and
web browser-based application and will be outputs more readable.
available via the One NASA Cost Engineering
(ONCE) model portal. • A table of Flight Types, where each
flight type is simply a combination of
systems which would be part of a single
35_ Integrating Architecture,
launch manifest.
Programmatic, and
• A table of Flights, which defines the
Affordability Viewpoints: date and number of launches for all
The Programmatic Cost Tool Flight Types.
(PCT) PCT OUTPUTS:
Authors: Joe Mrozinski, Robert Shishko
Armed with these three tables, as well as with a
Presenters: Joe Mrozinski user-supplied budget profile, PCT then provides
the following outputs:
Abstract: The Programmatic Cost Tool (PCT) is
a software package which provides a framework • Summary tables of the total dollars
to produce affordability assessments and needed per system for Development,
perform trades for complex spaceflight Production and Operations, as well as
architectures, such as multi-system, multi- the total of all three, on an annual
decade Human Moon-to-Mars Program basis.
architectures. This talk will build upon our 2018
NASA Cost and Schedule Symposium slide • A timeline showing each required
presentation by providing a live demonstration systems Development, Production and
of the tool, as well as many of the new features Operations schedules durations
added in the past 4 years. required to enable the architecture.
PCT INPUTS: • A “sand-chart” showing the total cost of
the architecture by year as compared to
While the architectures analyzed using PCT are the yearly budget. The different sand
grandiose in scale, usually billions of dollars layers can represent individual systems,
spread over decades, the inputs PCT requires or those systems can be grouped in
are fairly straightforward. The user need only user-defined ways, such as by NASA
provide information for three tables: center.
• A table of Systems, such as launch • A “dance card” showing the year-to-
vehicles and flight elements, and the year flights and their corresponding
corresponding programmatic details for systems. This allows the user to quickly
each, such as development, production see the “bigger picture” of the
and operations costs and schedules. architectures, allowing for easy
Thumbnails representing each system, verification that the architecture
if provided by the user, are fed represented is as expected. The displayalso allows the user to come up with Abstract: The Mission Operations Cost
trades should the current architecture Estimation Tool (MOCET) team will present an
iteration be in violation of the budget overview of recent research topics as well as
profile. the latest mission updates. Research topics this
year are focused on continued study &
PCT resides in an easy to use Excel file, making refinement of level 2 Work Breakdown
it intuitive to interact with. PCT has been
Structure (WBS) modeling, and modeling for
developed with support from both the Jet extended missions cost. An overview of the
Propulsion Laboratory and the Johnson Space
state of the user community will be presented
Center. including statistics from the One NASA Cost
Engineering (ONCE) model portal and
36_NICM Version 10 software.nasa.gov. MOCET is a model
Authors: Joe Mrozinski, Luther Beegle , Kyle developed by the Aerospace Corporation in
Brown , Robert Cesarone, Michael DiNicola, partnership with NASA’s Science Office for
Michael Fong, Melissa Hooke, Alfred Nash, Mission Assessments (SOMA), which provides
Sherry Stukes the capability to generate cost estimates for the
operational, or Phase E, portion of NASA
Presenters: Joe Mrozinski science missions. MOCET is comprised of CERs
Abstract: The NASA Instrument Cost Model that have been derived from historical data for
Team announces NICM Version 10 will be Planetary, Earth Science, and Explorer missions.
released in the Fall of 2022. Several of the The resulting CERs and accompanying
upcoming upgrades will be discussed and/or documentation have been implemented as a
demonstrated, including: 1) A new repeatable, standalone Excel based tool which is now
analytic solution in both the System and available via the One NASA Cost Engineering
Subsystem Tools, 2) Isoquants introduced to the (ONCE) model portal and software.nasa.gov.
JCL plots, 3) Bayesian imputation improved with
boundary conditions, 4) K a new Nearest 38_ Electrified Aircraft
Neighbors weighted average tool added to the Propulsion Economics
NICM Search Engine Outputs, 5) Expanded
Authors: Peter Frederic
Search Engine capabilities and summaries, 6)
and more! Presenters: Peter Frederic
37_ Mission Operations Cost Abstract: Tecolote Research recently
participated in a design exploration study
Estimation Tool (MOCET) conducted by NASA’s Advanced Air Transport
Research and Status Update Technology (AATT) project focused on regional
transport aircraft concepts employing
Authors: Marc Hayhurst, Brian Wood, Cindy
electrified aircraft propulsion (EAP). NASA’s
Daniels, Lissa Jordin,Washito Sasamoto, Waldo
interest in EAP aircraft concepts stems from the
Rodriguez
potential to reduce fuel consumption,
Presenters: Marc Hayhurst emissions, noise, and operating costs.We selected direct operating cost plus interest EAP aircraft. Several technology-enhanced
(DOC+I) as the primary economic figure of merit cases were considered for each configuration,
used to assess the EAP concepts. DOC+I relates trading battery size, battery specific energy,
directly to profitability for aircraft operators electric motor size, and hybrid power split. We
(given market-driven ticket prices). DOC+I also performed sensitivity cases to explore the
includes the following elements: aircraft impact of changes in jet fuel cost and ground-
ownership, energy, maintenance, flight crew, supplied electrical energy cost.
and flight equipment financing (the “+I” in
This study showed that overcoming the
DOC+I).
additional cost and weight associated with the
DOC+I analysis was based on a tool called the EAP systems will be very challenging if EAP
Probabilistic Technology Investment Ranking aircraft are to be economically competitive with
System (PTIRS). PTIRS is a business case model conventional aircraft. This paper will highlight
for evaluating emerging technologies in the the challenges identified.
context of commercial aircraft development,
manufacturing, and operations. 39_ Beyond the Box:
This was a comparative cost study. The Utilization of Underlying
economic impacts of EAP were assessed by JACS Simulation Results for
comparing estimated DOC+I for enhanced
aircraft employing EAP versus estimated DOC+I
Advanced SRA and JCL
for corresponding baseline aircraft not analysis
employing EAP. The study focused on regional Authors: Steve Wilson, Mike Stelly
class aircraft because conventional wisdom and
prior studies suggested that regional class EAP Presenters: Steve Wilson
transports may see larger benefits for a given
Abstract: Tecolote Research maintains its
battery technology level than single-aisle or
widely utilized JACS add-on to Microsoft Project
larger aircraft. We modelled five specific
that facilitates a suite of analyses involving
regional aircraft configurations: 18 Passenger
stochastic cost, schedule, and risk simulation,
Turboprop, 48 Passenger Turboprop, 70
such as Schedule Risk Analysis (SRA) and Joint
Passenger Turboprop, 50 Passenger Turbofan,
Confidence Level (JCL) analysis. A prominent
and 78 Passenger Turbofan.
feature of JACS is its native expression of
Each aircraft configuration was based on an measures and visuals associated with these
actual historical aircraft. Actual cost and analyses, but underlying them is a massive
technical data the historical was used to register of the integrated simulation results
calibrate the engineering sizing and cost models called the cache file, which includes per
used in the study. The aircraft were then iteration information for model elements.
brought up to a modern state-of-art baseline by Transforming the data in this file (using a data
assuming technology improvements from each processing tool) and injecting it into a data
aircraft’s entry-into-service to the year 2020. visualization tool enables special sophistication
The baseline aircraft were then updated with in SRA/JCL analysis not previously enjoyed by
EAP technologies and resized to produce the the NASA community.In this paper, we will discuss our exploration of assessment and forecasting capability. The
the cache file along these dimensions: success of HSF projects will depend upon, in
large part, proper appreciation of the complete
• Simulation results processing, including human space story, which, for now, remains
tool choice (e.g. R or Python), riddled with ambiguity and omissions. Though
transformations, pivots, joins, cleaning, contributions from CADRe and other ongoing
output structure options, and data collection work have been significant, HSF
computational constraints lacks a definitive source for accurate, high-
resolution, and ready-to-use schedule data that
• Task-based analysis, including critical
captures all major legacy projects and
path identification, perspectives on
precursors.
criticality measures, and driver
identification Investigating this issue, we found that some of
the quoted schedule data points and Rules-of-
• Risk-based analysis, including various
Thumb (RoT) used in programmatic analysis
methods of conditional analysis,
around the agency and domestic space
stochastic omissions, the importance of
community often lack citations or proper
the role of parallelism and
historical interpretation. In fact, some datasets
compensatory risk analysis strategies,
contradict one another or amount to viral
visualizations, and interpretation of
reproductions of previously flawed sets that
risk-based results and measures
have been circulating through back-channel
• Integrated cost and schedule analysis networks for decades.
This paper marks the beginning of a new effort In this paper, we will discuss our nascent
by our JSC team to advance the frontier of solution to this longstanding issue: The Human
integrated programmatic analysis. Future Spaceflight Schedule Database, a compendium
papers in this series will reflect updates to this of over 2300 authoritative, meticulously
project, rendering more discoveries and documented data points spanning the full scope
examples. of HSF history. We will show several cuts of
these data and share new Rules-of-Thumb at
40_ Human Spaceflight various levels:
Schedule Study & Database • Total DDTE duration: ATP thru
Authors: Steve Wilson, Ashley Varma hardware delivery & launch
Presenters: Steve Wilson • X Category: Hardware type,
programmatic era, crewed systems,
Abstract: The rise of human spaceflight (HSF) commercial development, etc.
activity attributable to the advent of the
Artemis campaign, commercial enterprises, and • X Milestone: ATP, PDR, CDR, delivery,
international partnerships demands a crisp, and launch
contextual understanding of programmatic
• X Growth: Total, over time and between
history as NASA continues to sharpen its
milestones• X Analysis: Select SRA s-curves (some and the lack of accompanying meta data,
over time) vs actual dates of milestones criteria and assumptions. We will demonstrate
they forecast the ARES Schedule Integration Tool and
Integration web site that provides program and
This paper marks the beginning of a new effort project teams access to a quick schedule
by our JSC team to compile historical technical, development, data integration, automation and
cost, schedule, and risk (TCSR) data into our HSF illustration tool set, an integrated method to
Body of Knowledge. Future papers in this series perform data and meta data capture, and an
will reflect updates to this project, rendering integrated cloud web platform for sharing data
more analyses and RoT. instantly. With this schedule integration
capability teams at all levels can configuration
41_ ARES Schedule manage and control their schedule data and the
Integration Tool - data they provide to their customers.
Integrating Artemis The ARES Schedule Integration Tool (ASIT)
Schedules provides a framework that allows individuals
and teams to quickly build and maintain visual
Authors: Mark Miller
schedule products consisting of shared task and
Presenters: Mark Miller milestone data with low operational impact.
Several key innovations were involved such as
Abstract: Major endeavors like the Artemis creating data-driven graphics, using a
Program require the coordination of many distributed team to identify and manage
centers, project organizations, programs, schedule information as they normally perform
directorates, contracts, and support groups. The their work using an easy-to-use tool to integrate
main challenge for these groups is the and share key pieces of information. The tool
integration of a lot of people, what they are ensures that data is normalized and can easily
supposed to do and when they are supposed to be made available to any interested peers,
get it done. Often this integration is just as interface organizations, management, or
complex as the hardware and software systems customers through an integrated online library.
being developed. Although a great deal of effort Using the same tool set any group or individual
goes into the development of detailed with access to the system can reference that
requirements and configuration control of the data, add logic that applies to them, and add
hardware and software end items, not so much their own data to create composite schedules.
goes into the development, configuration Custom data can then also be published.
management, control of clear integrated
schedules. In addition, schedule integration in The ASIT Web Site provides a single location to
large projects suffers from such things as poorly find toolset information, an API (Application
understood interdependencies, loose control Programming Interface) for real-time data
over implementation level schedules, lack of exchange between ASIT powered tools and
illustrated products for teams, lack of a other systems, administrative pages for teams
common share location for those products, and to perform their own schedule configuration
frequent confusion over top level milestones management tasks, and an online catalog ofYou can also read
