Automation of Scenarios and Test Cases - Franz Wotawa, TU Graz, Inst. for Software Technology, Christian-Doppler Laboratory for Quality Assurance ...

Speaker
 Live View

Automation of Scenarios and Test Cases
Franz Wotawa, TU Graz, Inst. for Software Technology, Christian-Doppler Laboratory for Quality
Assurance Methodologies for Autonomous Cyber-Physical Systems

 © 2021 carhs.training gmbh

 Funding obtained from:

  Involved projects:

  Joint work: Mihai Nica, Hermann Felbinger, Jianbo Tao from AVL and my
 colleagues at TU Graz: Florian Klück, Yihao Li, Martin Zimmermann, ...

  Objective: Making autonomous and automated systems safe!

1 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Speaker
 Live View

Motivation

 © 2021 carhs.training gmbh

The Reality for self-driving cars

 “On the one hand…“

  Automotive companies drive the development
 and testing of automated and autonomous
 vehicles.

  Semi-autonomous systems are already available
 on the market (e.g. Tesla Autopilot)

  Increasing importance of automated and
 autonomous driving functions (AD/ADAS)

 Source: https://www.sbdautomotive.com/en/germany-self-driving-car-test
 Source: https://solartribune.com/the-tesla-network/

3 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

The Reality for self-driving cars

 “But on the other hand…“

  Severe road accident with
 automated/autonomous vehicles

  Serious challenges to overcome for the
 development, verification and validation of
 autonomous systems

  How to foresee all critical testing scenarios?

4 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

„On March 18th 2018, a 49-year-old woman was struck by a self-driving Uber vehicle in Tempe, Arizona. She was transported
 to the hospital, where she died. In the aftermath, Uber’s self-driving program is hanging on by a thread, while the rest of the
 industry debates the speed in which these vehicles were being rushed to market. It is widely seen as the first person to be
 killed by an autonomous vehicle.“

5 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Tesla accident in Taiwan, 2020.

6 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Important Aspects for Testing AD/ADAS

  Demonstrate reliability and safety:
  Safer than human driving to gain public acceptance
  Statistics (U.S. Department of Transportation, National Highway Traffic Safety
 Administration):
  36,096 deaths on road per year (USA,2019)
  1.11 fatalities per 100 Mio vehicle miles (USA, 2019)
  >90% caused by humans’ driving behavior

  Ensure safety in any situation:
  Master standard situations and corner cases
  No or just limited human back-up
  Fail-operational (HW/SW redundancy)

  Manage system complexity:
  No “pure mechanical system”
  Highly intervened SW and HW components
  Sensor false-positive, sensor noise, etc.

7 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Challenges for Testing AD/ADAS

  Complete software testing of complex systems?
  Close to infinite parameter space
  “Testing is the process of executing a program with the intention of finding
 errors”

  How to assure “good-enough” testing?
  275 million miles required to demonstrate safe driving
  Physically collecting millage is infeasible
  Unlikely to cover a sufficient amount of critical situations

 → Virtual testing methodology required to:
  Consider all environmental parameter interactions that influence the AD/
 ADAS system
  Identify and reproduce critical scenarios as basis for ADAS Verification and
 Validation

8 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Differences between ADAS and ordinary vehicles

 Ordinary vehicles ADAS vehicles
 Driver as fallback No (human) fallback
 Known basic functionality Complex functionality / high
 interactivity between components
 No AI technology AI technology
 Interactions with environment High / potentially unknown / complex
 controllable interactions with the environment
 Simple to complicated environment / Complicated to complex environment /
 functionality functionality

9 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

10

 Challenges

  There are estimated 1.4 billion cars on the road now
  Even when testing a car to be 99.999999 % functionally correct, we have 1,400
 faults occurring every second (assuming equal distribution of faults)

  Driving 275 million miles only on a race track or limited area is not
 enough!
  Gain only some confidence on some parts of the interaction between the
 autonomous vehicle and its surrounding
  Do not capture different static parts of the environment (road conditions,
 different national regulations, etc.)

 10 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Challenges

  Verifying an autonomous system within its real environment seems to be
 infeasible!
  Need about 9,200 cars, each driving 30,000 miles for one year, to come up with
 the required 275 Mio miles
  There is no guarantee that critical interactions are tested

 Critical interactions: A collection of objects of the environment, their (static) appearance and
 (dynamic) behavior that when orchestrated in the right way, reveals a situation on side of the
 autonomous ego vehicle that lead to a crash even in a case where the ego vehicle would have
 enough time to come up with crash preventing action.

11 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Challenges – Critical scenarios

  Scenario example:
  The ego vehicle drives on the
 right lane of a street
  A car in front of the ego
 vehicle stops and accelerates
 again afterwards
  A pedestrian crosses the road
 coming from left
  Another pedestrian crosses
 the road from right

12 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Summary / Verification architecture based on simulation

  There is a need for verification methodologies allowing to find critical
 scenarios based on simulation!
  Suggested verification architecture:
 Simulation environment
 (test execution)
 Ontology describing
 the environment
 Environment simulation
 Static part
 Ego
 Test case generation vehicle
 Dynamic part

 Pass ✓
 Observer / Oracle
 Fail ✗

13 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Ontologies describing the environment

 0%
 10%
 Ontology = Concepts and their relationship …
 100%
 Domain
 true
 false

14 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Have to consider a lot of scenarios!

  Static part: streets, traffic signs, houses, marks on street, weather
 conditions,…

  Dynamic part: pedestrians, other cars, indention of ego vehicle,… → behavior
 over time

15 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Complexity of ADAS testing

  Number of concepts: n Assuming 100
 concepts, each
  Average number of values for each concepts: k domain of size 5
  Search space for testing (at least): kn → 5100 approx. 1070 !!!

 How to find critical scenarios effectively?
 How to find all critical scenarios?
 How to find the most important critical
 scenarios?

 When to stop generating tests?
 Are there any guarantees?

16 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Test generation approaches

17 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Testing methods (not exhaustive)

  Search-based testing (SBT)  Combinatorial Testing (CT)
  Search for values  A particular combination of
  E.g. using genetic algorithms parameters reveals a fault

 p1

 p2

18 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

� Search-based testing (SBT)
 [focusing on genetic algorithms]

19 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Search-based testing
 Problem: reaching an optimum

 ( )
 ⃗
 This optimum
 Algorithms may not be the
 global optimum!
  Graph-based search algorithms
  A* makes use of heuristics function to
 find a path through a graph
 Hill climbing
  Heuristics search as generalization
 Simulated annealing
  Backtracking
  Random search
  Simulated annealing Given:
 Fitness function ( )
 ⃗ time to collision (TTC) in case of ADAS
  Hill climb search Wanted:
  Genetic programming ⃗ such that f reaches the optimal value

20 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Genetic programming

  Population where each element has a chromosome
  Apply operators like

  Selection based on
 a fitness function
  Crossover
  Mutation

 to generate new populations

21 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

mutate P’1
 P’1
 p1

 p2
 p2 crossover
 p3 P5 p3
 p4
 P5
 select

 populationi populationi+1

22 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

 Represent problem as chromosomes comprising genes. A set of
 chromosomes is called a population
  Chromosomes can be stated as strings (or any other collection)
 crossover
 C1 0 0 0 0 0 0
 1 1 1 0 0 0
 C2 1 1 1 1 1 1
 mutation
 C3 1 0 1 0 1 1 1 0 1 1 1 1

 C4 1 1 0 1 1 0

23 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

 Maps chromosomes to a particular fitness value

 f(.)
 0 0 0 0 0 0 0.923

24 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Initialize Population
 In every step we may have (Generate population randomly)
 a population of same size.
 Use fitness function to
 select n chromosomes.
 no Select Population using fitness
 We stop when reaching a function
 criterion, e.g., one
 chromosome reached a
 certain fitness level.
 yes Stop
 criterion
 Perform crossover
 reached?

 We may generate a lot of
 mutations and other Perform mutation
 offsprings using mutation
 operators and crossover.

26 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Generating scenarios using GA (example)

  Have a look at active cruise control (ACC)
 Initial speed
 Speed at time 1, 2, 3
 Initial speed Initial speed
 Speed wanted Speed at time 1, 2, 3
 80 100 25 80 120 70 40

 Initial speed Speed wanted
 Initial distance
 ego vehicle vehicle in front
 Initial • May use mutation function for
 distance integers
 • Alternatively, we may come up with a
 collection of values for each gene, we
 may select. Mutation would be a
 change of a value for one gene.

27 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Using GA for ADAS verification (case study)

  Using GA for testing an automated emergency braking system (AEB)
  See:
 Florian Klück, Martin Zimmermann, Franz Wotawa, and Mihai Nica. Genetic algorithm-based test parameter
 optimization for ADAS system testing. In Proceedings of the 19th IEEE International Conference on Software
 Quality, Reliability, and Security (QRS), 2019.

28 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Using GA for ADAS verification

  Use Time-to-Collision (TTC) as fitness function
  Parameters:

  Results:

29 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Using GA for ADAS verification

  Random testing performs well
  The probability of finding a fault
 is high

  GA’s focus a lot towards finding
 optimal test cases
  Lower number of invalid tests

30 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

� Combinatorial testing (CT)

31 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

32

 Combinatorial testing principles

  A method that aims to improve the  A combinatorial testing input model
 effectiveness of software testing while is a tuple (V, DOM, CONS) where:
 lowering its cost at the same time.  V is a set of variables
 The essence of CT is that not all  DOM is a function mapping variables
 parameters contribute to failures but from V to a set of values
 by interactions between relatively few  CONS is a set of constraints over
 parameters. variables that have to be fulfilled for
 each test case
  In combinatorial testing we search for
 all tests that cover all combinations Kuhn, D.R., Kacker, R.N., Lei, Y.: Combinatorial testing.
 for any subset of size k of the In: Laplante, P.A. (ed.) Encyclopedia of Software
 variables, where k is called the Engineering. Taylor & Francis (2012)
 strength of the generated test suite.
 32 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Combinatorial testing principles (cont.)

33 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Applying CT to ADAS verification

  See:
 Ontology
 Yihao Li, Jianbo Tao, and Franz Wotawa. Ontology- a formal conceptualization of
 Widely used for different application
 based Test Generation for Automated and Autonomous entities, their interfaces and domains. E.g., making, describing
 behaviors, and relationships, traffic situations and navigations in
 Driving Functions. Information and Software describe the knowledge the context of autonomous driving
 Technology, Volume 117, January, 2020. behind
 https://doi.org/10.1016/j.infsof.2019.106200
 Provides information about
 Conversion
 certain entities and their
 Florian Klück, Yihao Li, Mihai Nica, Jianbo Tao, and relationships to extract test cases
 Franz Wotawa. Using Ontologies for Test Suites
 Generation for Automated and Autonomous Driving Effective
 CT input model
 Functions. In Proc. of the IEEE 29th International takes a set of variables software
 Symposium on Software Reliability Engineering (ISSRE- representing inputs and Combinatorial Testing
 testing for
 parameters together with
 2018) – Industrial Track, Memphis, USA, October 15-18, their domains as input model Wildly used test autonomou
 generation s driving!
 2018 technique

34 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

AEB use case

 Ontology Ont2CT input CT Test Suite
 model algorithm CT input Generation
 Ontology for AEB based on Euro NCAP protocol
 model using UML

 TestCase2Open
 Scenario
 (.xml)

 Parameter values based on Euro NCAP protocol

 Source: Euro NCAP AEB C2C and VRU protocol

 CT Test Execution/Simulation

 Source: Euro NCAP AEB C2C and VRU protocol

35 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Initial results - # of crashes
 Depends on AEB function, Ontology,
 CT input model, combinatorial strength,…

 Crash_flag 17 crashes were found in total
 1
0 - no crash
1- crash

 0
 0 100 200 300 400 500 600 700 800 900 1000
 Test case #

 36 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Comparing GA & CT (considering ADAS verification)

  Comparing GA (SBT), CT and
 Random Testing (RT) making use
 of an AEB function and the same
 input model.
  Distinguish crashes
  Front collision with vehicle
  Front collision with pedestrian
 1/2
  Side collision with pedestrian 1/2

37 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

� Industrial application at AVL

38 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Industrial application – AVL SCENIUS
 Scenario-based Test Preparation, Planning and Execution

 Ontology

 Search-based testing
 AVL Scenario Data AVL Test Case Generator Combinatorial testing
 Manager
 AVL Scenario Designer
 Simulation

 Test Methods
 Test
 AVL Driving Cube
 Scenario Database Test Database Allocation

Easy traffic scenario design, Efficient and high-quality scenario Optimized test planning with
editing, parametrization and handling guided by a standardized increased test performance and Proving Ground
 verification workflow maximized test coverage

 Connecting Data & Ensuring Traceability

39 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Industrial application – AVL Corner and Edge Test Case Generation

 Search-based testing

 Search
 Configuration

 Corner Case
 Parameter

 Simulation Result

 Automatic
 Modification and
 Edge Case
 Evaluation
 Parameter

 Edge Case
 probability
 evaluation

40 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Summary & concluding remarks

41 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Summary

  Finding critical scenarios is essential for testing ADAS!
  The search space for testing is huge!
  Because of the huge number of cars used at any time, a fault is likely to be
 revealed!
  In AD/ADAS there are limitations regarding a fallback driver
  Discussed the automation of test execution (using simulation) and test
 generation (using GA and CT)
  The generated test cases are able to reveal bugs
  There are indications that we need a combinatorial strength of at least 4 if we
 have to guarantee finding a fault in case of the AEB function.

42 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Summary – Automated test execution

  Use simulation for automated test execution
  Combining physic simulation and the ADAS implementation
  There are tools available for simulation (CARLA, Virtual Test Drive, LG, …)
 Simulation environment
 (test execution)
 Ontology describing
 the environment
 Environment simulation
 Static part
 Ego
 Test case generation vehicle
 Dynamic part

 Pass ✓
 Observer / Oracle
 Fail ✗

43 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Summary – Automated test suite generation

  Using different approaches:  Need further studies for coming
  GA (SBT) – focus on finding up with guarantees and an
 critical scenarios estimation of residual risks
  CT – focus on concept  Prototype implementations are
 interactions / completeness available
  Rely on ontologies  Industrial grade tools following
  There are initiatives for coming up some principles are available from
 with ontologies for ADAS (see or partner AVL (Scenario Designer,
 ASAM - OpenXOntology) Scenario Data Manager, Test Case
 Generator, Driving Cube,…)

44 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Guarantees of testing when using automated test suite generation for
 ADAS

  Ontology coverage
  Which parts of the ontology are covered/used for test case generation?
  Fraction of the concepts used and the total number of concepts in the ontology
  Can be adapted to fit a certain purpose, e.g., testing AEBs
  Reduces the search space making test case generation feasible
  Combinatorial strength
  How many concepts to combine for searching of all combinations?
  Reduces the search space
  In testing practice a combinatorial strength of 6 seems to be enough
  Code/model coverage or mutation score

45 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Summary – Concluding remarks

  Virtual testing is necessary for achieving high-quality ADAS
  Automated test execution using simulation environments & standards for
 describing and running scenarios
  Automated test suite generation is possible and relies on finding critical
 scenarios:
  Using search (e.g. GA) trying to minimize TTC or other similar measures
  Using combinations of parameters
  In case virtual testing is not sufficient, we have to think about online
 verification, e.g., monitoring ADAS states, decisions, and perceived
 information, and checking their validity using propositions that must always
 hold (e.g., in case of contradicting information slow down).

46 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh

Thank you for your attention!
 Prof. Dr. Franz Wotawa
 Technische Universität Graz, Institute for Software Technology
 CD Laboratory for Quality Assurance Methodologies for
 Autonomous Cyber-Physical Systems
 Inffeldgasse 16b/2, A-8010 Graz, Austria
 T +43 316 873 5711
 E wotawa@ist.tugraz.at

47 July 01, 2021 Franz Wotawa, TU Graz / IST / QAMCS | © 2021 carhs.training gmbh