THE VISUAL AI LAB IN 2022 - Prof Fabio Cuzzolin The Visual Artificial Intelligence Laboratory
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
THE VISUAL AI LAB IN 2022
Prof Fabio Cuzzolin
The Visual Artificial Intelligence Laboratory
L a b s e m i n a r, S e p t e m b e r 2 2 2 0 2 1
Main News
• Three new funded projects: Epistemic AI (H2020), MAESTRO Jr (EPSRC) and Supponor KTP, for a
total value for us of 1.65M
• Createc KTP is over, SARAS and UKIERI will expire by the end of the year
• Four new members have joined the lab in August/September (Claire, NK, Aytac and Shireen), three
have left us (Neha, Mohamed and Sinesh), three more will join us in the next 3 months
• We have organised three workshops/challenges @ ICCV, MICCAI and IJCAI 2021
• We released a number of datasets: ROAD, ROAD-R, SARAS-MESAD, CAR, CCC
• Avalanche won the "W&B Best Library Award" at the CLVision Workshop this year
• OBR Autonomous came 2nd place in 2021 Formula Student: Artificial Intelligence
• Frontiers topics on theory of mind and continual SS learning, IEEE TNNLS special issue on sensor
fusion, AC for ICCV and ECCV
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 2
Grants
• Current funding: ca £3,200,000
• Nine live funded projects for a total value of around £10M
• ECM project, EU H2020 SARAS, EU FET Epistemic AI, Leverhulme Trust RPG, Innovate UK KTP with
Createc, new KTP with Supponor, UKIERI exchange with IIT Bombay, Huawei Technologies research
agreement, EPSRC THT MAESTRO Jr
• Many pending or upcoming grant applications (list is not complete):
• Olympia.ai (£8M): venture capital funding for a start-up on AI-powered coaching in sports
• EPSRC on Continual epistemic learning (£1.5M)
• Leverhulme on Neurosymbolic AI (£350K)
• EPSRC HT on Theory of mind for empathetic healthcare (£1.5M)
• SARAS++, MAESTRO+, Adaptive Machines (Horizon Europe)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 3
Journal papers
• 1 book
• F. Cuzzolin, The geometry of uncertainty - The geometry of imprecise probabilities, Springer International Publishing, January 2021
• 4 journals
• D. Jackson Samuel and F. Cuzzolin, Unsupervised anomaly detection for a Smart Autonomous Robotic Assistant Surgeon (SARAS) using a deep residual autoencoder, IEEE
Robotics and Automation Letters 6(4), pp. 7256-7261, October 2021 (I.F. 3.6)
• M. Khan et al., An intelligent system for complex violence pattern analysis and detection, International Journal of Intelligent Systems (IF 10.312), July 5 2021;
https://doi.org/10.1002/int.22537
• S. Khan, K. Muhammad, T. Hussain, F. Cuzzolin, S. Bhattacharyya, Z. Akhtar, V.H.C de Albuquerque, DeepSmoke: An enhanced deep learning model for smoke detection and
segmentation in outdoor environments, Expert Systems with Applications 182, pp. 115-125, November 2021 (I.F. 5.452),
• Wojtek Buczynski, Fabio Cuzzolin and Barbara J. Sahakian, A review of machine learning experiments in equity investment decision-making: why most published research
findings do not live up to their promise in real life, Journal of Data Science and Analytics, April 5 2021 https://doi.org/10.1007/s41060-021-00245-5
• Under review or in preparation
• V. Singh et al., The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods, major revision, Medical Image Analysis, April 2021 (IF 11.148)
• G. Singh et al., ROAD: The ROad event Awareness Dataset for Autonomous Driving, major revision, IEEE Transaction on Pattern Analysis and Machine Intelligence (I.F. 17.861)
• B. Cirstea, C. Langley, F. Cuzzolin and B. J. Sahakian, Theory of mind in humans and in machines: A survey, submitted to the Research Topic “Theory of Mind in Humans and in
Machines”, Frontiers in Artificial Intelligence, September 17 2021
• I. Skarha-Bandurova et al., Decision making in the SARAS autonomous assistant surgeon platform, to submit to IEEE Transactions on Medical Robotics and Bionics Journal
• F. Cuzzolin, K. Cannons, M. Hossain and Z. Xu, Continual semi-supervised learning, to submit to IEEE Transaction on Pattern Analysis and Machine Intelligence (I.F. 17.861)
• F. Cuzzolin, Uncertainty measures: the big picture, to submit to IEEE Fuzzy Systems (I.F. 12.03), to submit 2021
SHORT TITLE PROF FABIO CUZZOLIN DATE 4
Conference papers
• So far 4 workshop papers
• Aduen Benjumea, Izzeddin Teeti, Fabio Cuzzolin and Andrew Bradley, YOLO-Z: Improving small object detection in YOLOv5 for autonomous
vehicles, ICCV 2021 Workshop: The ROAD challenge, Oct 16 2021
• A. Shahbaz, S. Khan, M.A. Hossain, V. Lomonaco, K. Cannons, Z. Xu and F. Cuzzolin, International Workshop on Continual Semi-Supervised Learning:
Introduction, Benchmarks and Baselines, IJCAI 2021 - First International Workshop on Continual Semi-Supervised Learning (CSSL @ IJCAI 2021),
August 2021
• V. Musat, I. Fursa, P. Newman, F. Cuzzolin and A. Bradley, Multi-weather city: Adverse weather stacking for autonomous driving, ICCV 2021 - The 2nd
Autonomous Vehicle Vision Workshop (AVVision 2021), October 2021
• V. Lomonaco et al., Avalanche: an End-to-End Library for Continual Learning, CVPR 2021 – 2nd Workshop on Continual Learning in Computer Vision
(CL@VISION), March 2021
• Conference papers under review
• E. Tsamoura, L. Michael, D. Buffelli and F. Cuzzolin, Scalable and Modular Theory-Driven Regularization of Neural-Based Models for Scene Graph
Generation, submitted to NeurIPS 2021
• V. Singh and F. Cuzzolin, Feature boosting with channel attention for scene parsing, submitted to BMVC 21
• S. Khan and F. Cuzzolin, Spatiotemporal Deformable Scene Graphs for Complex Activity Detection, submitted to BMVC 2021
• D. Jackson Samuel and F. Cuzzolin, SVD-GAN for Real-Time Unsupervised Video Anomaly Detection, submitted to BMVC 2021
• E. Giunchiglia, S. Khan, and F. Cuzzolin, ROAD-R: the Autonomous Driving Dataset for Learning with Requirements, submitted to AAAI 2021
• All in all we can still target 15 publications in 2021
• Good mix of lead authors from the lab and external collaborators
SHORT TITLE PROF FABIO CUZZOLIN DATE 5
Our projects THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 6
SARAS – Smart Autonomous Robotic Assistant Surgeon
• Funder: European Union (EU), Horizon 2020
• Budget: €4,315,640 (our share: €596,073)
• Personnel: 4 research fellows (Inna, Vivek,
Jackson and Mohamed)
• Coordinator: Dr Riccardo Muradore,
University of Verona, Italy
• Partners: Verona, Ferrara, Modena,
Ospedale San Raffaele (Milan), Dundee,
Medineering (Munich), ACMIT, Barcelona
• Ends December 2021
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 7
Createc Knowledge Transfer Partnership
• Funder: Innovate UK, Knowledge
Transfer Partnerships
• Amount: £190,000
• Duration: May 2019 – May 2021
• Personnel: Neha Bhargava
• Partner companies: Createc
Technologies, Sportlight.ai
• https://www.sportlight.ai/
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 8
Theory of mind at the interface of neuroscience & AI
• Funder: Leverhulme Trust, Research
Project Grant RPG-2019-243
• Amount: > £273,000
• Personnel: two postdoctoral research
assistants (Bogdan and Christelle)
• Duration: Feb 2020 – Feb/Mar 2024
• Co-investigator: Professor Barbara
Sahakian, University of Cambridge
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 9
AI for Autonomous Driving
• Funder: School of Engineering, Computing and
Mathematics
• Amount: £100,000
• Personnel: Aytac Kanaci
• 15 months left on the project
• Co-investigator: Dr Andrew Bradley,
Autonomous Driving group
• Perception for autonomous driving, simulated
environments (CARLA)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 10Some novel paradigms for analyzing human actions
in complex videos
• Funder: UKIERI
• Amount: £48,076
• Personnel: Omkar Gune, Dipesh Tamboli
• Duration: Apr 2019 – Dec 2021
• Topic: continual zero-shot action
recognition
• Creating benchmark from standard UCF-
101 dataset
• Co-investigators: Prof Biplab Banerjee,
Prof Subhasis Chaudhuri
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 11Modelling complex activities in videos
• Funder: Huawei Technologies
• Amount: £278,000
• Personnel: Salman Khan and
Ajmal Shahbaz
• Duration: Dec 2019 – Nov 2021
• Topic: modelling of complex
activities composed by atomic
actions
• Salman augmented ROAD with
complex road activities
• Implemented 3D ROI pooling +
scene graph representation
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 12Epistemic AI
• Funder: EU Horizon 2020
• Amount: 1,209,000 euros (3M in
total)
• We are the Coordinators
• Personnel: two research fellows
(NK) + two PhD students
(Shireen) + one lab
administrator (Claire)
• Duration: Mar 2021 – Feb 2025
• Topic: injecting second-order
uncertainty into the foundations
of artificial intelligence
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 13KTP with Supponor
• Funder: Innovate UK
• Amount: £310,000
• Personnel: one KTP associate
(Vivek Singh)
• Duration: 3 years
• Topic: video scene
understanding applied to
professional sports footage
• Consistent segmentation, video
inpainting
• Handling of occlusions,
boundaries
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 14Our team THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 15
Personnel
• Faculty members: Prof Fabio Cuzzolin (Director), Dr Tjeerd Olde-Scheper (Senior Lecturer), Dr Andrew
Bradley (Senior Lecturer), Dr Alex Rast (Lecturer), Dr Matthias Rolf (Senior Lecturer), Prof Inna Skarga-
Bandurova (Senior Lecturer)
• Inna has secured a Senior Lectureship and will stay with us
• Current personnel: 7 Research Fellows (Vivek, Inna, Bogdan, Christelle, Ajmal, Aytac, Naeemullah), 5
PhD students (Salman, Devashish, Izzeddin, Shireen and Wojtek, based in Cambridge), 4 MSc students,
8-10 remote collaborators (Filippo, Andrea, Gurkirt, Suman, Reza, Silvio, Vincenzo and others)
• Four new members: Naeemullah, Shireen, Aytac, Claire
• Upcoming: another E-pi RF, another E-pi PhD student, a postdoc funded by MAESTRO
• In total we are expected to be around 32-34 people in 2022
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 16Epistemic AI E-PI CREATEC KTP
Zero-shot learning
UKIERI Autonomous driving
N. YORKE-SMITH I. COWLING
NEW PHD NEW FELLOW
B. BANERJEE D. TAMBOLI
A. DIEPA NEW FELLOW A. BRADLEY V. MUSAT
K. SHARIATMADAR NEW PHD NEW FELLOW
SARAS N. BHARGAVA
Uncertainty Surgical G. SINGH I. TEETI R. ALITAPPEH S. SAHA
theory robotics S. KHAN A. SHAHBAZ
LAB ADMIN
ECM
HUAWEI Activity detection
I. SKARHA M. MOHAMED
RL & IRL LEVERHULME The team
D. BHARTI
Federated
learning
G. ROSSI
J. SAMUEL V. SINGH
F. VELLA
F. CUZZOLIN M. ROLF
B. CIRSTEA A. MORELLI
AI in finance
SUPPONOR
E. TSAMOURA S. OLIVASTRI J. BREWSTER
KTP
A. RAST W. BUCZYNSKI
NEW ASSOCIATE
Neurosymbolic Video captioning K. CANNONS V. LOMONACO C. LANGLEY B. SAHAKIAN
reasoning Scene understanding Continual learning Machine theory of mind
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 17Our research themes THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 18
Overview
• Computer Vision (deep learning for action
detection, video captioning, complex activities,
future event prediction, scene understanding)
• Machine Learning (continual learning, federated
learning, self-supervised learning)
• Artificial Intelligence (machine theory of mind,
epistemic AI, neuro-symbolic reasoning)
• AI for Healthcare (audio-visual monitoring of
people in a coma, empathetic AI)
• Surgical Robotics (SARAS, MAESTRO)
• Autonomous Driving (Formula Student AI, ROAD)
• Uncertainty Theory (random sets, belief
functions, geometry of uncertainty)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 19Computer vision THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 20
Real time human action detection
• The Visual AI Lab is leader in the field, multi-
year expertise, first system able to detect
multiple human actions and events in real time
• https://sahasuman.bitbucket.io/
• Won awards at both CVPR 17 Charades and
ActivityNet 2016 action detection challenges
• Based on deep neural networks [BMVC’16,
BMVC’18, ICCV’17x2, ACCV’18]
• Frame-level detections are linked up in real
time to form ‘action tubes’
• Still very influential work in the field, how do
we capitalise on this? Move into object
detection proper?
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 21Prediction of future events
• We are now moving towards the early recognition of events and activities [ECCVW’18, ICCV’19]
• Just by observing a small fraction of the video frames we can guess what is going on there
• We aim to frame this in the context of modelling complex activities
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 22Causal 3D CNNs
• 3D CNNs (C3D, I3D, (2+1)D) have risen to
the forefront of video classification
• However, they work on entire videos as a
whole in a batch manner
• We recently proposed a Recurrent
Convolutional Network designed to replace
the temporal convolution component with a
recurrent model in the temporal dimension
• [Gurkirt Singh, ICCV’19 submission, PAMI in
preparation]
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 23From simple actions to complex activities
• Topic: modelling complex road activities as graphs of ‘atomic’ actions/events
• A deep network architecture is designed to detect those graphs of actions
• When one is partially observed, we can guess what events may take place in the future and where
• 3-year Huawei research agreement, EU H2020 SARAS surgical robotics project
• 3D ROI pooling pipeline, graph GCN representation, extra annotation provided to the ROAD dataset
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 24Dynamic heterogenous scene graphs
• From complex activities to full dynamic
scene representations
• Adding scene elements (e.g. traffic
lights, lamp posts, etc) and attributes to
the graph modelling a dynamic scene
• Approach: heterogenous graphs with
nodes and edges of different types
• Inference on heterogenous graphs
• Dynamic HGs have not been proposed
yet
• Role of knowledge bases
• Links to symbolic reasoning?
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 25Video anomaly detection
• Part of the SARAS project; research fellow Dinesh Jackson Samuel
• Problem: identify any deviation from nominal behaviour in a video (in particular the endoscopic video
of a surgical procedure)
• Approach based on a new SVD loss in an architecture inspired by GANomaly
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 26Video scene understanding (and inpainting)
• Funded through the new KTP with
Supponor, but also part of SARAS
• The problem is do accurate video scene
understanding from a streaming video, in
a temporally-consistent fashion
• We are exploring the boosting of features
from the intermediate layers ..
• .. as well as the use of scene graphs to • In the sports advertisement domain, we want to
help the labelling .. identify and the commercial signage around the
• .. and the notion of ‘soft’ segmentation to pitch and inpaint them with virtual content
address boundary artifacts • Replaces Supponor’s current IR-based technology
and active sensors
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 27Surgical Robotics THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 28
SARAS – Smart Autonomous Robotic Assistant Surgeon
• We are in charge of WorkPackage 6
(Situation awareness)
• Online surgeon action detection
• Procedure stage segmentation
• Anomaly detection
• We also contribute to WP5 (Perception)
• Dynamic semantic understanding
• Surgical tool tracking and pose estimation
• WorkPackage 3
• Speech recognition module
• Inna, Vivek, Mohamed and Dinesh
• Extended to December 2021
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 29The SARAS Surgical Action Dataset (ESAD)
• Dataset of 4 real life, four-hour-
long videos capturing
lapascopic surgery procedures
• Annotation done with
Microsoft VoTT
• 21 action classes
• Released for MIDL 2020 ESAD
Challenge
• https://saras-esad.grand-
challenge.org/Home/
• MIA journal under major
revision (Vivek)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 30Multi-domain surgeon action detection (MESAD)
• Builds on the MIDL 2020 challenge to
launch a MICCAI 2021 challenge
• Makes use of the 4 real videos and 6
videos captured using artificial anatomies
(phantoms) as part of SARAS
• Concept: exploring the opportunity of
utilising cross-domain knowledge (in
particular videos capturing procedures
conducted on real as opposed to artificial
organs and anatomies) to boost model
performance on each individual task
• Compound performance measure
assessing accuracy and parameter sharing
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 31MAESTRO Jr - Multi-sensing AI Environment for Surgical Task &
• EPSRC THT grant Role Optimisation
• Goal: to devise an AI-
assisted operating room of
the 2050s
• Array of multiple sensors to
monitor surgical checklists,
teamwork
• Monitoring of cognitive
load
• MORL for improving team
performance
• Led by George Mylonas at • Continual learning of perception and decision making models, both
Imperial College supervised and unsupervised
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 32Machine learning THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 33
Statistical learning theory
• Problem: model adaptation in supervised learning
• Statistical learning theory: computes generalisation bounds for the test error of models
• Assumes training and testing distribution are the same!
• Proposal: assume they come from a common convex set of probability distributions (now part of
Epistemic AI FET project)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 34Continual semi-supervised learning
• Interesting work has been recently directed at continual
learning from streaming data in a fully supervised setting
• Focus mostly on avoiding catastrophic forgetting
• Continual learning in a semi-supervised setting remains a
wide-open research question
• The problem can be reduced to a continual supervised
learning setting under a 'multiple worlds' assumption
• CSSL @ IJCAI 2021 workshop in August 2021
• Possible Horizon Europe collaborative project (see later)
• CSSL @ IJCAI workshop paper
• Possible follow-up agreement with Huawei
• CAR and CCC continual learning benchmarks • IEEE PAMI paper in preparation
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 35CAR and CCC benchmarks and challenges
• Our CAR (Continual Activity Recognition) dataset is built
upon the MEVA (Multiview Extended Video with Activities)
dataset
• Aim: benchmarking continual, long duration semi-
supervised learning in a classification setting by classifying
the input video frames in terms of activity classes
• Each video frame is annotated in terms of 8 different
activity classes selected from the 37 original classes (e.g.
person_enters_scene_through_structure,
person_exits_vehicle, vehicle_starts) + a bkground class
• Continual Crowd Counting (CCC) dataset combines three
existing crowd counting datasets (Mall, UCSD and FDST),
for continual learning
• Open challenges, move to Avalanche platform
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 36Avalanche
• An End-to-End Continual Learning PyTorch Library
• Provides a shared and collaborative open-source
codebase for fast prototyping, training and reproducible
evaluation of continual learning algorithms
• Initiative of ContinualAI
• Avalanche can help Continual Learning researchers and
practitioners in several ways:
• Write less code, prototype faster & reduce errors
• Improve reproducibility
• Improve modularity and reusability
• Increase code efficiency, scalability & portability
• Augment impact and usability of your research products
• Organised into 5 modules: Benchmarks, Training,
Evaluation, Models, Logging
SHORT TITLE PROF FABIO CUZZOLIN DATE 37Federated learning
• Concept: different parties can work together on
better models without having to share the data
• Best suited to healthcare domain: hospitals
working to achieve better diagnosis from x-ray
images without disclosing patient data
• PhD student Devashish Bharti
• Ongoing collaboration with Converz, pending
Innovate UK grant (https://www.converz.co.uk/)
• Possible approaches: meta-learning
• Devashish is working on a benchmark and
performance measures for the problem
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 38Artificial Intelligence THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 39
Machine theory of mind
• Humans have complex mental lives, which make them unpredictable
• E.g.: children walking on the pavement spot an ice cream van on the other side – are they gonna cross?
• Solution: giving machines theory of mind abilities, i.e.
the ability to guess other agents’ reasoning process
• https://www.facebook.com/oxfordbrookes/videos/101
56698398637908/
• Idea: reinforcement learning of ‘accurate’ simulations
of people’s mental reasoning
• [Leverhulme Grant with Cambridge Neuroscience,
2020-2024]
• Bogdan-Ionut Cirstea, Barbara Sahakian and Christelle
Langley (Cambridge)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 40Neuro-symbolic AI
• Integration of learning and reasoning is one of the elements of a future
artificial intelligence
• We have started collaborating on this topic with Samsung AI
Cambridge (Efi Tsamoura)
• Various lines of research:
• Loss functions expressing logical constraints on the output of a nettwork
• Logical theories processing the output of a NN
• From probabilistic logic to epistemic logic (Leverhulme Grant)
• Appeal: possibility of transferring knowledge across disparate domains
• Devising networks that generate propositional solutions to problems
(e.g. scene understanding)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 41ROAD-R: learning with requirements
• AAAI 2022 paper led by Eleonora Giunchiglia
(Oxford) and Salman Khan
• Idea: assess how often neural network
predictions violate common-sense
requirements
• Introduces the Autonomous Driving Dataset
for Learning with Requirements (ROAD-R)
• In ROADR, the requirements express
conditions on the set of admissible road
events, and are formulated as constraints
• We show that current SOTA models produce
non-admissible predictions on ROAD-R
• We propose two baseline methods for
computing a minimal set of corrections
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 42Neural-guided projection
• NeurIPS 2021 submission led by Efi Tsamoura (Samsung AI, Oxford)
• Example of training-time regularization (TTR) in stratified neural symbolic architectures
• Proposes a neural-guided projection (NGP) procedure that identifies a small subset of constraints
whose theory-prescribed values are maximally violated under the neural predictions
• Solves an optimization problem in which the neural module is updated to minimize the maximum
“unsatisfiability" of the constraints
• Scalable, modular, outperforms all state of the art when applied to scene graph generation
• Can be embedded in the neurosymbolic NeuroLog framework
SHORT TITLE PROF FABIO CUZZOLIN DATE 43Epistemic artificial intelligence
• Horizon 2020 FET (Future Emerging Technologies)
project, coordinated by us
• 3M euros over 4 years, with KU Leuven and TU Delft
• Bottom line: uncertainty is not well modelled in AI,
and that makes AI models brittle
• Change of paradigm: from striving for ‘generalisation’
or ‘adaptation’ to working with convex sets of models
• Use the available information to reduce the
uncertainty about the ‘true’ model
• Well suited to the continual learning paradigm
• Contributions to supervised, unsupervised and
reinforcement learning
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 44Epistemic AI
• What are the main targets of the Epistemic AI programme?
• Theory of constrained optimisation under epistemic
uncertainty
• Losses for random sets that generalise cross entropy in deep
learning
• GANs (generative adversarial networks) rely on stochastic
sampling from a latent variable
• Design robust GAN framework based on random set theory
• Reinforcement learning also relies on the framework of
(Partially Observable) Markov Decision Models
• Generalisations of MDPs and Markov chains exist
• Can we lay down the foundations of a ‘robust’ reinforcement
learning framework?
• Generalising Belmann’s equations
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 45Autonomous driving THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 46
AI for Autonomous Driving
• Can we predict future
behaviour using machine
theory of mind?
• (1) future behaviour is
guessed
• (2) by explaining the chain of
reasoning
• AI Research Fellow (Aytac
Kanaci) and PhD student
Izzeddin Teeti
• Collaboration with
Autonomous Driving group
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 47ROAD – The ROad event Awareness Dataset for Autonomous Driving
• We created the world's first ROad event
Awareness Dataset for Autonomous
Driving (ROAD)
• Annotating a number of videos from the
existing Oxford RobotCar Dataset
• 22 videos, 122K frames, 560K bounding
boxes, 1.7M labels
• Multi-label dataset: agent (11 classes),
action (23), location (15)
• Aim: test prediction capabilities,
complex road activities, situation
awareness
• [IEEE PAMI, ROAD @ ICCV’21 workshop]
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 48Formula Student - AI
• Collaboration with the Autonomous Tech
student society and the Autonomous Driving
group (Andy Bradley, Petar Georgiev)
• https://www.imeche.org/events/formula-
student/team-information/fs-ai
• FS-AI has been introduced to challenge student
teams to develop an AI driver capable of
controlling a purpose designed racing car
• Also ‘static’ event categories, e.g. students to
also consider Real World Autonomous scenarios
• Our brand new Autonomous Formula Student
team participated in the first edition of the UK
Formula Student AI competition, coming 3rd
place in 2019, 1st overall in 2020 and 2nd in 2021!
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 49Video synthesis for model adaptation
• Led by final year student Valentina
Musat
• Inspired by Vid2vid framework
(https://arxiv.org/pdf/1808.06601)
• GAN (Generative Adversarial
Network) – based approach, need
to enforce temporal consistency
• Goal is be able to synthesise
realistic videos for training
autonomous cars
• Leading to a weather
augmentation paper to submit
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 50Scale-invariant object detectors
• Robust object detectors with respect to scale
• Object-tube approach to detection
• Topic of final-year project by Aduen Benjumea Diepa
• Based on the SNIPER framework
• Instead of processing every pixel in an image pyramid,
SNIPER processes context regions around ground-truth
instances (referred to as chips) at the appropriate scale
• B. Singh, M. Najibi and L. S. Davis. SNIPER: Efficient
Multi-Scale Training. NeurIPS 2018
• Application to autonomous driving and F Student AI
• ROAD @ ICCV workshop paper
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 51AI for healthcare THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 52
Multimodal monitoring of people in a coma
• Automated monitoring of people with prolonged disorder of consciousness
• Past EPSRC bid with Prof Helen Dawes and Prof Derick Wade (MOReS), in process of rewriting
• Core: modelling and detection of audio-visual events using multimodal deep learning
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 53Empathetic AI
• Problem: communicating with people
with cognitive impairment in care homes
• Idea: bespoke machine theory of mind
approach
• Understanding both physical and
emotional needs
• Machines willing to help?
• Linked to the Leverhulme grant
• Bid in preparation for EPSRC Healthcare
Technologies call
• Partners: Derick Wade, Helen Dawes and
Bogdan Cirstea
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 54Uncertainty theory THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 55
Geometry of uncertainty
• Probability theory is not the only mathematical
theory of uncertainty
• Others exists: imprecise probabilities, random
sets, belief functions are the most popular
• Uncertainty measures can be represented as
points of a convex space, and there analysed
• I wrote two books on the topic, one coming out
with Springer - Artificial Intelligence:
Foundations, Theory, and Algorithms
• Past work on the geometry of belief functions,
fuzzy sets, combination rules, conditioning
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 56The geometry of uncertainty – The geometry of imprecise
probabilities
• My monograph on the geometry of uncertainty
measures is finally out (after 9 years!)
• 860 pages!
• Most complete compendium of belief function theory
• Most complete overview of the general landscape of
uncertainty theory
• Compendium of my geometric approach
• Agenda for the future of random set theory
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 57Generalised statistical inference
• How to infer a belief function/random set from
sample data?
• This is the inference problem
• The two main existing mechanisms in belief
theory rely on classical probabilistic inference
• Concept: generalise maximum likelihood
inference in order to infer directly a belief
function from data
• Further step: generalise logistic regression
• Can be applied to estimation of rare events
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 58Our partners THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 59
Institute for Ethical AI
• Created by Prof Nigel Crook
• https://ethical-ai.ac.uk/
• Co-director Kevin Maynard
• Fabio is in Board
• Aims at establishing and maintaining a cross-
university network on the topic
• Themes: explainability, ethical implications of AI,
ethical AI from a technical point of view, advising
• A network of interested companies has been set up
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 60Healthy Ageing network
• Cross-university network involving Business, Health
Sciences, Architecture, and us
• Coordinated by Juliet Bligh of RBDO
• Topics: sustaining physical activity, living well with
cognitive impairment, design for age-friendly
homes, etcetera
• Goal: stimulate new ideas from a wide range of
businesses and social enterprises to develop and
deliver products, services and business models at a
large scale to support people as they age
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 61OxCATTS
• Oxford Clinical Allied Technology and Trial Services
Unit
• https://oxcatts.com/
• Funded by Helen Dawes
• Fabio is in the Steering Committee
• Aim: increase cross-fertilisation between our work
in AI and in healthcare/rehabilitation
• Mission is to provide consultancy and support
research grant applications
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 62Within the School and University
Cognitive Robotics (Prof Crook and Dr Rolf)
MOReS (Prof Dawes, Wade; Dr Esser)
Autonomous Driving group (Andrew Bradley)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 63Within the UK
Torr Vision Group, Oxford University (Prof Torr)
Computer Science Dept, Oxford University
Neuroscience, Cambridge University (Prof Sahakian)
Imperial College London (Dr Mylonas)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 64Across the world
Altair Robotics Lab, Verona (Prof Fiorini, Dr Muradore)
Federico II University, Naples (Prof Di Gironimo)
Indian Institute of Technology, Bombay (Prof Banerjee and Chaudhuri )
KU Leuven, Belgium (Dr Shariatmadar)
TU Delft, Netherlands (Prof Yorke-Smith)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 65Companies
Huawei Technologies (Dr Zhan Xu)
Cisco Systems (Dr William Wu)
Createc (Matt Mellor, Ian Cowling)
Supponor (Andrew Crean, Steve Plunkett)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 66Companies
AI Labs, Bologna (Massimo Ciociola)
Samsung AI (Efi Tsamoura)
Peoplespace (Halbyn Rich)
Ocado (Dr Graham Deacon)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 67Societies and networks
• ContinualAI (Vincenzo Lomonaco): https://www.continualai.org/
• CLAIRE: European federation of AI laboratories (https://claire-ai.org/ )
• The Society for Imprecise Probability (http://www.sipta.org/)
• The Belief Functions and Applications Society (https://bfasociety.org/)
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 68The next steps THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 69
Upcoming grant applications
• A number of grant applications are in the making
• €10M Horizon Europe - MAESTRO - Multi-sensing AI Environment for Surgical Task & Role
Optimisation (led by Imperial)
• €4-5M SARAS++ - follow up of SARAS
• Horizon Europe on Adaptive Machines (with ContinualAI and others)
• £1.5M EPSRC bid on Continual epistemic learning (natural prosecution of E-pi)
• £1.5M EPSRC Healthcare Technologies on Theory of mind for the cognitively impaired (CoIs
Dawes and Wade)
• £350K Leverhulme RPG on neurosymbolic reasoning (led by Samsung AI)
• Other high-impact areas to seek funding for: self-supervised learning, goal-changing machines,
epistemic modelling of pandemics
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 70Project Olympia
• The world’s first AI coaching app
• Powered by our action detection
technologies
• Negotiating initial venture capital for
£8,000,000, dozens of researchers and
engineers
• Co-founders: Pascal Pert, EIT, Chairman
of innovation and marketing at World
Taekwondo Europe
• Marc Faubeau
• Numerous gold medallists and
testimonials and influencers
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 71Project Olympia
• Various platforms, based on our action
detection know how
• Platform 1: Compilation of simple stats
on no of kicks, jumps, …
• Platform 2: Assisted video review of
significant events
• Platform 3: Video-based assessment of
quality of exercise
• Platform 4: RL-based personalised
recommendations for improvement
• Will be based in the Wood Centre for
Innovation, Oxford (?)
• Initial prototype and demo
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 72SARAS++
• Follow-up from the SARAS project on
surgical robotics
• Big industrial partners like Philips,
Medtronic (the largest supplier of
medical electronics), Karl Storz (the
leading endoscope manufacturer)
• Budget: TBD, but in excess of 5M
• Looking for appropriate Horizon Europe
call
• Probably also led by Verona?
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 73MAESTRO • Follow up to MAESTRO Jr, aims to deliver the smart operating room of the future • Big industrial partners including Johnson & Johnson; Again looking for new Horizon Europe calls • Coordinated by Imperial College (Mylonas); Budget: ca 10M euros THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 74
Theory of self-supervised learning
• Self-supervised learning empowers us to exploit the
variety of labels that usual come with the data for
free – leading to ‘human-level’ intelligence??
• Technically, the idea is to define an auxiliary task for
which we already have labels, 'hidden' within the
structure of the data itself
• This happens by defining a self-supervised task, also
known as pretext task, which guides us to a
supervised loss function
• However, no theoretical foundations for self-
supervised learning yet exist
• Our goal is to provide a theoretical justification for
self-supervised learning through a combination of
functional analysis and optimisation theory
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 75Goal-changing machines
• That’s fine, but what about the ability of
people to change their goal(s) in time, or
juggle various different goals at the same
time?
• Seeking models that update their very
objective function with time
• Interesting discussion with Prof Gary
Browning
• Again, strong intuition and insights can
come from psychology and neuroscience
• https://chrisguillebeau.com/how-goals-
change-over-time/
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 76Our targets THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 77
Targets
• Increasing output in terms of publications -> ideally we should be in a position to submit 10 papers
a year to major computer vision and AI conferences
• Boosting our citations figures -> investing in frontier topics with high potential impact
• Target top venues such as Nature (as we did for our ToM editorial)
• Further doubling the research funding to reach £5-6M
• Research fellows should start engaging with grant applications
• Attracting venture capital and generating spin-offs, starting from the Olympia project -> another
option is applying our ToM concept to autonomous driving
• Continue organising workshops and challenges, releasing new benchmark datasets (e.g. ROAD,
ESAD and the continual learning ones)
• In the medium term, the creation of an Oxford Institute for Next-generation AI
THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 78You can also read
