THE VISUAL AI LAB IN 2022 - Prof Fabio Cuzzolin The Visual Artificial Intelligence Laboratory
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
THE VISUAL AI LAB IN 2022 Prof Fabio Cuzzolin The Visual Artificial Intelligence Laboratory L a b s e m i n a r, S e p t e m b e r 2 2 2 0 2 1
Main News • Three new funded projects: Epistemic AI (H2020), MAESTRO Jr (EPSRC) and Supponor KTP, for a total value for us of 1.65M • Createc KTP is over, SARAS and UKIERI will expire by the end of the year • Four new members have joined the lab in August/September (Claire, NK, Aytac and Shireen), three have left us (Neha, Mohamed and Sinesh), three more will join us in the next 3 months • We have organised three workshops/challenges @ ICCV, MICCAI and IJCAI 2021 • We released a number of datasets: ROAD, ROAD-R, SARAS-MESAD, CAR, CCC • Avalanche won the "W&B Best Library Award" at the CLVision Workshop this year • OBR Autonomous came 2nd place in 2021 Formula Student: Artificial Intelligence • Frontiers topics on theory of mind and continual SS learning, IEEE TNNLS special issue on sensor fusion, AC for ICCV and ECCV THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 2
Grants • Current funding: ca £3,200,000 • Nine live funded projects for a total value of around £10M • ECM project, EU H2020 SARAS, EU FET Epistemic AI, Leverhulme Trust RPG, Innovate UK KTP with Createc, new KTP with Supponor, UKIERI exchange with IIT Bombay, Huawei Technologies research agreement, EPSRC THT MAESTRO Jr • Many pending or upcoming grant applications (list is not complete): • Olympia.ai (£8M): venture capital funding for a start-up on AI-powered coaching in sports • EPSRC on Continual epistemic learning (£1.5M) • Leverhulme on Neurosymbolic AI (£350K) • EPSRC HT on Theory of mind for empathetic healthcare (£1.5M) • SARAS++, MAESTRO+, Adaptive Machines (Horizon Europe) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 3
Journal papers • 1 book • F. Cuzzolin, The geometry of uncertainty - The geometry of imprecise probabilities, Springer International Publishing, January 2021 • 4 journals • D. Jackson Samuel and F. Cuzzolin, Unsupervised anomaly detection for a Smart Autonomous Robotic Assistant Surgeon (SARAS) using a deep residual autoencoder, IEEE Robotics and Automation Letters 6(4), pp. 7256-7261, October 2021 (I.F. 3.6) • M. Khan et al., An intelligent system for complex violence pattern analysis and detection, International Journal of Intelligent Systems (IF 10.312), July 5 2021; https://doi.org/10.1002/int.22537 • S. Khan, K. Muhammad, T. Hussain, F. Cuzzolin, S. Bhattacharyya, Z. Akhtar, V.H.C de Albuquerque, DeepSmoke: An enhanced deep learning model for smoke detection and segmentation in outdoor environments, Expert Systems with Applications 182, pp. 115-125, November 2021 (I.F. 5.452), • Wojtek Buczynski, Fabio Cuzzolin and Barbara J. Sahakian, A review of machine learning experiments in equity investment decision-making: why most published research findings do not live up to their promise in real life, Journal of Data Science and Analytics, April 5 2021 https://doi.org/10.1007/s41060-021-00245-5 • Under review or in preparation • V. Singh et al., The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods, major revision, Medical Image Analysis, April 2021 (IF 11.148) • G. Singh et al., ROAD: The ROad event Awareness Dataset for Autonomous Driving, major revision, IEEE Transaction on Pattern Analysis and Machine Intelligence (I.F. 17.861) • B. Cirstea, C. Langley, F. Cuzzolin and B. J. Sahakian, Theory of mind in humans and in machines: A survey, submitted to the Research Topic “Theory of Mind in Humans and in Machines”, Frontiers in Artificial Intelligence, September 17 2021 • I. Skarha-Bandurova et al., Decision making in the SARAS autonomous assistant surgeon platform, to submit to IEEE Transactions on Medical Robotics and Bionics Journal • F. Cuzzolin, K. Cannons, M. Hossain and Z. Xu, Continual semi-supervised learning, to submit to IEEE Transaction on Pattern Analysis and Machine Intelligence (I.F. 17.861) • F. Cuzzolin, Uncertainty measures: the big picture, to submit to IEEE Fuzzy Systems (I.F. 12.03), to submit 2021 SHORT TITLE PROF FABIO CUZZOLIN DATE 4
Conference papers • So far 4 workshop papers • Aduen Benjumea, Izzeddin Teeti, Fabio Cuzzolin and Andrew Bradley, YOLO-Z: Improving small object detection in YOLOv5 for autonomous vehicles, ICCV 2021 Workshop: The ROAD challenge, Oct 16 2021 • A. Shahbaz, S. Khan, M.A. Hossain, V. Lomonaco, K. Cannons, Z. Xu and F. Cuzzolin, International Workshop on Continual Semi-Supervised Learning: Introduction, Benchmarks and Baselines, IJCAI 2021 - First International Workshop on Continual Semi-Supervised Learning (CSSL @ IJCAI 2021), August 2021 • V. Musat, I. Fursa, P. Newman, F. Cuzzolin and A. Bradley, Multi-weather city: Adverse weather stacking for autonomous driving, ICCV 2021 - The 2nd Autonomous Vehicle Vision Workshop (AVVision 2021), October 2021 • V. Lomonaco et al., Avalanche: an End-to-End Library for Continual Learning, CVPR 2021 – 2nd Workshop on Continual Learning in Computer Vision (CL@VISION), March 2021 • Conference papers under review • E. Tsamoura, L. Michael, D. Buffelli and F. Cuzzolin, Scalable and Modular Theory-Driven Regularization of Neural-Based Models for Scene Graph Generation, submitted to NeurIPS 2021 • V. Singh and F. Cuzzolin, Feature boosting with channel attention for scene parsing, submitted to BMVC 21 • S. Khan and F. Cuzzolin, Spatiotemporal Deformable Scene Graphs for Complex Activity Detection, submitted to BMVC 2021 • D. Jackson Samuel and F. Cuzzolin, SVD-GAN for Real-Time Unsupervised Video Anomaly Detection, submitted to BMVC 2021 • E. Giunchiglia, S. Khan, and F. Cuzzolin, ROAD-R: the Autonomous Driving Dataset for Learning with Requirements, submitted to AAAI 2021 • All in all we can still target 15 publications in 2021 • Good mix of lead authors from the lab and external collaborators SHORT TITLE PROF FABIO CUZZOLIN DATE 5
SARAS – Smart Autonomous Robotic Assistant Surgeon • Funder: European Union (EU), Horizon 2020 • Budget: €4,315,640 (our share: €596,073) • Personnel: 4 research fellows (Inna, Vivek, Jackson and Mohamed) • Coordinator: Dr Riccardo Muradore, University of Verona, Italy • Partners: Verona, Ferrara, Modena, Ospedale San Raffaele (Milan), Dundee, Medineering (Munich), ACMIT, Barcelona • Ends December 2021 THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 7
Createc Knowledge Transfer Partnership • Funder: Innovate UK, Knowledge Transfer Partnerships • Amount: £190,000 • Duration: May 2019 – May 2021 • Personnel: Neha Bhargava • Partner companies: Createc Technologies, Sportlight.ai • https://www.sportlight.ai/ THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 8
Theory of mind at the interface of neuroscience & AI • Funder: Leverhulme Trust, Research Project Grant RPG-2019-243 • Amount: > £273,000 • Personnel: two postdoctoral research assistants (Bogdan and Christelle) • Duration: Feb 2020 – Feb/Mar 2024 • Co-investigator: Professor Barbara Sahakian, University of Cambridge THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 9
AI for Autonomous Driving • Funder: School of Engineering, Computing and Mathematics • Amount: £100,000 • Personnel: Aytac Kanaci • 15 months left on the project • Co-investigator: Dr Andrew Bradley, Autonomous Driving group • Perception for autonomous driving, simulated environments (CARLA) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 10
Some novel paradigms for analyzing human actions in complex videos • Funder: UKIERI • Amount: £48,076 • Personnel: Omkar Gune, Dipesh Tamboli • Duration: Apr 2019 – Dec 2021 • Topic: continual zero-shot action recognition • Creating benchmark from standard UCF- 101 dataset • Co-investigators: Prof Biplab Banerjee, Prof Subhasis Chaudhuri THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 11
Modelling complex activities in videos • Funder: Huawei Technologies • Amount: £278,000 • Personnel: Salman Khan and Ajmal Shahbaz • Duration: Dec 2019 – Nov 2021 • Topic: modelling of complex activities composed by atomic actions • Salman augmented ROAD with complex road activities • Implemented 3D ROI pooling + scene graph representation THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 12
Epistemic AI • Funder: EU Horizon 2020 • Amount: 1,209,000 euros (3M in total) • We are the Coordinators • Personnel: two research fellows (NK) + two PhD students (Shireen) + one lab administrator (Claire) • Duration: Mar 2021 – Feb 2025 • Topic: injecting second-order uncertainty into the foundations of artificial intelligence THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 13
KTP with Supponor • Funder: Innovate UK • Amount: £310,000 • Personnel: one KTP associate (Vivek Singh) • Duration: 3 years • Topic: video scene understanding applied to professional sports footage • Consistent segmentation, video inpainting • Handling of occlusions, boundaries THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 14
Our team THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 15
Personnel • Faculty members: Prof Fabio Cuzzolin (Director), Dr Tjeerd Olde-Scheper (Senior Lecturer), Dr Andrew Bradley (Senior Lecturer), Dr Alex Rast (Lecturer), Dr Matthias Rolf (Senior Lecturer), Prof Inna Skarga- Bandurova (Senior Lecturer) • Inna has secured a Senior Lectureship and will stay with us • Current personnel: 7 Research Fellows (Vivek, Inna, Bogdan, Christelle, Ajmal, Aytac, Naeemullah), 5 PhD students (Salman, Devashish, Izzeddin, Shireen and Wojtek, based in Cambridge), 4 MSc students, 8-10 remote collaborators (Filippo, Andrea, Gurkirt, Suman, Reza, Silvio, Vincenzo and others) • Four new members: Naeemullah, Shireen, Aytac, Claire • Upcoming: another E-pi RF, another E-pi PhD student, a postdoc funded by MAESTRO • In total we are expected to be around 32-34 people in 2022 THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 16
Epistemic AI E-PI CREATEC KTP Zero-shot learning UKIERI Autonomous driving N. YORKE-SMITH I. COWLING NEW PHD NEW FELLOW B. BANERJEE D. TAMBOLI A. DIEPA NEW FELLOW A. BRADLEY V. MUSAT K. SHARIATMADAR NEW PHD NEW FELLOW SARAS N. BHARGAVA Uncertainty Surgical G. SINGH I. TEETI R. ALITAPPEH S. SAHA theory robotics S. KHAN A. SHAHBAZ LAB ADMIN ECM HUAWEI Activity detection I. SKARHA M. MOHAMED RL & IRL LEVERHULME The team D. BHARTI Federated learning G. ROSSI J. SAMUEL V. SINGH F. VELLA F. CUZZOLIN M. ROLF B. CIRSTEA A. MORELLI AI in finance SUPPONOR E. TSAMOURA S. OLIVASTRI J. BREWSTER KTP A. RAST W. BUCZYNSKI NEW ASSOCIATE Neurosymbolic Video captioning K. CANNONS V. LOMONACO C. LANGLEY B. SAHAKIAN reasoning Scene understanding Continual learning Machine theory of mind THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 17
Our research themes THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 18
Overview • Computer Vision (deep learning for action detection, video captioning, complex activities, future event prediction, scene understanding) • Machine Learning (continual learning, federated learning, self-supervised learning) • Artificial Intelligence (machine theory of mind, epistemic AI, neuro-symbolic reasoning) • AI for Healthcare (audio-visual monitoring of people in a coma, empathetic AI) • Surgical Robotics (SARAS, MAESTRO) • Autonomous Driving (Formula Student AI, ROAD) • Uncertainty Theory (random sets, belief functions, geometry of uncertainty) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 19
Computer vision THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 20
Real time human action detection • The Visual AI Lab is leader in the field, multi- year expertise, first system able to detect multiple human actions and events in real time • https://sahasuman.bitbucket.io/ • Won awards at both CVPR 17 Charades and ActivityNet 2016 action detection challenges • Based on deep neural networks [BMVC’16, BMVC’18, ICCV’17x2, ACCV’18] • Frame-level detections are linked up in real time to form ‘action tubes’ • Still very influential work in the field, how do we capitalise on this? Move into object detection proper? THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 21
Prediction of future events • We are now moving towards the early recognition of events and activities [ECCVW’18, ICCV’19] • Just by observing a small fraction of the video frames we can guess what is going on there • We aim to frame this in the context of modelling complex activities THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 22
Causal 3D CNNs • 3D CNNs (C3D, I3D, (2+1)D) have risen to the forefront of video classification • However, they work on entire videos as a whole in a batch manner • We recently proposed a Recurrent Convolutional Network designed to replace the temporal convolution component with a recurrent model in the temporal dimension • [Gurkirt Singh, ICCV’19 submission, PAMI in preparation] THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 23
From simple actions to complex activities • Topic: modelling complex road activities as graphs of ‘atomic’ actions/events • A deep network architecture is designed to detect those graphs of actions • When one is partially observed, we can guess what events may take place in the future and where • 3-year Huawei research agreement, EU H2020 SARAS surgical robotics project • 3D ROI pooling pipeline, graph GCN representation, extra annotation provided to the ROAD dataset THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 24
Dynamic heterogenous scene graphs • From complex activities to full dynamic scene representations • Adding scene elements (e.g. traffic lights, lamp posts, etc) and attributes to the graph modelling a dynamic scene • Approach: heterogenous graphs with nodes and edges of different types • Inference on heterogenous graphs • Dynamic HGs have not been proposed yet • Role of knowledge bases • Links to symbolic reasoning? THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 25
Video anomaly detection • Part of the SARAS project; research fellow Dinesh Jackson Samuel • Problem: identify any deviation from nominal behaviour in a video (in particular the endoscopic video of a surgical procedure) • Approach based on a new SVD loss in an architecture inspired by GANomaly THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 26
Video scene understanding (and inpainting) • Funded through the new KTP with Supponor, but also part of SARAS • The problem is do accurate video scene understanding from a streaming video, in a temporally-consistent fashion • We are exploring the boosting of features from the intermediate layers .. • .. as well as the use of scene graphs to • In the sports advertisement domain, we want to help the labelling .. identify and the commercial signage around the • .. and the notion of ‘soft’ segmentation to pitch and inpaint them with virtual content address boundary artifacts • Replaces Supponor’s current IR-based technology and active sensors THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 27
Surgical Robotics THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 28
SARAS – Smart Autonomous Robotic Assistant Surgeon • We are in charge of WorkPackage 6 (Situation awareness) • Online surgeon action detection • Procedure stage segmentation • Anomaly detection • We also contribute to WP5 (Perception) • Dynamic semantic understanding • Surgical tool tracking and pose estimation • WorkPackage 3 • Speech recognition module • Inna, Vivek, Mohamed and Dinesh • Extended to December 2021 THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 29
The SARAS Surgical Action Dataset (ESAD) • Dataset of 4 real life, four-hour- long videos capturing lapascopic surgery procedures • Annotation done with Microsoft VoTT • 21 action classes • Released for MIDL 2020 ESAD Challenge • https://saras-esad.grand- challenge.org/Home/ • MIA journal under major revision (Vivek) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 30
Multi-domain surgeon action detection (MESAD) • Builds on the MIDL 2020 challenge to launch a MICCAI 2021 challenge • Makes use of the 4 real videos and 6 videos captured using artificial anatomies (phantoms) as part of SARAS • Concept: exploring the opportunity of utilising cross-domain knowledge (in particular videos capturing procedures conducted on real as opposed to artificial organs and anatomies) to boost model performance on each individual task • Compound performance measure assessing accuracy and parameter sharing THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 31
MAESTRO Jr - Multi-sensing AI Environment for Surgical Task & • EPSRC THT grant Role Optimisation • Goal: to devise an AI- assisted operating room of the 2050s • Array of multiple sensors to monitor surgical checklists, teamwork • Monitoring of cognitive load • MORL for improving team performance • Led by George Mylonas at • Continual learning of perception and decision making models, both Imperial College supervised and unsupervised THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 32
Machine learning THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 33
Statistical learning theory • Problem: model adaptation in supervised learning • Statistical learning theory: computes generalisation bounds for the test error of models • Assumes training and testing distribution are the same! • Proposal: assume they come from a common convex set of probability distributions (now part of Epistemic AI FET project) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 34
Continual semi-supervised learning • Interesting work has been recently directed at continual learning from streaming data in a fully supervised setting • Focus mostly on avoiding catastrophic forgetting • Continual learning in a semi-supervised setting remains a wide-open research question • The problem can be reduced to a continual supervised learning setting under a 'multiple worlds' assumption • CSSL @ IJCAI 2021 workshop in August 2021 • Possible Horizon Europe collaborative project (see later) • CSSL @ IJCAI workshop paper • Possible follow-up agreement with Huawei • CAR and CCC continual learning benchmarks • IEEE PAMI paper in preparation THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 35
CAR and CCC benchmarks and challenges • Our CAR (Continual Activity Recognition) dataset is built upon the MEVA (Multiview Extended Video with Activities) dataset • Aim: benchmarking continual, long duration semi- supervised learning in a classification setting by classifying the input video frames in terms of activity classes • Each video frame is annotated in terms of 8 different activity classes selected from the 37 original classes (e.g. person_enters_scene_through_structure, person_exits_vehicle, vehicle_starts) + a bkground class • Continual Crowd Counting (CCC) dataset combines three existing crowd counting datasets (Mall, UCSD and FDST), for continual learning • Open challenges, move to Avalanche platform THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 36
Avalanche • An End-to-End Continual Learning PyTorch Library • Provides a shared and collaborative open-source codebase for fast prototyping, training and reproducible evaluation of continual learning algorithms • Initiative of ContinualAI • Avalanche can help Continual Learning researchers and practitioners in several ways: • Write less code, prototype faster & reduce errors • Improve reproducibility • Improve modularity and reusability • Increase code efficiency, scalability & portability • Augment impact and usability of your research products • Organised into 5 modules: Benchmarks, Training, Evaluation, Models, Logging SHORT TITLE PROF FABIO CUZZOLIN DATE 37
Federated learning • Concept: different parties can work together on better models without having to share the data • Best suited to healthcare domain: hospitals working to achieve better diagnosis from x-ray images without disclosing patient data • PhD student Devashish Bharti • Ongoing collaboration with Converz, pending Innovate UK grant (https://www.converz.co.uk/) • Possible approaches: meta-learning • Devashish is working on a benchmark and performance measures for the problem THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 38
Artificial Intelligence THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 39
Machine theory of mind • Humans have complex mental lives, which make them unpredictable • E.g.: children walking on the pavement spot an ice cream van on the other side – are they gonna cross? • Solution: giving machines theory of mind abilities, i.e. the ability to guess other agents’ reasoning process • https://www.facebook.com/oxfordbrookes/videos/101 56698398637908/ • Idea: reinforcement learning of ‘accurate’ simulations of people’s mental reasoning • [Leverhulme Grant with Cambridge Neuroscience, 2020-2024] • Bogdan-Ionut Cirstea, Barbara Sahakian and Christelle Langley (Cambridge) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 40
Neuro-symbolic AI • Integration of learning and reasoning is one of the elements of a future artificial intelligence • We have started collaborating on this topic with Samsung AI Cambridge (Efi Tsamoura) • Various lines of research: • Loss functions expressing logical constraints on the output of a nettwork • Logical theories processing the output of a NN • From probabilistic logic to epistemic logic (Leverhulme Grant) • Appeal: possibility of transferring knowledge across disparate domains • Devising networks that generate propositional solutions to problems (e.g. scene understanding) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 41
ROAD-R: learning with requirements • AAAI 2022 paper led by Eleonora Giunchiglia (Oxford) and Salman Khan • Idea: assess how often neural network predictions violate common-sense requirements • Introduces the Autonomous Driving Dataset for Learning with Requirements (ROAD-R) • In ROADR, the requirements express conditions on the set of admissible road events, and are formulated as constraints • We show that current SOTA models produce non-admissible predictions on ROAD-R • We propose two baseline methods for computing a minimal set of corrections THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 42
Neural-guided projection • NeurIPS 2021 submission led by Efi Tsamoura (Samsung AI, Oxford) • Example of training-time regularization (TTR) in stratified neural symbolic architectures • Proposes a neural-guided projection (NGP) procedure that identifies a small subset of constraints whose theory-prescribed values are maximally violated under the neural predictions • Solves an optimization problem in which the neural module is updated to minimize the maximum “unsatisfiability" of the constraints • Scalable, modular, outperforms all state of the art when applied to scene graph generation • Can be embedded in the neurosymbolic NeuroLog framework SHORT TITLE PROF FABIO CUZZOLIN DATE 43
Epistemic artificial intelligence • Horizon 2020 FET (Future Emerging Technologies) project, coordinated by us • 3M euros over 4 years, with KU Leuven and TU Delft • Bottom line: uncertainty is not well modelled in AI, and that makes AI models brittle • Change of paradigm: from striving for ‘generalisation’ or ‘adaptation’ to working with convex sets of models • Use the available information to reduce the uncertainty about the ‘true’ model • Well suited to the continual learning paradigm • Contributions to supervised, unsupervised and reinforcement learning THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 44
Epistemic AI • What are the main targets of the Epistemic AI programme? • Theory of constrained optimisation under epistemic uncertainty • Losses for random sets that generalise cross entropy in deep learning • GANs (generative adversarial networks) rely on stochastic sampling from a latent variable • Design robust GAN framework based on random set theory • Reinforcement learning also relies on the framework of (Partially Observable) Markov Decision Models • Generalisations of MDPs and Markov chains exist • Can we lay down the foundations of a ‘robust’ reinforcement learning framework? • Generalising Belmann’s equations THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 45
Autonomous driving THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 46
AI for Autonomous Driving • Can we predict future behaviour using machine theory of mind? • (1) future behaviour is guessed • (2) by explaining the chain of reasoning • AI Research Fellow (Aytac Kanaci) and PhD student Izzeddin Teeti • Collaboration with Autonomous Driving group THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 47
ROAD – The ROad event Awareness Dataset for Autonomous Driving • We created the world's first ROad event Awareness Dataset for Autonomous Driving (ROAD) • Annotating a number of videos from the existing Oxford RobotCar Dataset • 22 videos, 122K frames, 560K bounding boxes, 1.7M labels • Multi-label dataset: agent (11 classes), action (23), location (15) • Aim: test prediction capabilities, complex road activities, situation awareness • [IEEE PAMI, ROAD @ ICCV’21 workshop] THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 48
Formula Student - AI • Collaboration with the Autonomous Tech student society and the Autonomous Driving group (Andy Bradley, Petar Georgiev) • https://www.imeche.org/events/formula- student/team-information/fs-ai • FS-AI has been introduced to challenge student teams to develop an AI driver capable of controlling a purpose designed racing car • Also ‘static’ event categories, e.g. students to also consider Real World Autonomous scenarios • Our brand new Autonomous Formula Student team participated in the first edition of the UK Formula Student AI competition, coming 3rd place in 2019, 1st overall in 2020 and 2nd in 2021! THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 49
Video synthesis for model adaptation • Led by final year student Valentina Musat • Inspired by Vid2vid framework (https://arxiv.org/pdf/1808.06601) • GAN (Generative Adversarial Network) – based approach, need to enforce temporal consistency • Goal is be able to synthesise realistic videos for training autonomous cars • Leading to a weather augmentation paper to submit THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 50
Scale-invariant object detectors • Robust object detectors with respect to scale • Object-tube approach to detection • Topic of final-year project by Aduen Benjumea Diepa • Based on the SNIPER framework • Instead of processing every pixel in an image pyramid, SNIPER processes context regions around ground-truth instances (referred to as chips) at the appropriate scale • B. Singh, M. Najibi and L. S. Davis. SNIPER: Efficient Multi-Scale Training. NeurIPS 2018 • Application to autonomous driving and F Student AI • ROAD @ ICCV workshop paper THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 51
AI for healthcare THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 52
Multimodal monitoring of people in a coma • Automated monitoring of people with prolonged disorder of consciousness • Past EPSRC bid with Prof Helen Dawes and Prof Derick Wade (MOReS), in process of rewriting • Core: modelling and detection of audio-visual events using multimodal deep learning THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 53
Empathetic AI • Problem: communicating with people with cognitive impairment in care homes • Idea: bespoke machine theory of mind approach • Understanding both physical and emotional needs • Machines willing to help? • Linked to the Leverhulme grant • Bid in preparation for EPSRC Healthcare Technologies call • Partners: Derick Wade, Helen Dawes and Bogdan Cirstea THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 54
Uncertainty theory THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 55
Geometry of uncertainty • Probability theory is not the only mathematical theory of uncertainty • Others exists: imprecise probabilities, random sets, belief functions are the most popular • Uncertainty measures can be represented as points of a convex space, and there analysed • I wrote two books on the topic, one coming out with Springer - Artificial Intelligence: Foundations, Theory, and Algorithms • Past work on the geometry of belief functions, fuzzy sets, combination rules, conditioning THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 56
The geometry of uncertainty – The geometry of imprecise probabilities • My monograph on the geometry of uncertainty measures is finally out (after 9 years!) • 860 pages! • Most complete compendium of belief function theory • Most complete overview of the general landscape of uncertainty theory • Compendium of my geometric approach • Agenda for the future of random set theory THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 57
Generalised statistical inference • How to infer a belief function/random set from sample data? • This is the inference problem • The two main existing mechanisms in belief theory rely on classical probabilistic inference • Concept: generalise maximum likelihood inference in order to infer directly a belief function from data • Further step: generalise logistic regression • Can be applied to estimation of rare events THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 58
Our partners THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 59
Institute for Ethical AI • Created by Prof Nigel Crook • https://ethical-ai.ac.uk/ • Co-director Kevin Maynard • Fabio is in Board • Aims at establishing and maintaining a cross- university network on the topic • Themes: explainability, ethical implications of AI, ethical AI from a technical point of view, advising • A network of interested companies has been set up THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 60
Healthy Ageing network • Cross-university network involving Business, Health Sciences, Architecture, and us • Coordinated by Juliet Bligh of RBDO • Topics: sustaining physical activity, living well with cognitive impairment, design for age-friendly homes, etcetera • Goal: stimulate new ideas from a wide range of businesses and social enterprises to develop and deliver products, services and business models at a large scale to support people as they age THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 61
OxCATTS • Oxford Clinical Allied Technology and Trial Services Unit • https://oxcatts.com/ • Funded by Helen Dawes • Fabio is in the Steering Committee • Aim: increase cross-fertilisation between our work in AI and in healthcare/rehabilitation • Mission is to provide consultancy and support research grant applications THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 62
Within the School and University Cognitive Robotics (Prof Crook and Dr Rolf) MOReS (Prof Dawes, Wade; Dr Esser) Autonomous Driving group (Andrew Bradley) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 63
Within the UK Torr Vision Group, Oxford University (Prof Torr) Computer Science Dept, Oxford University Neuroscience, Cambridge University (Prof Sahakian) Imperial College London (Dr Mylonas) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 64
Across the world Altair Robotics Lab, Verona (Prof Fiorini, Dr Muradore) Federico II University, Naples (Prof Di Gironimo) Indian Institute of Technology, Bombay (Prof Banerjee and Chaudhuri ) KU Leuven, Belgium (Dr Shariatmadar) TU Delft, Netherlands (Prof Yorke-Smith) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 65
Companies Huawei Technologies (Dr Zhan Xu) Cisco Systems (Dr William Wu) Createc (Matt Mellor, Ian Cowling) Supponor (Andrew Crean, Steve Plunkett) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 66
Companies AI Labs, Bologna (Massimo Ciociola) Samsung AI (Efi Tsamoura) Peoplespace (Halbyn Rich) Ocado (Dr Graham Deacon) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 67
Societies and networks • ContinualAI (Vincenzo Lomonaco): https://www.continualai.org/ • CLAIRE: European federation of AI laboratories (https://claire-ai.org/ ) • The Society for Imprecise Probability (http://www.sipta.org/) • The Belief Functions and Applications Society (https://bfasociety.org/) THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 68
The next steps THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 69
Upcoming grant applications • A number of grant applications are in the making • €10M Horizon Europe - MAESTRO - Multi-sensing AI Environment for Surgical Task & Role Optimisation (led by Imperial) • €4-5M SARAS++ - follow up of SARAS • Horizon Europe on Adaptive Machines (with ContinualAI and others) • £1.5M EPSRC bid on Continual epistemic learning (natural prosecution of E-pi) • £1.5M EPSRC Healthcare Technologies on Theory of mind for the cognitively impaired (CoIs Dawes and Wade) • £350K Leverhulme RPG on neurosymbolic reasoning (led by Samsung AI) • Other high-impact areas to seek funding for: self-supervised learning, goal-changing machines, epistemic modelling of pandemics THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 70
Project Olympia • The world’s first AI coaching app • Powered by our action detection technologies • Negotiating initial venture capital for £8,000,000, dozens of researchers and engineers • Co-founders: Pascal Pert, EIT, Chairman of innovation and marketing at World Taekwondo Europe • Marc Faubeau • Numerous gold medallists and testimonials and influencers THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 71
Project Olympia • Various platforms, based on our action detection know how • Platform 1: Compilation of simple stats on no of kicks, jumps, … • Platform 2: Assisted video review of significant events • Platform 3: Video-based assessment of quality of exercise • Platform 4: RL-based personalised recommendations for improvement • Will be based in the Wood Centre for Innovation, Oxford (?) • Initial prototype and demo THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 72
SARAS++ • Follow-up from the SARAS project on surgical robotics • Big industrial partners like Philips, Medtronic (the largest supplier of medical electronics), Karl Storz (the leading endoscope manufacturer) • Budget: TBD, but in excess of 5M • Looking for appropriate Horizon Europe call • Probably also led by Verona? THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 73
MAESTRO • Follow up to MAESTRO Jr, aims to deliver the smart operating room of the future • Big industrial partners including Johnson & Johnson; Again looking for new Horizon Europe calls • Coordinated by Imperial College (Mylonas); Budget: ca 10M euros THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 74
Theory of self-supervised learning • Self-supervised learning empowers us to exploit the variety of labels that usual come with the data for free – leading to ‘human-level’ intelligence?? • Technically, the idea is to define an auxiliary task for which we already have labels, 'hidden' within the structure of the data itself • This happens by defining a self-supervised task, also known as pretext task, which guides us to a supervised loss function • However, no theoretical foundations for self- supervised learning yet exist • Our goal is to provide a theoretical justification for self-supervised learning through a combination of functional analysis and optimisation theory THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 75
Goal-changing machines • That’s fine, but what about the ability of people to change their goal(s) in time, or juggle various different goals at the same time? • Seeking models that update their very objective function with time • Interesting discussion with Prof Gary Browning • Again, strong intuition and insights can come from psychology and neuroscience • https://chrisguillebeau.com/how-goals- change-over-time/ THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 76
Our targets THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 77
Targets • Increasing output in terms of publications -> ideally we should be in a position to submit 10 papers a year to major computer vision and AI conferences • Boosting our citations figures -> investing in frontier topics with high potential impact • Target top venues such as Nature (as we did for our ToM editorial) • Further doubling the research funding to reach £5-6M • Research fellows should start engaging with grant applications • Attracting venture capital and generating spin-offs, starting from the Olympia project -> another option is applying our ToM concept to autonomous driving • Continue organising workshops and challenges, releasing new benchmark datasets (e.g. ROAD, ESAD and the continual learning ones) • In the medium term, the creation of an Oxford Institute for Next-generation AI THE VISUAL AI LAB IN 2022 PROF FABIO CUZZOLIN 22/09/2021 78
You can also read