DIG THAT LICK: EXPLORING PATTERNS IN JAZZ SOLOS - UIO

Dig that Lick:
                  Exploring Patterns in Jazz Solos

 Simon Dixon1 , Polina Proutskova1 , Tillman Weyde2 , Daniel Wolff2 ,
   Martin Pfleiderer3 , Klaus Frieler3 , Frank Höger3 , Hélène-Camille
 Crayencour4 , Jordan Smith1,4 , Geoffroy Peeters5 , Doğaç Başaran6 ,
    Gabriel Solis7 , Lucas Henry7 , Krin Gabbard8 , Andrew Vogel8

  (1) Queen Mary University of London; (2) City, University of London; (3) University of Music
Weimar; (4) CNRS, IRCAM Lab, Sorbonne Université; (5) Telecom ParisTech; (6) Audible Magic;
                      (7) University of Illinois; (8) Columbia University

                      Mirage Symposium, June 8-9, 2021

1
                                    Dixon et al.   Dig that Lick                             1 / 14

The Dig that Lick Project (2017-2019)

   Full title: Dig that lick: Analysing large-scale data for melodic
   patterns in jazz performances
   Enhance existing infrastructures for the deployment of semantic
   audio analyses over large collections
   Facilitate access to large audio and metadata collections via
   interfaces for content selection, semantic analysis, and aggregation
   Use the developed infrastructure to analyse the use of melodic
   patterns in a large jazz corpus of monophonic solos
   Relate analytic results to background knowledge to trace and
   interpret musical influence across time, space, cultures and
   societies
   Convince musicologists (!)

                            Dixon et al.   Dig that Lick               2 / 14

Data: Audio and Metadata

                                Discographies
   Data                             Up to 70 000
                                      sessions

    Audio Datasets                                                  Linked
                                                                   Open Data
     U.Columbia
                                                                     LinkedJazz
      ~10 000
      tracks            Jazz                                                      VIAF
                     Encyclopedia                           Smithsonian
                      ~10 000
                      tracks
      U.Illinois
                                                                    LoC
                                                                            Wikipedia
      ~30 000
      tracks                                                         9 000 musicians
                                                                      + relationships

                             Dixon et al.   Dig that Lick                                3 / 14

Metadata Ontology for Jazz

                    Dixon et al.   Dig that Lick   4 / 14

(Automatic) Metadata Cleaning

   Named Entity Resolution
       Charlie Parker, Charley Parker, Чарли Паркер, Charlie “Bird” Parker,
       Charlie Parker Quartet, Charlie Parker Quintet, Charlie Parker All Stars
       b, el-b, synt-b, fretless-b, string-b, el-fretless-b, fretless-el-b,
       keyboard-b, amplified-b, bass
   Reconciliation:
       Louis Armstrong (1901-1971) = Louis Armstrong (1900-1971)
   Disambiguation
       Bill Evans (p) ̸= Bill Evans (ss)
       Camden, on: Adam Birnbaum, Travels (Smalls Records SRCD-0036)
       ̸=
        Camden, on: Rodney Green Quartet, Live At Smalls (SmallsLIVE
        SL0036)

                            Dixon et al.   Dig that Lick                    5 / 14

Audio Processing: Automatic Melody Extraction

   Task: estimate the notes of the main melody from the complex
   mixture of melody and accompaniment
   Our approach uses advanced AI and signal processing techniques
   Stage 1: Compute a pitch salience representation: using a
   convolutional neural network (CNN) with source-filter non-negative
   matrix factorisation pretraining
   Stage 2: Exploit temporal information to track pitch over time:
   using a recurrent neural network (RNN)
   Results: generally successful, with some missed and extra notes,
   octave errors and semitone errors
   Example: Original:        Estimated:                  Both:

                          Dixon et al.   Dig that Lick                6 / 14

Pattern Extraction

   Importance of patterns to jazz is well evidenced
   Patterns in pitch (absolute or relative), time (absolute durations or
   relative to metre), or both
   We focus on pitch, expressed as n-grams
   Selection criteria: played multiple times, in multiple tracks, by
   multiple people
   Levenshtein (edit) distance used for exact or inexact matching

                            Dixon et al.   Dig that Lick                   7 / 14

DTL1000 Dataset

   1060 tracks selected randomly (100+ per decade from 1920-2019)
   Manual segmentation and labelling of solo instrument (player)
   Note tracks automatically extracted from monophonic solos
       1700 solos, 6M pitch n-gram instances, 5.6M interval n-grams
   Metadata (tune, band, musician, instrument, date, location, etc.)
       Linked with our semantic model
       Can be used to filter searches
       Displayed with results
   Similarity search combining DTL1000 with other datasets
       Weimar Jazz Database
       Charlie Parker Omnibook
       Essen Folk Song Collection

                           Dixon et al.   Dig that Lick                8 / 14

Pattern Search: List Results

                      Dixon et al.   Dig that Lick   9 / 14

Pattern Similarity Search: Timeline Results

                      Dixon et al.   Dig that Lick   10 / 14

Pattern Similarity Search: Graphical Results

                      Dixon et al.   Dig that Lick   11 / 14

Conclusions

   Data and interfaces for exploring melodic patterns in jazz solos
       Multiple data types (human and automatic transcriptions, collections)
       Audio and symbolic data
       Metadata filters to constrain cultural context
   Challenges: data coverage and reliability
       Limited availability of data, especially contextual metadata
       Current methods only address monophonic instruments
       Automatic transcription and metadata processing are error-prone
   Useful tools for case studies
       To discover and trace the history of patterns
       To investigate how jazz musicians draw on each other
       To make inferences about influence of race, class, and gender

                            Dixon et al.   Dig that Lick                 12 / 14

Publications and Presentations
   Başaran, D., Essid, S., and Peeters, G. (2018).
   Main melody estimation with source-filter NMF and CRNN.
   In 19th International Society for Music Information Retrieval Conference, pages 82–89.

   Frieler, K. (2019).
   Constructing jazz lines: Taxonomy, vocabulary, grammar.
   In M. Pfleiderer, W.-G. Z., editor, Jazzforschung heute: Themen, Methoden, Perspektiven, pages 103–132. Edition EMVAS,
   Berlin.
   Frieler, K., Başaran, D., Höger, F., Crayencour, H.-C., Peeters, G., and Dixon, S. (2019a).
   Don’t hide in the frames: Note- and pattern-based evaluation of automated melody extraction algorithms.
   In 6th International Conference on Digital Libraries for Musicology, pages 25–32.

   Frieler, K., Höger, F., and Pfleiderer, M. (2019b).
   Anatomy of a lick: Structure and variants, history and transmission.
   In Book of Abstracts of the Digital Humanities Conference.

   Frieler, K., Höger, F., and Pfleiderer, M. (2019c).
   Towards a history of melodic patterns in jazz performance.
   In 6th Rhythm Changes Conference.

   Frieler, K., Höger, F., Pfleiderer, M., and Dixon, S. (2018).
   Two web applications for exploring melodic patterns in jazz solos.
   In 19th International Society for Music Information Retrieval Conference, pages 777–783.

   Gabbard, K. (2019).
   What we are digging out of the data?
   In 6th Rhythm Changes Conference.

   Höger, F., Frieler, K., Pfleiderer, M., and Dixon, S. (2019).
   Dig that lick: Exploring melodic patterns in jazz improvisation.
   In 20th International Society for Music Information Retrieval Conference: Late Breaking Demo.

   Solis, G. and Henry, L. (2019).              Dixon et al.      Dig that Lick                                       13 / 14

Acknowledgements

This research was funded under the Trans-Atlantic Program Digging into
Data Challenge with the support of the UK Economic and Social
Research Council (ES/R004005/1), the French National Research
Agency (ANR-16-DATA-0005), the German Research Foundation (PF
669/9-1), and the US National Endowment for the Humanities
(NEH-HJ-253587-17).

                           Dixon et al.   Dig that Lick            14 / 14