(Apologies for cross-postings)
MIREX 2010 Call for Evaluation Task Proposals
The International Music Information Retrieval Systems Evaluation
Laboratory (IMIRSEL) team at the University of Illinois at
Urbana-Champaign, is soliciting proposals for evaluation tasks to be
performed as part of the 2010 Music Information Retrieval Evaluation
eXchange (MIREX 2010). Potential participants are invited to submit
proposals for novel evaluation tasks, the re-running of past evaluation
tasks and additional datasets and evaluation metrics to used to run tasks.
Since 2005, MIREX has conducted 760 evaluation runs over a wide range of
MIR tasks. Evaluation tasks from past MIREX evaluations include:
- Audio Test/Train Tasks
- Audio Artist Identification
- Audio Genre Classification
- Audio Music Mood Classification
- Audio Classical Composer Identification
- Audio Onset Detection
- Audio Key Detection
- Audio Tag Classification
- Audio Cover Song Identification
- Real-time Audio to Score Alignment (a.k.a Score Following)
- Query-by-Singing/Humming
- Multiple Fundamental Frequency Estimation & Tracking
- Audio Chord Estimation
- Audio Melody Extraction
- Query-by-Tapping
- Audio Beat Tracking
- Audio Music Similarity and Retrieval
- Structural Segmentation
So that the MIREX team can gauge the level of interest in each task an
online poll has been setup. If you are considering participating in
MIREX 2010 please indicate your likelihood of participation at:
https://spreadsheets.google.com/viewform?formkey=dHNtUjkxQ1M2Q0dQa293cC02TnM2OFE6MA
Please use the MIREX organisation mailing list (EvalFest) for discussion
of MIREX task proposals and other MIREX related issues:
https://mail.lis.illinois.edu/mailman/listinfo/evalfest
The MIREX 2010 wiki will be used to embody and disseminate task
proposals and can be found at:
http://www.music-ir.org/mirex/2010/index.php/Main_Page
Where possible, task related discussions, which should be conducted on
the MIREX organization mailing list (EvalFest), should be summarized on
the appropriate MIREX 2010 wiki pages.
Where possible, definitions or example code for new evaluation metrics
or tasks should be provided to the IMIRSEL team who will embody them in
software as part of the NEMA analytics framework, which will be released
to the community at or before ISMIR 2010 - providing a standardised set
of interfaces and output to disciplined evaluation procedures for a
great many MIR tasks.
Projected Target Dates:
- 1st May 2010: MIREX submission system opens (somewhat hopeful
target date)
- 15th June - 1st July 2010: Rolling MIREX submission system closures
(dates to be announced)
- 15th July 2010: MIREX results posting begins
- 1st August 2010: All MIREX results posted (somewhat hopeful target
date)
- 2-6th August 2010: USMIR Summer School
(http://ismir2010.ismir.net/usmir-summer-school/)
- 9-13th August 2010: ISMIR 2010 Conference
(http://ismir2010.ismir.net/)
Finally, a web-accessible submission and execution system is to be
provided for MIREX by the NEMA project (http://nema.lis.illinois.edu/).
This prototype "MIREX DIY" system will allow users to very simply
submit, configure, test, execute and debug their own submissions and to
inspect or publish their results. As this system is still in its infancy
it may (initially) only be available for selected tasks and participants
at MIREX 2010. Hence, we would like to invite institutions likely to
produce multiple submissions to MIREX 2010 to contact the IMIRSEL team
at [log in to unmask] and to nominate an individual from their
institution to be tutored on the use of the NEMA service and to manage
their institutions submissions.
Looking forward to the best MIREX yet.
Cheers,
J. Stephen Downie on behalf of the MIREX team.
--
**********************************************************
"Research funding makes the world a better place"
**********************************************************
J. Stephen Downie, PhD
Associate Professor,
Graduate School of Library and Information Science; and,
Center Affiliate, National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
[Vox/Voicemail] (217) 649-3839
NEMA Project Home: http://nema.lis.uiuc.edu
