SIGNLL (ACL's Special Interest Group on Natural Language Learning) invites proposals for the CoNLL Shared Task 2025.
Background
Since 1999, CoNLL (the Conference on Computational Natural Language Learning) has regularly included a shared task in which training and test data are provided by the organizers, which allows participating systems to be evaluated and compared in a systematic way. Descriptions of the participating systems and an evaluation of their performance are presented both at the conference and in the proceedings.
Previous shared tasks
Previous shared tasks have been devoted to:
- Chunking (1999, 2000)
- Clause identifications (2001)
- Named entity recognition (2002, 2003)
- Semantic role labeling (2004, 2005)
- Dependency parsing (2006, 2007)
- Joint parsing of syntactic and semantic dependencies (2008, 2009)
- Hedge detection (2010)
- Co-reference resolution (2011, 2012)
- Grammatical error correction (2013, 2014)
- Universal morphological reinflection (2017, 2018)
- Multilingual parsing from raw text to universal dependencies (2017, 2018)
- Cross-framework Meaning Representation Parsing (2019, 2020)
- BabyLM Challenge (2023)
Submission guidelines
For CoNLL-2025 proposals are invited for any shared task involving natural language learning and processing, that will directly contribute to <b>theoretically, cognitively and scientifically motivated approaches to computational linguistics </b> (rather than driven by particular engineering applications).
Proposals should contain the following information:
- Names and affiliations of the members of the organizing committee (including a designated contact person).
- Description of the shared task, including its relevance to theoretically, cognitively and scientifically motivated approaches to computational linguistics.
- Description of the process for selecting, acquiring and preparing the data for training and evaluation, including conditions of availability
- Description of the procedure for evaluating the performance of participating systems, including the availability of evaluation software.
- Description of any ethical or licensing issues that may arise from the organization and participation in the proposed task.
- Any other relevant information.
The exact timeline of the shared task will be determined according to the venue CoNLL will be collocated with (either NAACL, ACL or EMNLP; aiming for ACL). Proposals should be submitted by October 1, 2024. Proposals should be submitted by e-mail to proposals@signll.org, from the email of the contact person, with the subject āCoNLL-2025 Shared Task Proposalā.
A tentative timeline for the CoNLL Shared Task 2025 is as follows:
- October 1, 2024: deadline for proposals
- October 15, 2024: confirmation of acceptance
- December 15, 2024: first version of task website online
- December 15, 2024: first call for papers out
- January 30, 2019: all trial datasets (plus documentation and scorer) available
- February 15, 2025: test data available
- April 15, 2025: test phase ends
- April 22, 2025: submission of system description papers
Proposals should make clear that a similar schedule can be implemented, especially for the preparation of resources (trial/training/development/test data sets).
Proposals should be 2 to 3 pages long, in PDF format. We encourage proposers to include appendices with examples that help understanding the data and type of evaluation.
Proposals will be reviewed by the SIGNLL steering committee. Proposers might be contacted during the reviewing period to provide further information. Potential proposers should feel free to indicate the intention to submit by emailing proposals@signll.org. We are also happy to answer your questions about the process, or to solicit early feedback for a proposal.