Call for Lab Proposals

CLEF 2022: Conference and Labs of the Evaluation Forum

Conference and Labs
of the Evaluation Forum

Bologna, Italy, September 5-8, 2022

 

 

We invite researchers and practitioners from all areas of information access as well as closely related fields to submit proposals for the organization of an Evaluation Lab (a.k.a. “Shared Task”) as part of CLEF 2022.

The CLEF conference will be hosted by Università di Bologna on September 5-8, 2022, in Bologna, Italy, and accepted labs will be organized within the twelve months beforehand.

 

Background

CLEF is a leading annual international conference organized by the CLEF Initiative, which builds on a twenty-three-year history of successful and diverse labs exploring the evaluation of information access systems.


Its main mission is to promote research, innovation, and development of information access systems by providing an infrastructure for:

  • - independent evaluation of information access systems;
  • - investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access;
  • - creation of reusable test collections for benchmarking;
  • - exploration of new evaluation methodologies and innovative ways of using experimental data;
  • - discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge.

In particular, CLEF welcomes labs studying low-resource languages, as well as multilingual and multimodal information access, albeit, this is not a mandatory requirement for a successful proposal.

 

Scope of CLEF Labs

Proposals are accepted for two different types of labs:

-   Shared-task-style labs. Labs of this type implement an evaluation of specific information-access tasks, adopting the best practices of shared tasks. A lab can consist of multiple different tasks.

 

-   Workshop-style labs. Labs of this type organize a workshop with the goal of exploring issues of evaluation methodology, metrics, processes, etc., or, as the first step toward creating new Shared-task-style labs.

 

CLEF is interested in receiving and facilitating innovative lab proposals. Potential lab proposers who are unsure of the suitability of their lab proposal or its format for inclusion at CLEF are encouraged to contact the CLEF 2022 Lab Organizing Committee Chairs to discuss its suitability or design at an early stage, as well as participate in the lab mentorship programme.

Proposal Submission

Lab proposals must provide sufficient information to judge the relevance, timeliness, scientific quality, benefits for the research community, and the competence of the proposers to coordinate the lab. Each lab proposal should identify one or more organisers as responsible for ensuring the timely execution of the lab. Proposals should be 3 to 4 pages long and should provide the following information:

  1. Title of the proposed lab.
  2. The planned format of the lab; i.e., shared-task-style or workshop-style.
  3. A brief description of the lab topic and goals, its relevance to CLEF, and the significance for the field.
  4. A brief but clear statement of usage scenarios or domain to which the activity is intended to contribute. For shared-task-style labs, include the evaluation setup and metrics. For workshop-style labs, describe the planned outputs of the workshop.
  5. Details on the lab organiser(s), including (for shared-task-style labs) identifying the task chair(s) responsible for ensuring the running of the task(s). This should include details of any previous involvement in organising or participating in similar evaluation activities at CLEF or similar venues.
  6. Is the lab a continuation of an activity from previous year(s) or a new activity?
    1. For activities continued from previous year(s): Statistics from previous years (number of participants/runs for each task), a clear statement on why another edition is needed, an explicit listing of the changes proposed, and a discussion of lessons to be learned or insights to be made.
    2. For new activities: A statement on why a new lab is needed and how the community would benefit from the activity.
  7. Details of the expected target audience; i.e. who do you expect to participate in the task(s), and how do you propose to reach them.
  8. Shared-task-style labs only: Brief details of tasks to be carried out in the lab. The proposal should clearly motivate the need for each of the proposed tasks and provide evidence of its capability of attracting sufficient participation. The dataset(s) forming the basis of each task needs to be described and motivated in the perspective of the goals of the lab; also providing indications on how the dataset(s) will be shared is useful. It is fine for a lab to have a single task, but labs often contain multiple closely related tasks. Having more than 3 tasks needs a strong motivation to avoid fragmentation.
  9. Expected length of the lab session at the CLEF conference: half-day, one day, two days. This should include very high-level details of the planned structure of the session; e.g., participant presentations, invited speaker(s), panels, etc., to justify the requested session length. For workshop-style labs, indicate methods used to foster discussion, encourage audience participation, and gather information.
  10. Arrangements and other relevant information for the organization of the lab. E.g., for shared-task-style labs, who will be responsible for activities within the task; how will data be acquired or created, what tools or methods will be used; how will necessary queries be created or relevance assessment carried out?
  11. If the lab proposes to set up a steering committee to oversee and advise its activities, include names, addresses, and homepage links of people you propose to be involved.

The CLEF 2022 program will reserve about half of the conference schedule for lab sessions. The labs will present their overall results in "overview presentations" during the plenary scientific paper sessions to allow non-participants in the lab to get a sense of where the research frontiers are moving.

Important considerations:

  • - Organisers new to CLEF, or shared-tasks in general, are highly encouraged to first organise a workshop-style lab to discuss the format, the problem space, and the practicalities of their envisioned future shared task.
  • - Organisers of each lab are expected to chair sessions for their lab at the conference.
  • - Within these sessions, ample time should be given for general discussion and engagement by all participants---not just those presenting results and papers. For instance, this may include keynotes, panels, demos, poster sessions, discussion rounds, etc., as appropriate.
  • - Organizers of accepted labs are expected to advertise their labs at both CLEF 2021 (​21-24 September 2021, Bucharest, Romania) and ECIR 2022 (10-14 April 2022, Stavanger, Norway). At least one lab representative should participate in these events.
  • - Advertising at CLEF 2021 will consist of running a break-out session to discuss the lab with prospective participants, and advertising/announcing it during the closing session.
  • - Advertising at ECIR 2022 will consist of submitting a short lab description to be included in the ECIR 2022 proceedings (submission by October 7, 2021), advertising the lab in a booster session during ECIR 2022, and presenting a poster about the lab.

Lab proposals have to be submitted at the following address:
https://easychair.org/conferences/?conf=clef2022
choosing the “​Lab Proposals​” track.

Examples of previous labs can be found at
http://clef2021.clef-initiative.eu/index.php?page=Pages/labs.html

Reviewing Process

Each submitted proposal will be reviewed by the CLEF 2021 Lab Organizing Committee. The decision will be sent by email to the responsible organiser by July 29, 2021. The final length of the lab session will be determined based on the overall organization of the conference and the number of participant submissions received by a lab.

Mentorship Program for Lab Proposals from Newcomers

As in previous editions, CLEF 2022 will run a mentorship program to support the preparation of lab proposals for newcomers to CLEF. We encourage newcomers to refer to Friedberg et al. (2015) for initial guidance on preparing their proposal: ​Friedberg I, Wass MN, Mooney SD, Radivojac P. Ten simple rules for a community computational challenge. PLoS Comput Biol. 2015 Apr 23;11(4):e1004150.

The CLEF newcomers mentoring program offers help, guidance, and feedback on writing the proposal draft by assigning a mentor, helping with preparing and maturing the proposal for submission. If your proposal falls into the scope of an already existing CLEF lab, the mentor will make contact with the respective lab organisers so that the lab’s teams can join forces.

Lab proposals for mentorship have to be submitted at the following address:
https://easychair.org/conferences/?conf=clef2022
choosing the "Lab Mentorship" track.

Important Dates

- Requests for mentorship submission (only newcomers):10 May 2021
- Mentorship period starts:10 May 2021
- Lab proposals submission (newcomers and veterans):1 July 2021
5 July 2021
- Notification of lab acceptance:29 July 2021
3 August 2021
- Advertising labs at CLEF 2021, Bucharest, Romania:21-24 Sep 2021
- Submission of short lab description for ECIR 2022: 7 October 2021
- Lab registration opens:1 December 2021
- Advertising labs at ECIR 2022, Stavenger, Norway:10-14 April 2021

CLEF 2022 Lab Chairs

  • Allan Hanbury, TU Wien, Austria
  • Martin Potthast, Leipzig University, Germany

CLEF 2022 Lab Mentorship Chair

  • Paolo Rosso, Universitat Politècnica de València, Spain