Ever wondered how WebCamp talks are selected? This article outlines this evolving process as it stands now.
Last year we formed the talk selection committee. The idea was to have a dedicated team of professionals from the community whose job is to review all the talks submitted to our call for papers and select the best ones to appear on stage that year.
It was a big success. The core team was offloaded to work on organizational aspects of the conference and the committee focused on selecting the talks. While it worked well, the team lacked clear guidelines of how to rate and select the talks, what were the exact criteria and goals. Because of this, we spent a lot of time discussing and coordinating.
To address this issue, we sat down and wrote down the guidelines we came up with last year, refined it and called it Talks committee guidelines v1.0. Following is the whole text.
Talk committee guidelines
Version 1.0, May 2018
About this document
This document provides the overview of the talks selection process, and provides some advice for rating talk submissions. The document is informal in its nature, so treat it more as a set of guidelines, rather than a bunch of must-follow rules. The talks committee is a small team, and it's assumed that committee members are all sensible people who can reach a consensus without needing a large set of bureaucratic rules.
Input from committee members is highly encouraged. If you find anything you disagree with, or you think some additional points are worth including, feel free to submit your proposals.
As the talks committee, our task is to select the talks for the conference program. We want the conference attendees to have the great experience, and walk away from the conference with good impressions, and many new insights. To make that happen, we aim to reach the following goals:
- Create an interesting program which will educate and inspire people.
- Provide a balanced selection of topics for designers, frontend, and backend developers.
- Have a diverse lineup of speakers of different genders, nations, and experience levels.
In the first part of the process each member of the committee rates the submissions. Preferably, this rating is done in isolation, without discussing submissions with other members. Then, about a week after the CFP is closed, the committee and the members of the core team will meet, and select the final lineup.
During the meeting, we'll first compare individual ratings, and discuss the submissions with large rating deviations. In this phase, each member will be given the opportunity to change their rating.
Then, we'll pick the conference program taking into account the following factors:
- average rating by committee members
- balance of topics
- diversity of speakers
In other words, the final lineup will not be based exclusively on ratings, but also on the points mentioned in the "Goals" section. So, for example, if two talks about the same topic have a high average rating, we might select just one of them. Likewise, we might drop some better rated talks in one area (e.g. design) to reach a better balance. Those decisions will be reached by consensus of all the participants.
The members of the core team will have an advisory role in the meeting. They will also have the power to veto certain decisions, and break the tie in situations where the committee members can't reach the consensus.
The final output of the meeting will be:
- the list of selected short and long talks
- the list of talks which will enter the community vote
- the list of backup talks
Each talk is rated with a 1-5 grade. A rough meaning of grades is:
- 5 - Excellent talk which must be included
- 4 - Very good talk, would make a great addition to the lineup
- 3 - A good talk, would like to see it included
- 2 - Not a great talk, but can be included if there are no better options
- 1 - I don’t think that this talk is appropriate for webcamp (VETO)
As a committee member, you're expected to rate most of the submissions. However, you can decide to skip some submissions in some cases, such as:
You don't feel qualified to grade the talk (e.g. a backend developer might skip rating designer submissions, and vice versa).
You feel that there's some conflict of interest regarding a particular submission.
All submissions should be rated on their own merit. For example, if there are many submissions on the same subject, the committee member shouldn't decrease the grade of some of them in fear that multiple talks with the same subject will be accepted. Likewise, if the same speaker has multiple submissions, don't try to promote one submission by degrading all others.
As explained in the "Selection process" section, we'll address these concerns in the final meeting.
This section contains some tips you can consider during the rating process. These are not rules which are meant to be followed strictly. You're of course advised to use your own judgement. Ultimately, the goal of the rating process is to help us choose a great conference program, not to follow some set of bureaucratic rules.
Prefer submissions which are clearly formulated and explained. If a speaker can properly summarize the topic into a brief text, it's a good indication that they can also produce an informative talk.
Prefer topics which are interesting to a wider audience. This include introductory talks, use-case demonstrations, general purpose topics about IT. The niche talks, such as "Performance tips when using library Foo with the language Bar" will likely not be interesting nor relevant to a wider group of people.
Consider speaker's biography, public profile (e.g. social networks, YouTube, GitHub, joind.in), and the ratings of their past talks (if they spoke on WebCamp). If a video is available, take a brief look to verify if the speaker presentation style and clarity.
However, try to avoid negative discrimination. If a speaker doesn't have a high public profile, they shouldn't be punished for that. We also shouldn't discriminate first time speakers.
To summarize, if there are some positive traits about the speaker's biography or their public profile, you may consider increasing the grade. On the other hand, a lack of public profile or past talking experience shouldn't be punished by lowering the grade.
Don't let your personal feelings about the topic affect your rating. For example, if the talk is about a programming language which you personally dislike, but the language is widely used and interesting, you shouldn't give the submission a lower grade. Likewise, don't give a higher grade just because the discussed technology is something you're personally passionate about. It goes without saying that you should avoid promoting submissions by friends and family. If you feel that the conflict of interest is too strong, you can always skip rating the submission.
Be mindful of the proposed talk duration, particularly for 45 minutes slots. Some topics work better as short ones, while others work better as long ones. Some candidates may submit the same submission for both 25 and 45 minute slot, and it's perfectly fine if your grade is different between the two submissions.
The grading dashboard offers the option of commenting talks. Links can be added as well as text, and videos will be expanded on the submission page. When reviewing an application, you’re encouraged to add links of other talks of the applicant to save others the effort. However, don’t include opinionated comments, to avoid influencing the opinion of others.