Conferences and program committee (PC) meetings serve critical social and research functions in the computer science community. Ensuring a fair process that encourages, guides, and rewards high quality and forward looking research is the responsibility of the reviewing process.
The reviewing process and PC meetings serve to develop and express the shared values of the research community. The PC chair is the most critical person in this process and should seek to ensure a fair and high quality process that encourages and builds the community.
Poor review quality and poor process is very damaging to our community. It alienates all community members and is especially damaging because it disproportionately discourages new community members (i.e., graduate students and young researchers), since they have no or many fewer good previous experiences to counter a bad experience. PC members and chairs should take their job very seriously.
To improve review expertise, reviewer accountability, handle PC submissions, and reduce PC member workload, I recommend a moderately sized Program Committee (PC) with an in person PC meeting and a named External Review Committee (XRC), as pioneered in the SIGPLAN and SIGARCH communities by Steve Blackburn at ISMM 2008. ASPLOS, ISCA, ISMM, PLDI, POPL, and other conferences now regularly use this structure.
Another key to high quality reviews and moderating the workload (which are related!), is two round reviewing, in which each paper gets two PC reviews and one XRC review in a first round. With this information in hand, the committee rejects at least half the submissions. The second round adds one or two reviews, such that every paper that will be considered further will have at least 3 PC reviews.
Your committee should be large and diverse enough to provide expert reviewers on a paper with one conflict in the topic area. Large committees decrease the work load, but sometimes increase the probability one or more members wont do their work or do it on time. Make sure everyone agrees to attend the PC meeting. PC attendance and applying no limit on paper acceptance are key to a diverse and forward looking program.
As the program chair, you want to create a committee that covers all the technical areas for the conference and that write useful reviews in a timely fashion (timely is key to reducing your stress). To get high quality reviews, you need responsible and knowledgeable reviewers. Your personal experience and those of prior PC chairs are a great source of this information. Ask them who does their work? Who wrote the best reviews? Who was productive in the PC meeting, e.g., made thoughtful comments and moved papers to their conclusion in a principled fashion?
Secondary, but very important should be diversity of institution, rank, and experience. I always use a spreadsheet that includes any previous service on this conference's PC, gender, institution, career stage (early, middle, late), and topic coverage. The conference should be tracking all PC members in the past in a spreadsheet. To create the topic list and a source of program committee members beyond my personal network, I use and recommend extracting topics and researchers from the past 5 years of the conference. I also look at related conferences for emerging topics and researchers.
Asking early or late? For ASPLOS, I asked very early (more than 1 year out from the work), but then several people had personal conflicts. For PACT, ISMM and CGO, we asked late (4 months out), which made it more difficult to obtain senior members but yielded a committees without turnover, but still the occasional emergency prevents PC attendance.
I recommend a change to the current system, where everyone turns over every year and thus the PC chair is left to choose the software submission and reviewing system, and then populate the entire process. I recommend a three year term for a web submission czar for each conference to configure and manage the submission software: one year in training, one year running it by herself or himself, and one year training the next person. Besides the obvious benefit that the PC chair has someone to help them who has a lot of experience with the software, the key benefit is that if the PC wants to change process, topics, conflicts, review form, etc., they still can, but at least the change is intentional instead of random.
Conflicts. I prefer that the authors specify their PC conflicts and institution conflicts with a check box, and others in a list which greatly eases paper assignation. However, this mechanism gives authors the ability to veto reviewers, without conflicts, and several PC chairs have discovered this problem. Reversing the process to avoid this problem reveals authors to PC members, slightly unblinding the process. Reminding authors to only submit conflicts based on your policy should be sufficient.
Matching. I recommend bidding for the papers by the PC and XRC, and then doing paper assignments with this information and topic expertise in hand. I used this system for all the PCs I chaired except ASPLOS. For CGO, Lieven Eeckhout and I together spent about 10 to 15 hours refining the automated system to make sure each paper had good expertise. For ASPLOS, I made all the paper assignments by matching paper topic areas to PC member topic areas. I found that I did not do a great job (with respect to PC interest and expertise without their input) and for 169 papers and 18 PC members, and highly recommend bidding, which is now broadly used.
I used two round reviewing at PLDI, ASPLOS, and CGO. I first used two round reviewing at ASPLOS by necessity rather than design due to an unexpected quantity of submissions. In the first round, I assigned each paper two committee member reviewers (about 20-25 hours of work for me), and each of them specified one or more reviewers from the PC. For the 112 papers with a significant number of positive reviews or conflicting reviews, I then assigned an additional committee member reviewer (another 10 hours of work) and in a few cases, external reviews. Thus, each paper had at least 4 reviews and all papers discussed at the committee meeting had 5 reviews, with 3 from committee members. Most of these reviews came in before the rebuttal period. In addition, each PC member had a higher density of papers in the top group. I recommend this two tier strategy for better focus with a large number of submissions, but it is more work for the PC chair.
For all the other PCs, I instead gave the PC members a list of all their non-conflict papers, and let them indicate their preferences. With this information, I achieved a 100% match of submissions to one or more PC and XRC members that wanted to review the submission, and it took less time. As a result, I believe we had better and more qualified reviewing than we would have otherwise.
I highly recommend blind submission to reduce bias, and the rebuttal process to encourage constructive and accurate reviewing.
I have seen the rebuttal process change several, but not numerous numbers of reject and accept decisions. These decisions are however important and it has the following additional benefits. Reviewers submit their reviews before the rebuttal period, instead of doing them on the plane on the way to the PC meeting. Reviewers are more likely to do a careful job for that reason, and because the authors can correct them and their peer reviewers are watching. Furthermore, rebuttal helps build community, when reviewers and authors communicate through rebuttal, especially when the reviewers add a final statement about the accept/reject result in light of the committee discussions and rebuttal comments.
I have a few points of possible interest about the purpose of a PC meeting and how to run the meeting itself smoothly.
Schedule a long day, but build in breaks to refresh everyone. Stick to your break schedule. In 8 hours of discussion and 1.5 hours of breaks, it is possible to discuss no more than 50 papers. Adjust either the papers or the meeting length accordingly.
I used the following group ordering and random within group in most of my meetings: top groups (all Accept, two accepts, one accept) until bogged down, bottom up (reject 10 at time until the PC saved more than one paper in the group), PC papers, back to groups based on one top score, and in the end, when you are running out of time, only the advocate system (widely used).
Minor edits in October 2014 corrected grammar errors and improved clarity.