Fall 2011 Crowdsourcing Reading Group


The Crowdsourcing Reading Group meets every other Friday from 2pm to 3pm in ACES 6.116. We read and discuss papers related to crowdsourcing, Amazon Mechanical Turk, annotator modeling, spam detection, task design, and other topics of interest to crowdsourcing research. We're interested in applications to machine learning (and often, natural language processing and information retrieval).

Scheduled Meetings

Date Time Place Agenda
4/27/2012 2:15 PM UTA 5.428 Heymann, P. and Garcia-Molina, H.
Turkalytics: analytics for human computation
WWW 2010 [pdf]

Past Meetings

Date Time Place Agenda
9/14/2011 10AM ACES 6.116 Yan Yan, Rómer Rosales, Glenn Fung and Jennifer Dy.
Active Learning from Crowds
ICML 2011
[pdf]
9/28/2011 10AM ACES 6.116 Florian Laws, Christian Scheible, Hinrich Schütze
Active Learning with Amazon Mechanical Turk
EMNLP 2011
[pdf]
10/12/2011 10AM ACES 6.116 S. Vijayanarasimhan and K. Grauman
Large-Scale Live Active Learning: Training Object Detectors with Crawled Data and Crowds
CVPR 2011
[pdf]
10/26/2011 10AM ACES 6.116 Donmez, P., Carbonell, J., and Schneider, J.
A Probabilistic Framework to Learn from Multiple Annotators with Time-Varying Accuracy
SIAM 2010
[pdf]
11/9/2011 10AM ACES 6.116 Peter Welinder, Steve Branson, Serge Belongie, Pietro Perona
The Multidimensional Wisdom of Crowds
NIPS 2010
[pdf]
11/23/2011 10AM ACES 6.116 Peng Dai, Mausam, and Daniel S. Weld
Artificial Intelligence for Artificial Artificial Intelligence
AAAI 2011
[pdf]
2/17/2012 2 PM ACES 6.116 Ryan Gomes, Peter Welinder, Andreas Krause, and Pietro Perona
Crowdclustering
NIPS 2011 Poster [pdf]
3/2/2012 2 PM ACES 6.116 Sonia Chernova, Nicholas DePalma, Elisabeth Morant and Cynthia Breazeal
Crowdsourcing Human-Robot Interaction: Application from Virtual to Physical Worlds
IEEE International Symposium on Robot and Human Interactive Communication [pdf]
3/23/2012 3 PM UTA 5.428 Shuo Chen, Jianwen Zhang, Guangyun Chen, Changshui Zhang
What if the Irresponsible Teachers Are Dominating? A Method of Training on Samples and Clustering on Teachers
AAAI 2010 [pdf]
3/30/2012 2 PM ACES 6.116 Liyue Zhao, Gita Sukthankar, Rahul Sukthankar
Incremental Relabeling for Active Learning with Noisy Crowdsourced Annotations
Proceedings of Social Computing [pdf]
4/13/2012 2 PM ACES 6.116 Anand Kulkarni, Matthew Can, Björn Hartmann.
Collaboratively Crowdsourcing Workflows with Turkomatic
CSCW 2012 [pdf]

Organized by Yinon Bentor. Site design by Karl Pichotta and others.

Proposed Papers:

  1. Learning with Annotation Noise
    Eyal Beigman and Beata Beigman Klebanov
    ACL 2009
  2. Vox Populi: Collecting High-Quality Labels from a Crowd
    Ofer Dekel and Ohad Shamir
    COLT 2009
  3. Learning From Crowds
    Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, Linda Moy
    JMLR 2010
  4. Interactively Building a Discriminative Vocabulary of Nameable Attributes
    Devi Parikh and Kristen Grauman
    CVPR 2011
  5. Active Learning with Multiple Annotations for Comparable Data Classification Task
    Vamshi Ambati, Sanjika Hewavitharana, Stephan Vogel and Jaime Carbonell
    ACL 2011 Workshop
  6. AutoMan : A Platform for Integrating Human-Based and Digital Computation
    Dan Barowy, Emery D. Berger, and Andrew McGregor.
    NIPS Workshop / Tech Report
  7. Iterative Learning for Reliable Crowdsourcing Systems
    David Karger, Sewoong Oh, Devavrat Shah
    NIPS 2011
  8. Real-time Crowd Control of Existing Interfaces
    Walter S. Lasecki, Kyle I. Murray, Samuel White, Robert C. Miller, and Jeffrey P. Bigham
    UIST 2011
  9. Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces
    Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger
    UIST 2011
  10. Human Computation Tasks with Global Constraints
    Haoqi Zhang, Edith Law, Robert C. Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz.
    CHI 2012, to appear

Mailing list:

We're using Google Groups to announce papers and meetings. Please sign up here: UT Mechanical Turk and Crowdsourcing Users Google Group

We limit non-UT access to the group so that we can occasionally share pre-publication papers for discussion. If you're interested in joining, please state your affiliation (department, etc) in the comment box.