Center for Information Technology Research in the Interest of Society (CITRIS)

Expert feedback is valuable but hard to obtain for many de- signers. Online crowds can provide a source of fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted an experiment with a 2x2 facto- rial design. Student designers received feedback on a visual design artifact from both experts and novices, who produced feedback using either an expert rubric or no rubric. We found that rubrics helped novice workers provide feedback that was rated just as valuable as expert feedback. A follow-up analy- sis on writing style showed that student designers found feed- back most helpful when it was emotionally positive and spe- cific, and that providing a rubric increased the occurrence of these characteristics in feedback. The analysis also found that expertise correlated with longer critiques, but not the other fa- vorable characteristics. An informal evaluation indicates that experts may instead have produced value by providing clearer justifications.