crowdsourcing

Crowdsourced investigations shore up democratic institutions by debunking misinformation and uncovering human rights abuses. However, current crowdsourcing approaches rely on simplistic collaborative or competitive models and lack technological sup- port, limiting their collective impact. Prior research has shown that blending elements of competition and collaboration can lead to greater performance and creativity, but crowdsourced investiga- tions pose unique analytical and ethical challenges. In this paper, we employed a four-month-long Research through Design process to design and evaluate a novel interaction style called collabora- tive capture the flag competitions (CoCTFs). We instantiated this interaction style through CoSINT, a platform that enables a trained crowd to work with professional investigators to identify and inves- tigate social media misinformation. Our mixed-methods evaluation showed that CoSINT leverages the complementary strengths of competition and collaboration, allowing a crowd to quickly identify and debunk misinformation. We also highlight tensions between competition versus collaboration and discuss implications for the design of crowdsourced investigations.

Social media platforms have enabled extremists to organize violent events, such as the 2021 U.S. Capitol Attack. Simultaneously, these platforms enable professional investigators and amateur sleuths to collaboratively collect and identify imagery of suspects with the goal of holding them accountable for their actions. Through a case study of Sedition Hunters, a Twitter community whose goal is to identify individuals who participated in the 2021 U.S. Capitol Attack, we explore what are the main topics or targets of the community, who participates in the community, and how. Using topic modeling, we fnd that information sharing is the main focus of the community. We also note an increase in awareness of privacy concerns. Furthermore, using social network analysis, we show how some participants played important roles in the community. Finally, we discuss implications for the content and structure of online crowdsourced investigations.

Online extremism can quickly spill over into the physical world and have dangerous consequences, as when rioters attacked the U.S. Capitol on January 6, 2021. While information and communication tech- nologies have enabled extremists to plan and organize violent events, they have also enabled collective action by others to identify the perpetrators and hold them accountable. Through a mixed-methods case study of Sedition Hunters, a Twitter-based community whose goal is to identify individuals who took part in the Capitol attack, we explore: 1) how the community formed and changed over time; 2) the motives, ethos, and roles of its members; and 3) the methods and software tools they used to identify individuals and coordinate their activities.

Investigators in fields such as journalism and law enforcement have long sought the public’s help with investigations. New technologies have also allowed amateur sleuths to lead their own crowdsourced investigations - that have traditionally only been the purview of expert investigators - with mixed results. Through an ethnographic study of a four-day, co-located event with over 250 attendees, we examine the human infrastructure responsible for enabling the success of an expert-led crowdsourced investigation. We find that the experts enabled attendees to generate useful leads; the attendees formed a community around the event; and the victims' families felt supported. Additionally, the co-located setting, legal structures, and emergent social norms impacted collaborative work practice. We also surface three important tensions to consider in future investigations and provide design recommendations to manage these tensions.

Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. Here, we discuss a real-world deployment of a system we built, GroundTruth, that enables experts to direct crowds in determining the exact geographic location where an image was taken, a key step in verifying visual evidence. We discuss the challenges working with these experts and the methods we have employed to study our software deployment.

Misinformation poses a threat to public health, safety, and democracy. Training novices to debunk visual misinformation with image verification techniques has shown promise, yet little is known about how novices do so in the wild, and what methods prove effective. Thus, we studied 225 verification challenges posted by experts on Twitter over one year with the aim of improving novices’ skills. We collected, annotated, and analyzed these challenges and over 3,100 replies by 304 unique participants. We find that novices employ multiple tools and approaches, and techniques like collaboration and reverse image search significantly improve performance.

Crowdsourced labeling of political social media content is an area of increasing interest, due to the contextual nature of political content. However, there are substantial risks of human biases causing data to be labelled incorrectly, possibly advantaging certain political groups over others. Inspired by the social computing theory of social translucence and findings from social psychology, we built PairWise, a system designed to facilitate interpersonal accountability and help mitigate biases in political content labelling.

Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. In this paper, we introduce the concept of shared representations for crowd–augmented expert work, focusing on the complex sensemaking task of image geolocation performed by professional journalists and human rights investigators. We built GroundTruth, an online system that uses three shared representations-a diagram, grid, and heatmap-to allow experts to work with crowds in real time to geolocate images. Our mixed-methods evaluation with 11 experts and 567 crowd workers found that GroundTruth helped experts geolocate images, and revealed challenges and success strategies for expert-crowd interaction. We also discuss designing shared representations for visual search, sensemaking, and beyond.