The concept of crowdsourcing has gained a lot of attention lately. Many companies are making use of this concept for value creation, as well as the performance of varied tasks. Despite its wide application, little is known about crowdsourcing, especially when it comes to crowdsourced software testing. This paper explores the crowdsourced software testing concept from a wider perspective ranging from a cost-benefit analysis, crowdsourcing intermediaries, and the level of expertise in the crowd. Drawing from a varied range of sources, a systematic literature review is done, where the research narrows down to ten most relevant peer-reviewed sources of high impact rating. In a comparative analysis between crowdsourced software testing and in-house testing, it is found that crowd testing has numerous advantages when it comes to efficiency, user heterogeneity, and cost-effectiveness. The study indicates that intermediaries play a key role in managing the connection between the crowd and crowdsourcing companies despite various challenges. A comparison between novice testers and expert testers reveals that both the two have their unique capabilities in their respective domains.
Given Enough Eyeballs, all Bugs are Shallow - A Literature Review for the Use of Crowdsourcing in Software Testing