Many scientists experience a practice-preference gap about peer review. Single-blind review---where authors' identities are revealed to reviewers---is often used for evaluation. Yet, double-blind review---where authors' identities are concealed---is seen as more fair. To understand this gap, we compared both systems in a high-stakes field study: submissions to the Society for Judgment and Decision Making’s annual conference, the leading international conference on this topic. Each submission received both review types. Reviewers were randomly assigned to the review system and submissions. Selected conference talks were evaluated for quality, popularity, and subsequent publication status. We assessed the two systems on reliability, bias, and validity. On reliability, while both systems had moderate reliability, agreement was higher on what constituted a poor submission than a strong one (Anna Karenina Principle). On bias, double-blind reviews showed a slight bias against submission by women (Matilda Effect), while single-blind reviews showed a preference for submissions with senior co-authors (Matthew Effect). On validity, neither system predicted talk quality or popularity, but both predicted publication status. Author characteristics did not consistently predict outcomes. Thus, we suggest the costs of single-blind review do not outweigh its benefits. Yet, double-blind review is also not a perfect solution. We propose an equitable approach for selecting scientific work may be an informed lottery: use double-blind review to identify submissions of merit, then randomly choose from this set.