The aggregation of consistent individual judgments on logically interconnected propositions into a collective judgment on those propositions has recently drawn much attention. Seemingly reasonable aggregation procedures, such as propositionwise majority voting, cannot ensure an equally consistent collective conclusion. In this paper, we motivate that quite often, we do not only want to make a factually right decision, but also to correctly evaluate the reasons for that decision. In other words, we address the problem of tracking the truth. We set up a probabilistic model that generalizes the analysis of Bovens and Rabinowicz (Synthese 150: 131–153, 2006) and use it to compare several aggregation procedures. Demanding some reasonable adequacy constraints, we demonstrate that a reasons- or premise-based aggregation procedure tracks the truth better than any other procedure. However, we also illuminate that such a procedure is not in all circumstances easy to implement, leaving actual decision-makers with a tradeoff problem.