In natural ecosystems, positive effects of plant diversity on ecosystem functioning have been widely observed, yet whether this is true in cropping systems remains unclear. Here we assessed the impact of crop diversification on soil microbial diversity, soil multifunctionality (SMF) and crop yields in 155 cereal fields across a 3,000 km north–south European gradient. Overall, crop diversity showed a relatively minor effect on soil microbial diversity, SMF and yields. In contrast, the proportion of time with crop cover (including cash crops, cover crops or forage leys) during the past ten-year crop rotation had a much stronger impact. This suggests that increasing crop cover can enhance both yields and soil functioning, while also providing habitat for soil microorganisms. We found that SMF did not positively contribute to crop yields, highlighting that care must be taken to balance the provision of food with environmentally beneficial functions and services, since they do not always go hand in hand.