Zinc (Zn) deficiency is a widespread problem in human nutrition and wheat grains are a major source of Zn intake in large parts of the population. It remains unclear to what extent organic and conventional farming practices, differing in organic matter management, influence Zn availability and uptake by wheat grains. Factors leading to an increased Zn uptake may also increase the Cd uptake in wheat grains, which can be harmful for humans. Here, we investigated the effects of different farming practices on Zn and Cd concentrations in wheat grains and their relationships with total and available soil Zn and Cd concentrations, and other soil properties. In northern Switzerland, 28 farms were sampled including 11 organic farms with compost use, 10 organic farms without compost use, and 7 conventional farms without compost use. Soil organic matter was a key factor for soil Zn and especially Cd concentrations across all three farming systems. Total and available soil Cd concentrations as well as soil organic carbon concentration (SOC) were significantly higher on the organic farms with compost use than on the conventional farms. However, only the compost farms with livestock showed significantly higher grain Cd concentrations in comparison to conventional and organic farms without compost use, although a nested effect of cultivar within the system also had an influence. In contrast to Cd, the soil and grain Zn concentrations showed no significant farming system effect although there was a correlation between total soil Zn and SOC when all farms were pooled. Grain Zn was decoupled from soil Zn indicating that under agricultural field conditions the farming systems are a minor factor in increasing grain Zn. Our study suggests that the Zn and Cd soil and grain concentrations were mediated by a combination of on-farm organic matter management, soil properties, cultivar, and livestock production.