A lack of public trust in farmers can impede having a food production system that is functional and socially acceptable. It can also lead to stricter and unfounded regulations on farming practices limiting the farmers and even impacting the viability and sustainability of farms. Therefore, we examine public image of farmers and the agriculture sector through a survey with 481 participants from the German-speaking part of Switzerland. More specifically, consumers' trust in crop and livestock production farmers are determined, along with perceived knowledge and affect towards farmers. Other individual and behavioral factors are identified to build trust in farmers. The results reveal that the public has a simplistic and traditional view of agriculture and farmers as they do not spontaneously associate them with new farming practices or technologies. However, crop farmers are more trusted than livestock production farmers as concerns related to transparency of farmers, care for the environment and animal welfare emerged. Trust in crop and livestock farmers is influenced by affect towards farmers, perceived knowledge, and political orientations, whereas behavioral factors (i.e., meat consumption) only influence trust in livestock production farmers. Building trust in farmers could need to be on the system-level and not solely on the farmer-level. Overall, our results provide suggestions on how to build and maintain public trust in both crop and livestock farmers through not only farmers’ initiatives but also agricultural policies.