Sustainability has become a key term for linking environmental, economic and social issues, in both the sciences and politics. Conceptions and frameworks of sustainability have thus arisen to evaluate agricultural systems on their sustainability. Within these conceptions and in political and scientific discourses, what can be understood as the social pillar of sustainability in agriculture varies greatly, especially in regards to the scope and the sustainability standards applied. While rural inhabitants have been subject of various ‘sustainability studies’, the consideration of the social dimension in agriculture is still rather underrepresented. Our conceptual framework can contribute to enhance the understanding of the social dimension of sustainability by utilizing a social science-based approach to comprehend the complexity of social interaction in agriculture: Based on Parsons' system approach, we capture the components of a social system that encompasses agriculture and its embeddedness in society. This includes all major actors, their interactions and institutions. Further, we develop Maslow's hierarchy of needs as well as the rights approach into a sustainability scale. We call the conceptual framework the sustainable agricultural social system. This general framework can later be adapted to local cultural and social settings, serving as a more comprehensive and flexible sustainability framework.