Semantic segmentation methods have become increasingly popular in the field of agronomy for their ability to accurately and efficiently analyse images of crops. These methods use machine learning algorithms to assign semantic labels to each pixel in an image and require extensive amount of labelled data. Currently, most of the available models focus on crop and weed identification and target a single growth stage. In this paper, we propose a robust semantic segmentation method for estimating the dynamics of multi-species cover in intercropping systems, using only 50 images for training the model. We applied transfer learning on the well-established convolutional neural network DeepLab to decrease the image annotation effort. Three models are trained using field images from a three-year field trial with canopy densities ranging from early development at 1.2% to well-developed canopy covers of 98.7%. Overall, we propose a two-class segmentation model to differentiate vegetation from soil, obtaining 96.8% mean accuracy. Two methods for three-class segmentation models to identify soil, oilseed rape and the other plants are proposed, reaching best mean accuracy of 96.2%. The proposed method is able to differentiate oilseed rape from service plant mixtures at all growing stages, allowing for accurate assessment of the dynamic competition for light between these species. Hence, semantic segmentation methods in agronomy have the potential to support study and management of crops, enabling more accurate and efficient data collection and analysis.