Great diversity of shape, size, and skin color is observed among the fruits of different apple genotypes. These traits are critical for consumers and therefore interesting targets for breeding new apple varieties. However, they are difficult to phenotype and their genetic basis, especially for fruit shape and ground color, is largely unknown. We used the FruitPhenoBox to digitally phenotype 525 genotypes of the apple reference population (apple REFPOP) genotyped for 303,148 single nucleotide polymorphism (SNP) markers. From the apple images, 573 highly heritable features describing fruit shape and size as well as 17 highly heritable features for fruit skin color were extracted to explore genotype–phenotype relationships. Out of these features, seven principal components (PCs) and 16 features with the Pearson’s correlation r < 0.75 (selected features) were chosen to carry out genome-wide association studies (GWAS) for fruit shape and size. Four PCs and eight selected features were used in GWAS for fruit skin color. In total, 69 SNPs scattered over all 17 apple chromosomes were significantly associated with round, conical, cylindrical, or symmetric fruit shapes and fruit size. Novel associations with major effect on round or conical fruit shapes and fruit size were identified on chromosomes 1 and 2. Additionally, 16 SNPs associated with PCs and selected features related to red overcolor as well as green and yellow ground color were found on eight chromosomes. The identified associations can be used to advance marker-assisted selection in apple fruit breeding to systematically select for desired fruit appearance.