Black dot and silver scurf are potato blemish diseases whose economic impact has increased in recent years. Because their symptomatology on tubers is visually similar, disease assessment does not usually differentiate between the two pathogens, which share the same ecological niche. The epidemiology of black dot has been extensively studied, especially in the UK, but the factors that influence silver scurf have been less investigated. In this study, the influence of cultivar, source of inoculum, and environmental conditions on both diseases was studied in field trials over a three-year period (2016–2018) in Switzerland. Planting minitubers did not prevent either disease in daughter tubers, indicating the contribution of soil as an inoculum reservoir. An arbitrary threshold of Colletotrichum coccodes soil inoculum could be set to discriminate between low and high disease risk. For the first time, Helminthosporium solani DNA was detected in stolons, and infections appeared earlier in stolons than in tubers. H. solani stolon and tuber infections usually appeared later in the season than those of C. coccodes. Black dot severity correlated positively with precipitation, while silver scurf severity correlated positively with temperature. Table potato cultivars commonly grown in Switzerland exhibited significant differences in susceptibility to both diseases, and cultivars with low susceptibility to both silver scurf and black dot were identified. These results gave new insights into understanding the factors driving the epidemiology of potato blemish diseases and may contribute to building a risk assessment scheme to manage both diseases simultaneously.