Winters and early springs are predicted to become warmer in temperate climates under continued global warming, which in turn is expected to promote earlier plant development. By contrast, there is no consensus about the changes in the occurrence and severity of late spring frosts. If the frequency and severity of late spring frosts remain unchanged in the future or change less than spring phenology of plants does, vulnerable plant organs (dehardened buds, young leaves, flowers or young fruits) may be more exposed to frost damage. Here we analyzed long-term temperature data from the period 1975–2016 in 50 locations in Switzerland and used different phenological models calibrated with long-term series of the flowering and leaf-out timing of two fruit trees (apple and cherry) and two forest trees (Norway spruce and European beech) to test whether the risk of frost damage has increased during this period. Overall, despite the substantial increase in temperature during the study period, the risk of frost damage was not reduced because spring phenology has advanced at a faster rate than the date of the last spring frost. In contrast, we found that the risk of frost exposure and subsequent potential damage has increased for all four species at the vast majority of stations located at elevations higher than 800 m while remaining unchanged at lower elevations. The different trends between lower and higher elevations are due to the date of the last spring frost moving less at higher altitudes than at lower altitudes, combined with stronger phenological shifts at higher elevations. This latter trend likely results from a stronger warming during late compared to earlier spring and from the increasing role of other limiting factors at lower elevations (chilling and photoperiod). Our results suggest that frost risk needs to be considered carefully when promoting the introduction of new varieties of fruit trees or exotic forest tree species adapted to warmer and drier climates or when considering new plantations at higher elevations.