Despite the importance of soil microorganisms for ecosystem services, long-term surveys of their communities are largely missing. Using metabarcoding, we assessed temporal dynamics of soil bacterial and fungal communities in three land-use types, i.e., arable land, permanent grassland, and forest, over five years. Soil microbial communities remained relatively stable and differences over time were smaller than those among sites. Temporal variability was highest in arable soils. Indications for consistent shifts in community structure over five years were only detected at one site for bacteria and at two sites for fungi, which provided further support for long-term stability of soil microbial communities. A sliding window analysis was applied to assess the effect of OTU abundance on community structures. Partial communities with decreasing OTU abundances revealed a gradually decreasing structural similarity with entire communities. This contrasted with the steep decline of OTU abundances, as subsets of rare OTUs (<0.01%) revealed correlations of up to 0.97 and 0.81 with the entire bacterial and fungal communities. Finally, 23.4% of bacterial and 19.8% of fungal OTUs were identified as scarce, i.e., neither belonging to site-cores nor correlating to environmental factors, while 67.3% of bacterial and 64.9% of fungal OTUs were identified as rare but not scarce. Our results demonstrate high stability of soil microbial communities in their abundant and rare fractions over five years. This provides a step towards defining site-specific normal operating ranges of soil microbial communities, which is a prerequisite for detecting community shifts that may occur due to changing environmental conditions or anthropogenic activities.