It has widely been acknowledged that the diversity of arbuscular mycorrhizal fungi (AMF) is greatly affected by climate, land use intensity, and soil parameters. The objective of this study was to investigate AMF diversity in multiple agricultural soils (154 sites; 92 grasslands and 62 croplands) distributed over all agricultural regions in Switzerland and differing in a number of soil parameters (e.g., land use type and intensity, and altitude). We highlighted the main factors responsible for major AMF community shifts and documented specific distribution patterns for each AMF species. AMF spores were morphologically identified and counted for each species. In total, 17,924 spores were classified and 106 AMF species were identified. In general, AMF species richness (SR) was higher in grasslands than in croplands. In croplands, SR increased with altitude but this trend was not observed in grasslands. Some species occurred at virtually all sites, while others were rarely detected, and for others, species-specific distribution patterns were revealed. Some species were affected by land use type or intensity, or related factors like soil organic matter, soil microbial biomass and respiration or nutrient availability. Other species were more affected by soil pH and related parameters like base saturation and carbonate contents, by soil texture, or by altitude, or by a combination of two to several of all these parameters. We conclude that a high number of AMF species may serve as indicator species for specific habitats and land use. These species might deliver certain ecosystem services at their habitats and deserve further investigation about their functional diversity.