To determine the effects of light grazing and grazing prohibition on the physical and mechanical properties of grassland soil in the landslide distribution area of the upper reaches of the Yellow River, the Xiazangtan landslide distribution area in the upper reaches of the Yellow River was selected as the study area. The plant growth and physical and mechanical properties of soil in different slope directions of lightly grazed grassland and grazing-prohibited grassland were compared and analyzed by conducting a field investigation, sampling, and testing the indoor soil physical and mechanical properties. The effects of light grazing and grazing prohibition on plant growth and soil physical and mechanical properties of slope in landslide distribution area were discussed. The results showed that the number of plant species, average cover, and average plant height in the grazing-prohibited grassland were higher than that in the lightly grazed grassland, and increased by 65.1%, 62.8%, and 13.6%, respectively. With the increase in soil depth, the soil density of the two types of grassland showed a trend of decreasing first and then increasing, soil water content and root dry weight density followed a decreasing trend, soil cohesion showed a “V”-type change, and the soil physical and mechanical properties of the same type of grassland with different soil depth were significantly different (P<0.05). The soil density, soil moisture content, root dry weight density, and soil cohesion of the two types of grassland followed an increasing trend from the top to the bottom of the slope. In the same slope direction, the soil density of lightly grazed grassland was significantly higher than that of grazing-prohibited grassland (P<0.05), whereas the soil moisture content, root dry weight density, and soil cohesion of grazing-prohibited grassland were significantly higher than those of lightly grazed grassland (P<0.05). The soil internal friction angle did not show a significant pattern of change in response to the soil depth and slope position, and there was no significant difference between the two types of grassland in the same slope direction. Soil cohesion was positively correlated with soil moisture content and soil density and soil cohesion was significant positively correlated with root dry weight density (P<0.05) in two types of grasslands. Therefore, grazing prohibition is beneficial for the restoration of degraded grasslands, the improvement of plant community structure, and inhibits soil erosion. The results of this study provide a theoretical basis and practical guidance for the effective prevention and control of secondary geological disasters, such as soil erosion and shallow landslides, on riverbank slopes in the study area.