We examine the determinants of off-farm work in Swiss agriculture between 2003 and 2013. By differentiating between the between-farm effects and the within-farm effects, our model provides new insights into the labour allocation process as compared with standard cross-sectional or panel data models. As regards the between-farm variations, our results show that younger farm families without children and farm households with higher non-agricultural education levels of both the farmer and the partner are more involved in working activities outside the farm. However, the within-farm time effects provided a more differentiated picture: impacts of changes in most variables over time tended to be smaller, and in case of two variables show opposite directions. In addition to a negative between-effect of farm income on the allocation of off-farm labour, our results on within-farm effects suggest that an increase in farm income per annual family work unit could be compatible with a higher share of off-farm work during the analysed period.