Despite great strides in reducing hunger over the last two decades, malnutrition remains a major challenge in Indonesia. High rates of child stunting coexist with high and increasing rates of overweight and obesity despite rapid economic growth and reductions in poverty over the last two decades. Part of this economic growth has been driven by a change in agricultural production systems from traditional farming techniques with farmers growing multiple crops to more intensified, specialized and commercialized farms. The objective of this study is to analyze how changes in the structure of agricultural production have affected diets in rural Indonesian households over time. We use three waves of a panel data set from the Indonesian Family Life Survey with a balanced sample of 2785 rural households covering the period between 2000 and 2015 to see how households’ food choices have been changing over time in response to the changes in production systems. We find positive relationships between production diversity and household dietary diversity as well as between market access and household dietary diversity. However, we see that there has been an overall decline in dietary diversity over time in the same households as their production diversity has declined. This decline in dietary diversity was mostly driven by the decreased consumption of nutritious food groups (fruits, vegetables, legumes, and fish). Although the magnitude of the association between dietary diversity and production diversity was relatively small, the association between household production and consumption of some of these important food groups was quite substantial. The overall impact of increased specialization in Indonesia during the period 2000–2015 on dietary quality appears to have been negative.