Most of these farmers can no longer support themselves by working the farm alone. The average farm income for this group is about $8,000 a year, according to USDA. As a result, many have at least one family member take a job outside the farm as a primary means of income.

But even though they may not be money machines, the farming lifestyle is still attractive for a lot of people.

"We don't need the big pickup and the fancy tractor," said one Vermont dairy farmer. "We do it because we enjoy it."