The court's more conservative members seized the initiative during arguments.
Roberts questioned how schools were to measure classroom diversity among students who come from mixed race families.
Justice Samuel Alito suggested it was unfair to place Asian-Americans into one group, when they come from a wide range of cultures-- Filipinos, Chinese, and Afghans among them.
"I thought that the whole purpose of affirmative action was to help students who come from underprivileged backgrounds, but you make a very different argument that I don't think I've ever seen before," he told Gregory Garre, representing the school.
Garre said the school wants minorities from different backgrounds.
"We go out of our way to recruit minorities from disadvantaged backgrounds," he said.
Kennedy jumped in.
"So what you're saying is that what counts is race above all," he said. "You want underprivileged of a certain race and privileged of a certain race. So that's race."
Kennedy added later: "I thought that the whole point is that sometimes race has to be a tie-breaker and you are saying that it isn't. Well, then, we should just go away. Then we should just say you can't use race, don't worry about it."
Fisher graduated this year from Louisiana State University.
A ruling is not likely before early next year.
The current case is Fisher v. University of Texas at Austin (11-345).