Chad Griffin, president of the Human Rights Campaign, said Giglio's decision to withdraw from the inauguration was the right one.

"Participants in the inaugural festivities should unite rather than divide. Choosing an affirming and fair-minded voice as his replacement would be in keeping with the tone the president wants to set for his inaugural," Griffin said in a statement.

Giglio represents a new type of evangelical leader who "doesn't like to get involved in the culture war because it blurs the larger points he wants to make," said Michael Cromartie, the vice president of the Ethics and Public Policy Center in Washington.

"What I want to remind his critics is he's not being named to a Cabinet position," Cromartie said. "He was being asked to deliver a prayer. All sorts of people deliver prayers who we don't agree with on a number of issues."

"It's unfortunate that this kind of political correctness doesn't allow people who are doing great work to pray at inauguration," he added.