In October, the International Cycling Union stripped Armstrong of his Tour de France titles. Armstrong responded a few weeks later by tweeting a photo of himself lying on a sofa in his lounge beneath the seven framed yellow jerseys from those victories.
The International Olympic Committee said in October that it was reviewing evidence against him.
Armstrong competed in two events in the 2000 games in Sydney, Australia -- the men's individual time trial, where he medaled, and the men's individual road race, where he finished 13th. He is retroactively disqualified in both races, the IOC said.
"We have written to Armstrong asking him to return the medal" and informed the U.S. Olympic Committee, Mark Adams, another IOC spokesman, said Thursday. It's up to the U.S. committee to handle retrieving the medal from Armstrong, the IOC said.
The decision was made "in principle" at a meeting of the IOC executive board in December, Adams said. The committee did not act on the decision until it received confirmation from the International Cycling Union that Armstrong was not appealing that agency's decision.
The USOC was notified Wednesday that the IOC wants the medal back, USOC spokesman Patrick Sandusky said.
"We will shortly be asking Mr. Armstrong to return his medal to us, so that we can return it to the IOC."
Deadline for Justice Department to join lawsuit
Thursday was also the deadline for the Justice Department to decide whether to intervene in a federal whistleblower lawsuit filed by former Armstrong teammate Floyd Landis, a source familiar with the matter told CNN. The source said the department and Armstrong's lawyers were in talks regarding the case, but Armstrong lawyer Robert Luskin would not comment on any reported talks Thursday.
"We remain hopeful there will be an agreed resolution," Luskin told CNN.
Landis, the 2006 Tour de France winner, lost his title for doping and is among those accusing Armstrong of taking banned substances while leading the U.S. Postal Service cycling team. His lawsuit accuses accuses the team's former management of defrauding the government of millions of sponsorship dollars because the team management knew about the drug use and didn't do anything.
The New York Times reported earlier this week that Armstrong has been considering testifying against others to help his own situation, but Luskin denied that allegation.
"There is no truth at all to any suggestion that Lance is trying to limit his own liability by implicating others, and that remains true," he said.