"I would be more worried if he were asexual," Baer said of Petraeus. "It shouldn't distract from his legacy."

Retired Gen. James "Spider" Marks, for whom Broadwell once worked and who knows Petraeus, said he doubts security protocols were breached despite what seems an unlikely indiscretion on the part of Petraeus.

A senior U.S. intelligence official said an extramarital affair by a CIA officer is not automatically considered a security violation.

"It depends on the circumstances," the official said.

The official also said Broadwell did not have a security clearance from the CIA. Another official said Broadwell, who is an officer in the Army reserve, did have some kind of security clearance and that there are no issues with Broadwell having unauthorized access to classified information.

"I do not know how she got that information," said Sen. Dianne Feinstein, chair of the Senate Intelligence Committee, referring to Broadwell's October speech. "We should find out."

Feinstein said she would investigate why the FBI investigation that uncovered the extramarital affair did not notify oversight committees about its investigation into Petraeus after the bureau determined he was having a secret and risky extramarital affair.

Fran Townsend, a former Bush Homeland security adviser and CNN national security contributor, agrees the situation lends itself for the potential that security would be breached.

"Is there the potential? Yes of course. I think there's no indication that there's a problem here. There's no indication this involved classified material from him to her."

She also said she thinks that Broadwell is "playing the journalists' game."

"If she's in an intimate relationship with the CIA director, he's not her only source." And, she might have heard the October remarks, made at the University of Denver, in passing from someone else.

As investigators try to determine who said what to whom, it might end up as a "he said, she said." And as a result, nothing will be firmed down or resolved.

"I worry about the distraction that this now becomes for Congress and the media," Townsend said.

Tom Fuentes, a former FBI assistant director and CNN contributor, said there are always potential worries about having appropriate relations and the potential of spies. In this situation, Petraeus is "very experienced" and knows about security clearances. Broadwell has scores of military contacts and is an author and academic.

But what if someone burglarized her house, for example, and found classified information there. Could she be blackmailed?

"That's the worry," he said.