Speaking at a press conference in Brussels, Rogge said the main concern for London 2012's organisers should be to avoid leaving behind "white elephants".
Rogge told the BBC: "If the best solution is to transform the track into something else then we would be in favour of that. We had the same situation in Atlanta where the Olympic Stadium was changed into a baseball stadium, which kept an interest for sport."
It had previously been believed the IOC wanted London 2012's organising committee (LOCOG) to keep its bid promise about having athletics at the heart of its post-Games plans for the stadium.
Despite this admission, Olympic Delivery Authority (ODA) chairman John Armitt said it was still the intention for London 2012 to leave an athletics legacy.
One potential anchor tenant, Tottenham Hotspur, today fell out of the running for taking over the London 2012 stadium after the club announced plans for a new stadium adjacent to its existing White Hart Lane ground.
LOCOG and the ODA would struggle to attract a Premier League club as the stadium's post-Games capacity will be just 25,000, following the removal of a temporary structure supporting 55,000 seats. East London side Leyton Orient and rugby team Wasps are among the current favourites to use the venue post-2012.