Ivanovic headed in a Frank Lampard free-kick unopposed after 73 minutes as debate raged over whether the Chelsea right-back should even have still been on the pitch.
Referee Kevin Friend had booked the Serbia defender moments earlier for catching Christian Benteke with a flailing arm in an aerial challenge which infuriated Villa boss Paul Lambert.
All the attention was on Jose Mourinho's return in Sunday's defeat of Hull, but the self-proclaimed Special One wanted to shift the focus to his team, who endured an altogether more difficult test than on Villa's most recent visit to Stamford Bridge.
Villa were beaten 8-0 last December, but were in buoyant mood following the 3-1 opening day win at Arsenal and determined to claim the scalp of another title contender.
Antonio Luna scored late on at the Emirates Stadium and was on target again, only this time into his own net following a Brad Guzan save from an Eden Hazard shot.
Benteke, whose double accounted for Arsenal, scored his third goal of the season with the final kick of the first half to equalise.
It came after a dart down the left from Gabriel Agbonlahor which is the kind of play which could bring the forward into England contention again.
Then, just as Villa looked their most threatening, Ivanovic struck Benteke before breaching the visitors' defence as Mourinho's unbeaten home record in the Premier League was extended.
The Portuguese has never lost in 62 top-flight matches at Stamford Bridge and next Monday takes his side to Manchester United for an early title bout, where transfer target Wayne Rooney is sure to be the focus of attention.
Source: PA
Source: PA