Bourgoin recorded their first win in this season's Heineken Cup, beating Ulster 24-17 having trailed 17-6 after half an hour.
Second-half tries from Arnaud Tchougong and speedster Jean-François Coux sealed the deal for the French, with Benjamin Boyet put the final nail in the Ulster coffin with a penalty from the last kick of the game.
Ulster proved there is life after Mark McCall with a battling display in their defeat.
The Ulstermen were rocked by McCall's resignation in midweek, with the 39-year-old citing his side's poor start to the season - including the Heineken Cup defeat to Gloucester - as his reason for stepping down.
But how the former chief must have wished Ulster had played like this in their opening matches of the 2007-08 campaign.
With the imperious Paddy Wallace pulling the strings the Irish province ran in classy tries through Kieran Dawson and Simon Danielli, and really Ulster should have scored more - and gone on to win.
But Benjamin Boyet's boot kept Bourgoin within touching distance, before second-half tries from Arnaud Tchougong and Jean-Francois Coux snatched victory for the home side.
It was still the type of performance from Ulster that will certainly put acting coach Steve Williams firmly in the frame for the full-time role.
Although Bourgoin may have struggled in the Heineken Cup - they have never won the Top 14 or French Cup either - they are fearsome force at the Stade Pierre Rajon and have two past victories over Ulster to their name.
And they proved how dangerous they are as they overturned a 17-9 half-time deficit to win.
A rip-roaring opening five minutes set the tone for the match, with both sides showing their intentions to attack at every opportunity.
In fact it took well over two minutes before the first line-out of the match, with a refreshing running game adopted.
Bourgoin certainly enjoyed the better of the opening exchanges with Boyet providing the impetus in the backs and a powerhouse pack attempting to hammer holes in the Ulster defence.
However, it was Ulster that registered the opening score with a moment of pure brilliance. Things had looked bad for the Irish when a massive tackle from Julien Frier on Roger Wilson saw the ball bouncing loose.
Tommy Bowe kept his composure to scoop up the ball and launch the counter. Having beating the first line of defence with his arcing run, Bowe found Wallace coming through at pace.
The Ulster fly-half skipped past two tacklers before drawing the last man for Dawson to power over.
Wallace added the conversion, but Ulster were guilty of gifting Bourgoin a simple chance to get back into the game, with Boyet slotting over a penalty after a high shot from Wilson.
Ulster continued to use their backs to good effect, with Mark Bartholomeusz and Boss producing a couple of class kicks that deserved better.
However it was through Trimble and Wallace that Ulster looked the most threatening with another scything break on 22 minutes almost resulting in a second try.
The travelling fans,did not have long to wait though, with Danielli scoring just a minute later. Wallace led the counter-attack from his own half and drew the challenge of Rudi Coetzee before releasing the Scottish winger, who showed phenomenal pace to beat Boyet on the outside to score.
Wallace converted, but again Ulster gifted Bourgoin a passage back through the boot of Boyet - with a silly offside seeing the French fly-half slot over three points.
A collapsed maul saw Wallace kick Ulster into a 17-6 lead but Boyet had the last word just before break with his third penalty.
There was little doubt that the Frenchmen would come out from the interval firing on all cylinders and they didn't disappoint.
Ulster found themselves on the back foot, both in the backs and forwards, as Bourgoin stormed forward.
Coux really should have done better when Bowe fumbled Boyet's kick ahead, but the Bourgoin winger knocked on himself.
However, despite having the opportunities, Bourgoin were their own worst enemies, with silly mistakes giving Ulster routes to escape.
But Ulster ran out of luck on 57 minutes though when Frier ran through the challenges of Wallace and Bartholomeusz before slipping the scoring pass to Tchougong. Boyet converted to cut the Ulster lead to just a point.
A few nerves began to creep into Ulster game, with sloppy mistakes putting them on the backfoot, while Wallace even missed a shot at goal to release the pressure.
And the breakthrough for the French came on 64 minutes when Coux scored in the corner. The try came out of nothing as Boyet shaped for a drop goal but slipped. That fooled the Ulster defence, allowing the Bourgoin fly-half to slip through and find Coux free on the wing.
Despite only trailing by four points, the wind seemed to have been knocked out of Ulster and they lacked the cutting edge that had come so easily to them in the first half.
And it was Bourgoin who came closest to adding to their try tally, with Bryce Williams going close to driving over.
However, fittingly, the final word went to Boyet with his fourth penalty of the match to seal the win.
Man of the match: It really was a battle between two classy fly-halves. Paddy Wallace had a hand in all that was good about Ulster's play and haset up both tries, but he sadly went missing in the second period. Benjamin Boyet though remained a threat throughout though, landing 14 points and setting up the killer second try.
Tries: Tchougong, Coux
Pens: Boyet 3
Tries: Dawson, Danielli
Cons: Wallace 2
Bourgoin: 15 Florian Denos, 14 David Janin, 13 Rudi Coetzee, 12 Yann David, 11 Jean-Francois Coux, 10 Benjamin Boyet, 9 Morgan Parra, 8 Yann Labrit, 7 Augusto Petrilli, 6 Julien Frier, 5 Bryce Williams, 4 Damien Fevre, 3 Olivier Sourgens, 2 Benoit Cabello, 1 Arnaud Tchougong.
Replacements: 16 Jean-Philippe Genevois, 17 Pablo Cardinali, 18 Chris Wyatt, 19 Karena Wihongi, 20 Mickaël Forest, 21 Sebastien Laloo, 22 Matthieu Nicolas.
Ulster: 15 Mark Bartholomeusz, 14 Tommy Bowe, 13 Andrew Trimble, 12 Paul Steinmetz, 11 Simon Danielli, 10 Paddy Wallace, 9 Isaac Boss, 8 Roger Wilson, 7 Kieran Dawson, 6 Neil Best, 5 Carlo Del Fava, 4 Matt McCullough, 3 Bryan Young, 2 Rory Best (c), 1 Justin Fitzpatrick
Replacements: 16 Neil Brady, 17 Declan Fitzpatrick, 18 Ryan Caldwell, 19 David Pollock, 20 Kieran Campbell, 21 Niall O'Connor, 22 Rob Dewey
Referee: Wayne Barnes (England)