Clermont's 77-game unbeaten streak on home soil came to an end as Castres claimed a hard-fought 22-16 victory in Saturday's Top 14 play-off at Parc de Sports Marcel Michelin.
The defending champions will now face Montpellier in Lille on May 17 in the semi-finals.
Out-muscled and out-thought, Clermont's season came to a screeching halt against a side playing pragmatic rugby, as les Jaunards failed to recover from a crushing loss to Saracens at Twickenham a fortnight ago.
Clermont led 9-6 at the break thanks to three penalties from fly-half Brock James but a big second half from CO saw the hosts buckle under pressure with centre Remi Lamerat's try around the hour mark proving the killer blow.
Castres had held the upper hand for large parts of the first period but the visitors paid the price for their indiscipline, finishing the half with 13 men after lock Rodrigo Capo Ortega and wing Max Evans both saw yellow.
With the scores level at 6-6, Capo Ortega was sent to the sin-bin for making a tackle without retreating 10m when Gerhard Vosloo tried to play a penalty quickly.
Jame slotted the three points from 50m out to put the hosts in front before Evans joined his team-mate in the bin for an intentional knock-on.
Clermont were unable to cash in on the two-man advantage and Castres scrum-half Rory Kockott's third penalty drew the sides level.
Evans returned just in time to see Kockott give his side the lead at 12-9 as Vosloo saw yellow for Clermont's repeated indiscretions at the breakdown.
Kockott was at the origin of Castres's try with a break from the side of a ruck. He found Capo Ortega in support before recycling to Lamerat, who crashed over from short range.
The South African scrum-half added the conversion and another penalty to put Castres 22-20 ahead going into the final ten minutes.
A desperate fightback from Clermont saw number eight Damien Chouly charge over in the dying minutes, but it proved too little to late to save Clermont's season.
Pens: James 3
Yellow card: Vosloo
Pens: Kockott 5
Yellow cards: Capo Ortega, Evans
Clermont: 15 Lee Byrne , 14 Sitiveni Sivivatu, 13 Aurelien Rougerie, 12 Wesley Fofana, 11 Naipolioni Nalaga, 10 Brock James, 9 Morgan Parra, 8 Damien Chouly, 7 Gerhard Vosloo, 6 Julien Bonnaire, 5 Nathan Hines, 4 Jamie Cudmore, 3 Davit Zirakashvili, 2 Benjamin Kayser, 1 Thomas Domingo
Replacements: 16 Ti'i Paulo, 17 Vincent Debaty, 18 Julien Pierre, 19 Fritz Lee, 20 Thierry Lacrampe, 21 Michael Delany, 22 Noa Nakaitaci, 23 Clement Ric.
Castres: 15 Brice Dulin, 14 Remy Grosso, 13 Romain Cabannes, 12 Remi Lamerat, 11 Max Evans , 10 Remi Tales, 9 Rory Kockott, 8 Antonie Claassen, 7 Yannick Caballero, 6 Piula Faasalele, 5 Rodrigo Capo Ortega, 4 Richie Gray, 3 Ramiro Herrera, 2 Mathieu Bonello , 1 Saimone Taumoepeau.
Replacements: 16 Brice Mach, 17 Yannick Forestier , 18 Christophe Samson, 19 Jan Bornman, 20 Cedric Garcia, 21 Daniel Kirkpatrick, 22 Seremaia Bai, 23 Mihaita Lazar.
Venue: Parc de Sports Marcel Michelin, Clermont
Referee: Romain Poite