The Belgian duo had a 6-5 lead in the third set tie-break - but failed to convert its match point - and eventually lost the match.
Smith tried to inspire Evans by consulting Tim Henman, who, like the Birmingham player, had a game much more suited to fast surfaces but reached the French Open semi-finals in 2004.
They have not met since the 2010 final, which Serbia won 3-2 at home.
"We knew it would be hard for us, but we kept going until the end and did a great job", Mahut said.
On Friday, Kyle Edmund lost the first singles match 7-5 7-6 (8-6) 6-2 to world number 17 Lucas Pouille.
After a brief rally, Britain saved it when Benneteau netted.
Having won a 12th successive singles rubber in the tournament on Friday to set Serbia on their way to this weekend's quarter-final victory against Spain, the 12-time major victor is crucial to his country's hopes of returning to the final for the first time since 2013.
"We are a great nation who have not won this competition since 2001".
"Do we lack depth?"
Spotlight on Max Verstappen
Shocker: Nico Hulkenberg's prospects looked good ahead of the grand prix, starting from seventh with a damp start. It comes after a string of performance and reliability issues surfaced for Red Bull during pre-season testing.
"It's a damn sight better than it used to be but there's still work to be done".
If Britain can not call on the younger Murray brother, Smith has few options beyond the team he picked to face France. "And we will now have to do it the extremely hard way". "But we have retained our World Group status and that has to be a positive".
Steve Johnson and Jack Sock weathered a rough start to defeat Sam Groth and John Peers 3-6, 6-3, 6-2, 2-6, 6-3.
"A quarter-final, a win, a semi-final, a quarter-final - in decades previous you weren't getting that", said Smith after the doubles defeat.
"Nick's got a lot of self belief - it's going to take one hell of a performance to beat him", said Australia captain Lleyton Hewitt.
Simone Bolelli and Andreas Seppi of Italy defeated Belgium's Ruben Bemelmans and Joris De Loore to keep the Italian side alive in Davis Cup.
Leon Smith insisted a Davis Cup whitewash by France should not mask the progress being made in British tennis.
But, if that was unlikely, this was miracle territory - winning the doubles a formidable task even before considering Sunday's reverse singles.
Comments