Steven Pressley headed home from close range in the 89th minute and Kenny Miller booked their semi-final slot with a fine low shot from the edge of the box in stoppage time.
Manager Gordon Strachan told BBC Radio: "At 1-1 I thought, 'that will do us', but the players themselves said 'no, that's not enough for us - we're off to score the winner.'
"So I am glad they are on my side. I was desperate for a draw with five minutes to go...especially after a big game on Tuesday (with AC Milan in the Champions League)."
Caley took the lead when Graham Bayne prodded the ball over the line from six metres following a good run and low cross by Barry Wilson on 19 minutes.
Celtic struggled to create chances although Japanese midfielder Shunsuke Nakamura blasted an angled, rising shot against the left junction of post and bar in 54
Caley manager Charlie Christie said: "It is a sore one to take and I just feel so sorry for the players. They put so much into it only to get nothing out of it.
"A replay would've been bad enough, we would have been disappointed enough with that. But to be out the Cup is a real, real sore one."
Celtic join Dunfermline and Hibernian in Thursday's semi-final draw.
Dunfermline won 2-0 at home to second division Patrick Thistle on Saturday while Hibernian went through with a 2-1 victory at second division Queen of the South.
Motherwell host another second division team, St Johnstone, in the remaining quarter-final clash on Wednesday.
