Now the once exclusive Kaaiman's Grotto is a squatter camp and home to a Christian mystic who believes God appointed him as its caretaker.
Transnet, the rail parastatal, says the cave is state property and is no place for trespassers.
The site is earmarked for redevelopment and features on a Western Cape government list of potentially lucrative tourist attractions.
But the self-proclaimed Kaaiman's Caveman, also known as Clifford, says God will not allow anyone else to move in: "I know this is the house that God has got me," he told the Sunday Times during a tour of the cave.
"When I asked (God) why, he said, 'I brought you here to sit at my feet.' This is his house. I am like his child. I can't just bring anybody in," said Clifford, who spent four years in Bible school before moving into the cave.
The 49-year-old former Capetonian moved in four-and-a-half years ago, shortly after a landslide closed the only access to the restaurant - the famous Outeniqua Choo-Tjoe railway line between George and Wilderness.
After nearly starving to death - "for days all that went down my throat was water" - he attracted sympathy from passers-by, some of whom left food and money.
His eccentric renovations and artwork inside the cave soon elevated the site to a local tourist attraction and place of pilgrimage. He featured in a German documentary, prompting a German tourist to make a trip to meet him after seeing him on TV.
Clifford also had visits from his family, including his ageing mother, who had previously disowned him. "I carried her down here on my shoulders. She had a look around and said, 'Now you are really living like a bergie,' " said Clifford. "Before she died she stayed here for three days. She was mesmerised by the place," he said.
In its heyday, the restaurant once featured chefs from the nearby upscale Fancourt golf estate and hosted corporate clients. It is now divided into a restaurant deck with tables and chairs, and a maze of small rooms decorated with shells, driftwood, and bric-a-brac.
There is a fully equipped kitchen, a "honeymoon suite", and a study and library area, where Clifford does daily Bible study.
The soft-spoken caveman, who wore Tintin pants when we visited, has adopted three friends to help maintain the cave, and to fish at the river mouth.
One of them, an unemployed waiter from Johannesburg, said the cave was a refuge from hard times. "Things didn't work out for us. Here we can get back on our feet," he said.
The cave dwellers have earned the blessing of the current cave lease-holder, businessman Johan Coetzee and his two partners, who still hope to develop the site.
Coetzee said Clifford contacted him when he first moved in.
"We had a very good conversation. He wanted to go and live there and felt that God had guided him to do that. I told him, 'I have no problem with you staying there.'"
Transnet has yet to decide on the future of the railway line, which many say is now beyond repair. Large sections of the track are still covered by rubble and soil from the landslide five years ago.
The Western Cape government has renewed its claim to the historic site. Tammy Evans, spokesman for provincial Tourism Minister Alan Winde, said: "We strongly believe that the Choo-Tjoe must (stay) running as it is one of our precious few heritage items."
Transnet spokesman Mboniso Sigonyela said: "Transnet has not granted permission for the tenancy of this property. Occupation of any Transnet property without a lease is deemed illegal and would constitute trespassing."