"We get reports that some people follow the pilgrims group, intimidate, abuse and even attack them with stones," Sri Lanka Deputy High Commissioner R K M A Rajakaruna told PTI in Chennai.
The Sri Lankan government evaluated the situation as the people "appealed to us to arrange to send them back to Sri Lanka," he said. We thought it would be advisable to arrange their quick eviction from here," the envoy said.
In New Delhi, an External Affairs Ministry spokesman said India would take all measures to ensure safety and security of Sri Lankans in the country.
"I wish to convey that the Government of India, in close consultation with the concerned state governments, has taken and will continue to take all measures to ensure the safety, security and well-being of Sri Lankan dignitaries and visitors to India, including to Tamil Nadu," he said.
"From childhood, we have been speaking Tamil. For the past 15 years, we have been visiting this place. We are very sad..." said one of the pilgrims.
The Lankan nationals had worshipped at Poondi Matha shrine on Monday evening on their annual pilgrimage when activists of Naam Tamizhar Iyakkam headed by film director Seeman, Viduthalai Chiruthaigal Katchi and P Nedumaran-led Tamizhar Desiya Poduvudamai Katchi, staged a protest, asking them to go back.
The incidents came even as Sri Lanka on Monday issued a travel advisory, asking its citizens not to visit Tamil Nadu until further notice in the wake of "increasing number of instances of intimidation" of Sri Lankans in the state.
The developments also come close on the heels of Tamil Nadu Chief Minister Jayalalithaa on Sunday ordering sending back a football team from Royal College, which played a friendly against Customs department in Chennai on Friday.
Jayalalithaa had also directed that eight students of a Lankan school who had come for a tournament with a city-based school and their coach be sent back.
Striking a discordant note against the action of his arch rival Jayalalithaa, Dravida Munnetra Kazhagam chief M Karunanidhi said his party is not averse to India's cultural and sporting ties with Sri Lanka, but to training of Lankan defence personnel by India.
"All that we can insist is there should be no military training to Sri Lankan personnel in India as the military was responsible for death of civilian Tamils (during the armed conflict). That is what we have been striving for and even stoutly opposing in Parliament," Karunanidhi, whose party is a key United Progressive Alliance ally, said.
Expressing concern at the hostility towards the people from Sri Lanka, Communist Party of India-Marxist said, "Whatever the differences and opposition that exists regarding the Sri Lankan government's approach to the Tamil question in Sri Lanka, nothing should be done to disturb people-to-people relations".
The All India Anna Dravida Munnetra Kazhagam, DMK and some other political parties have joined the chorus of opposing India training Lankan military personnel, alleging the island nation's forces had committed war crimes against Tamils in the last phase of 2009 war against the Liberation Tigers of Tamil Eelam.