BBCWork-related benefits were being claimed last year by more than 370,000 migrants who originally came to the UK to work, study or visit, the first research of its kind has shown.
The government matched benefit, border control and tax records from 2011.
A look at a sample of 9,000 claimants of certain nationalities found 2% were illegally claiming benefits.
Employment Minister Chris Grayling said most claims were perfectly legitimate and the full picture was not yet clear.
BBC correspondent Naomi Grimley says the government admits it found only a tiny number of cases where migrants were claiming illegally, but the research is likely to prompt a wider debate about whether the welfare system has become too generous and attracted so-called "benefit tourists".
Before the coalition government came to power the nationality of benefit claimants was not recorded.
The first detailed "data-matching" was carried out by the Department for Work and Pensions (DWP), the UK Border Agency and HM Revenue and Customs.
The research found:
As of February 2011, 371,000 people - out of a total of 5.5 million - claiming working-age benefits had been non-UK nationals when they first registered for a National Insurance Number
Of these, 258,000 were from outside the European Economic Area (non-EEA)
Of this figure, 54% were now British citizens and most others had an immigration status that meant they were eligible for benefits
What the government calls working-age benefits include jobseeker's allowance, income support, carer's allowance and disability living allowance, among others.
A follow-up sample looked at 9,000 non-EEA nationals, three-quarters of whose records were able to be matched.
Continue reading the main story
“
Start Quote
I'm very glad that ministers now responsible are saying in turns that the immigration system is in a serious mess”
End Quote
Sir Andrew Green
MigrationWatch UK
Of those matched, 98% were entitled to benefits but 2% appeared to have "no lawful immigration status".
The "small number" of cases is being investigated, the DWP said.
The government also said it was looking at ways to record nationality "at source" and at better tracking of the immigration status of claimants.
Mr Grayling admitted the vast majority of those claiming were entitled to the benefits, and denied the government was scaremongering.
"I think it's really important for the credibility of our benefits system... that we should understand the mix of people who come from other countries who are claiming benefits," he told BBC Radio 4's Today programme.
"We are now going to go through all the people who we've not being able to identify and we're going to repeat that exercise across the full 250,000 to have a system in which people can have confidence."
The employment minister said he wanted to reduce net migration and ensure the UK system did not attract "benefit tourists".
Earlier this month the government's official advisers on migration said there was a link between immigration from outside the European Union and job losses among UK workers.
The research was carried out by looking at tax, border control and benefits records
The Migration Advisory Committee said there were 23 fewer UK jobs for every 100 migrants from outside the EU.
But a separate report from the National Institute of Economic and Social Research (NIESR) said immigration had had little impact.
MigrationWatch UK, which campaigns for tougher controls on immigration, said the evidence was unclear but something was "clearly not right".
Chairman Sir Andrew Green told Today: "I'm very glad that ministers now responsible are saying in turns that the immigration system is in a serious mess."
He said the sheer scale of immigration put "huge pressure" on public services and there had been no link until now between the immigration and benefit systems.
It was "absolutely absurd" that people from Romania and Bulgaria have been claiming benefits by claiming to be self-employed because they sold the Big Issue newspaper, he added.
Keith Best, who used to run the Immigration Advice Service, said it was "very, very difficult" for migrants to claim benefits in the UK.
He said you have to have "status" to claim and you were not entitled if you came in as a student or on a work permit, for example.