The masts, which are scheduled to go up around Britain next year (we'll believe it when we see it), will help bring the next generation of mobile internet to gadget users across the nation.
However, it is expected that the activity of the structures will negatively impact the Freeview reception in 900,000 homes.
Mobile networks have accrued a fund of £180m to pay for £10 signal boosters for affected homes, but there are fears some customers will be forced to pay installation costs of £150.
Members of the House of Commons culture, media and sport select committee expressed concerns that residents will be left to foot the bill.
Committee chairman John Whittingdale said there "will be people who cannot watch television because mobile companies are broadcasting on 4G" and would have to pay through "no fault of their own."
Lib Dem MP Adrian Sanders added: "People in my constituency are looking at this as a government stealth tax. These companies are making profits year on year, on year."
Vaisey defended the measures, pointing out that vulnerable customers would get the installation for free.
Although the problem will not affect cable or satellite users, the £180m fund does not cover homes that use Freeview for the second television set.
"Some people take the view that every set should be covered but it's not the position we take," said the Culture Secretary.