The masts, which are scheduled to go up around Britain next year (we'll believe it when we see it), will help bring the next generation of mobile internet to gadget users across the nation.
However, it is expected that the activity of the structures will negatively impact the Freeview reception in 900,000 homes.
Mobile networks have accrued a fund of £180m to pay for £10 signal boosters for affected homes, but there are fears some customers will be forced to pay installation costs of £150.
Members of the House of Commons culture, media and sport select committee expressed concerns that residents will be left to foot the bill.
Committee chairman John Whittingdale said there "will be people who cannot watch television because mobile companies are broadcasting on 4G" and would have to pay through "no fault of their own."
Lib Dem MP Adrian Sanders added: "People in my constituency are looking at this as a government stealth tax. These companies are making profits year on year, on year."
Vaisey defended the measures, pointing out that vulnerable customers would get the installation for free.
Although the problem will not affect cable or satellite users, the £180m fund does not cover homes that use Freeview for the second television set.
"Some people take the view that every set should be covered but it's not the position we take," said the Culture Secretary.
Sign up to receive daily breaking news, reviews, opinion, analysis, deals and more from the world of tech.
A technology journalist, writer and videographer of many magazines and websites including T3, Gadget Magazine and TechRadar.com. He specializes in applications for smartphones, tablets and handheld devices, with bylines also at The Guardian, WIRED, Trusted Reviews and Wareable. Chris is also the podcast host for The Liverpool Way. As well as tech and football, Chris is a pop-punk fan and enjoys the art of wrasslin'.