While they were working out the best way to collect feedback for Internet Explorer 9, the IE team looked at the bug reporting tools used by Mozilla and WebKit to see if they were missing a trick themselves.
They made some changes like not making people get invited to report bugs and "we tweaked the forms a little bit to pick up some of the feedback we were missing that tended to make bugs less actionable," IE general manager Dean Hachamovitch told TechRadar.
But they also spotted that some bug reports for other browsers hang around for a long time (for years in some cases) without being fixed or even accepted as confirmed bug reports; and while the IE team closes every bug reported (whether by fixing it or saying it won't be fixed), as of this week Mozilla's Bugzilla system reports 12,779 unconfirmed bugs and WebKit has 2,616 unconfirmed bugs.
IE Test Manager Jason Upton told us he filed one of those languishing background reports himself. "There's a bug that I logged a couple of years ago, based on some feedback from our OEM guys and it sat there for 15 months.
"In the last couple of months people have talked about it and said 'yeah, we should probably fix it' but it's not getting much traction. There are a lot of bugs sitting there without getting traction; maybe that's the nature of an open source project but that's not the style of project we have."
Fair comparison?
That's certainly a lot more than the 533 bugs reported publicly for the IE9 Platform Preview so far (even allowing for the fact that Microsoft asks users to open a new bug even if they had previously reported it in IE8); but is it a fair comparison even to older browsers?
Dean Hachamovitch says the team "pulled the data and tried to do an apples to apples comparison" but pointed out that fewer bugs ever reach public IE testers to report in the first place.
"The bugs you see on Connect are bugs that are logged after the professionals in this hallway have their testing - so there's a smaller pool of bugs."
Upton's calculations on the IE blog show that for IE8, the IE team found five times as many bugs as general users (and a lot more bugs than other people inside Microsoft) and the bug reports the internal testers filed were six times more likely to have enough information in to fix the bug.
Feature or bug?
What about people who filed what they thought was a real bug, only for the IE team to treat it as a product suggestion and close the bug?
"Clearly these bugs raise visceral emotions in people who log them," agrees Hachamovitch. "From a customer point of view there's a no difference between a bug and a suggestion and we respect that."
But when you get beyond the definition of a bug as a product feature that doesn't work the way it's supposed to work and into the question of features that don't work the way people would like them to work, he suggests it becomes a question of perspective.
"We use many, many data sources from customers to inform what we build and how we build it," Hachamovitch told us.
"When you connect data from hundreds of millions of users around what they actually do and how they actually do it that is extremely powerful." The IE team is also balancing what a lot of different people are asking for; your perfectly reasonable feature request could be something that causes problems for people who care about security or accessibility.
It's not about the numbers of bugs or whether Mozilla and Webkit had more or less than IE, though. "The larger story here," Hachamovitch told us, "is around responsiveness and who takes responsibility. What we think of as pretty standard examples of things that were valid, reasonably important and well logged bugs sat around for a long time [in the other browsers]."
