10,000 bugs

I’m very fond of expressive and beautiful infographics and data visualizations, but for now, this is all I could come up with. Yesterday, we filed the 10,000th bug report in our issue tracking system. I don’t have any basis for comparison, as Kilgray is the first software development company I have worked for, but hey, this is a great number, isn’t it.
If you have no or little experience with software management, 10,000 bugs might look scary, but as far as I can tell, it is a fact of life: developers and even designers make mistakes, and there can always be new ideas to make something even better. Also, these 10,000 bugs are almost all fixed now, and are the product of several years of software development, covering all the functionality of every Kilgray product: no single user ever encountered more than 1% of this over the years. These 10,000 bugs also contain quite a few feature requests, because we use the same system to track those, but anyhow, it is still 10,000 cases of fixes and refinements to the “base” products (memoQ, memoQ server, and the APIs), the newer ones (qTerm, TM repository, and the upcoming web based translation module), as well as all the documentation, the Kilgray web site, localization, and our back-end system for product activation.

My colleagues have also worked some Excel magic to show the general trend of the amount of bugs in the system over time. It displays a definite acceleration, but it is quite consistent with the growth of the company, the number of products, and the amount of code. Also, a very important distinction is who is confronted with a defect: is it reported by the end user, or our own test team? Obviously, we aim to minimize the proportion of user-reported bugs. Even more important is the severity of each bug, especially if it is discovered by the user: is it just some “cosmetic” problem, or does it cause serious disruption or even destruction of some of the user’s work or data? Does it affect a small minority or a large amount of users? I can confidently say 5.0 has been a definite improvement over previous major releases in these areas. With this, we are not only patting our own backs: we got a tremendous amount of help and useful feedback from our beta testers, and users who went ahead and started using the release candidate builds of memoQ without hesitation. We can’t be thankful enough for that.

With every bug that we file, we record several important pieces of information like the severity of a bug, or its source (internal testing, pre-release testing, user report, etc.), so the data to follow the most important trends is in our hands. And, today more than ever, we are making very conscious and serious efforts to improve on every front of testing and quality assurance. The basic goal with this is better customer satisfaction and avoidance of (ideally) any serious bug induced stress for both our users and us. We realize that there is still a lot to learn for us, and as the company grows, we will need to adapt our strategies, so to get there, we will most probably rely on a partnership with a company that specializes in software QA and testing, as well as training in these areas. But, as with any major change, the serious results will take several months to show.

A moment like this can also be used to look back on our favorite and most hated bugs. Every now and then the bug tracking system becomes an unlikely source of fun and laughter. For example, just recently, when reporting a bug about a feature that was “allowed” on the user interface by mistake, I wanted to write “this should not be allowed”, but somehow, due to my sloppy typing, that became “this should be allowed” instead. The developer then didn’t ask anything, but went ahead and implemented that “missing” feature: this is why you can clone a remote translation memory into a local one today. There are also the occasional developer remarks that crack us up, like “fortunately, I have no social life at all nowadays, so nothing stops me from fixing this urgent bug for you on a Sunday evening”.

Of course, there were also some very disruptive and elusive bugs that we, and many of our users, absolutely hated. One recent example is a bug with “desktop document” synchronization that took us months to properly identify and fix, while it badly affected many people who were working with memoQ server based online projects. A lesson that I quickly learned is that, at least in our case, reproducing a bug (finding out the exact steps and circumstances that lead to the bug) can be much more difficult and time-consuming than fixing it. I hope that maybe in half a year, I can confidently promise that things like this will never happen anymore, or there will even be no need to talk about this topic, as this type of serious disruption gets completely forgotten as a thing of the past.