Oh Hai!

Joystiq: The (not so) stringent world of video game testing There have been a lot of complaints about publishers rushing buggy games out so they can hit a certain date, knowing they can update them via patches over the increasingly popular online gaming services. So why not try and address things in the testing phase of the game? Karla Starr, a reporter at The Seattle Weekly actually took a job as a tester with a game company, and went inside the the belly of the beast to bring us a real insider’s view of game testing.

Well no shit. I went to GDC and was… well, appalled by the view of Test and the views of testers in the industry at large. There were a couple of companies that were doing it “right”. During one of the round table sessions and I expressed my dismay at the majority of what I heard from other testers, I was told “Oh, no wonder… you’re with Microsoft.” I took it as a compliment, but it was an astute observation.

What is it that Microsoft does that’s right (and made us stand out at GDC)?

In all honesty, I’m very familiar with both sides of the QA/Test coin. When I got out of the hardware side of the industry and shifted to software, I started as a Software Quality Assurance analyst for a little yellow box company. We were working on a rapid application development tool that was pretty posh for it’s time. Actually, I think it did things that current RAD tools still can’t do… I don’t know if that means were that advanced or that it was a pointless feature… Hm. Anyway, while I worked there, there was some good and some bad in the QA discipline, but it’s also what I see as “wrong” in some test departments in today’s day and age. Lemme ‘splain what I mean by describing my first job and my current test job.

The First Job:

The good: QA was involved in all aspects of the product. We helped write product specs, design features, QA product specs, work with marketing, educate customer service… we also tested the shit out of the product. The 1 to 1 Dev/Test ratio that we worked to maintain was a boon. The bad: 16 hour days for weeks or months at a time though, just to keep up. We had no large amount of automation – we had some that one of the staff worked on but it was lightweight compared to today’s ideal automation suites. We viewed ourselves as the Gate Keepers of the product’s release.

The Current Job:

The good: Test is involved in all aspects of the development process. We help write product specs, design features, test product specs for missing functionality, write Test specs, design our test plans early, and interactively work with our Dev and PM teams for accurate reporting of status, features, and schedules… we test the shit out of the product and we rely on automation in all parts of the process, particularly for Build Verification Tests and regression testing. We try to maintain a 1 to 1 Dev/Test ratio. We are not the Gate Keepers. The bad: long hours during crunch times. Automation can sometimes offer a false sense of security: it takes vigilance to watch out for false positives… after all the tests are only as good as the test code!

While at GDC, I found that the majority of the testers I talked to were living the description of “The First Job” with very, very few living “The Current Job”. From my discussions with people, a tester’s biggest concern was “how do I get respect from dev?” or “how do I test surprise features?” or “how do I work if Test reports to [marketing/dev/management]?” At one point I asked a group of testers “Who views themselves as Gate Keepers rather than Reporters of Quality?” 60 out of 64 people said Gate Keepers, two people being from Microsoft and another two from a game studio that also “gets it”.

What’s wrong with using Test as the “Gate Keepers” for game? A lot, actually. The power to release any software should never be left to one discipline group. Dev’s would ship on compilation and maybe after a run of unit tests. Test wouldn’t ship until every single bug was fixed, and while that sounds like a great idea consider how many cars would be on the road if we waited for a car that never broke down and that would never be in an accident… right, we’d never have a car. PM is the best hope of a balanced release, but they also rely on status reports from Dev and Test. Either way, there’s also the business side of things to consider.

The biggest challenge QA faced – and what most game testers continue to face – is the simple fact that business people have the final say so. This is why I – and I believe most teams at Microsoft – see Test as Reporters of Quality. Test is testing a product. They generate reports of what’s working and what’s not; they are also very vocal and open about how and what they’re testing. They will tell anyone that will listen what the state of the project is at any given time. If the business teams know the product is in shambles and decides to ship anyway, it’s not Test’s fault. In fact, while working in QA, we called the biggest feature for our 1.1 release [circa 1994] “now with error handling” because the business team completely ignored the QA team and shipped 1.0 too early.

The major difference between the two is how Test feels about their work. If you believe you have the power and responsibility to stop a product from shipping, and you get overwritten for any reason, you’ll feel it in your gut. Hurt. Angry. Useless. Impotent. If you are accurately reporting the quality of a product, and they ship it anyway, you can still feel satisfied that you’ve done your job. You’ve told them it sucks, you did your job, what happened after that isn’t in your control.

From what I’ve seen at Microsoft, they listen to Test and take that feedback into consideration which shipping product. For the teams that I’ve worked on, Test doesn’t have to worry about getting respect from Dev; developers want Test there to find bugs and make a better product. There are very few feature surprises; PM and Test will help keep that aspect of the project under control. Test is part of the R&D Triumvirate, which is good: we’re given the same voice as Dev and PM which works very well in software development.

The article referenced by Joystiq takes a look at a different part of the process: the play tester, which is not found in “regular” software testing. I equate this to some of the specialized testing that web teams have to do: no amount of automation is going to tell you that a collection of pink letters on a light blue background looks like ass – that requires human eyes. Other applications have to worry about other specialized testing too – we still can’t automate tests for a good user experience. So play testing is something that’s unique to the gaming industry. And remarkably, it has little to do with the rest of this post… Joystiq’s observations just sparked off this observation that has been rattling around in my head for a few months now. Hell, I may have even blogged about it before and forgot that I posted it, but the truth is that I believe it needs repeating every now and again.

Does that mean that I’m supposed to be living Grandma’s Boy? Um. No… although some parts of it would be interesting…


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.