The Shape of Everything
A website mostly about Mac stuff, written by Gus Mueller
» Acorn
» Retrobatch
» Twitter
» Micro.blog
» Mastodon
» Instagram
» Github
» Maybe Pizza?
» Archives
» Feed
» Micro feed
September 1, 2009
(This post is from my old, old, super old site. My views have changed over the years, hopefully my writing has improved, and there is now more than a handful of folks reading my site. Enjoy.)

I thought I'd share how I do beta testing for my apps. It's pretty simple, just a couple of rules to follow:

1) Don't bother giving out a beta with known bugs, or at least obvious bugs. Your testers are going to find the bug, write up a nice report, and you're going to respond with something like "yea, that's a known bug". And now you've wasted their time, your time, and you've got a tester that probably doesn't want to help out anymore.

(Side note: this is what pisses me off the most about Apple's bug reporting system. I'm going to find the bug, write it up, and a month later it'll get marked as a duplicate. That saps all the motivation out of reporting the bug for me.)

2) Give out the beta in waves. If you have 100 folks that are interested, give it out to the first 20. Fix all the bugs they find. Give it out to the next 20, fix those bugs, etc.

This keeps everyone from finding the same bugs at once, and keeps your beta group full of fresh testers. Most people tend lose interest over time in testing your Shiny New. Unfortunately this tip really only works if you're doing a private beta and you can get a significant amount of testers.

3) Reward your bug reporters in little ways if you can. If you find a bug in a beta release of VoodooPad or Acorn, you get a free upgrade or new license if you don't have one already. And lately I've been putting people's names next to the writeup in the release notes.

I have a hidden pref in Acorn called "showAllBlendModesForKevin", which when turned on triggers a certain feature that Kevin B. requested. I think folks like this (and I shamelessly stole this idea from Brent Simmons).

4) Auto updates, and frequent builds. In beta releases I either have the update feed pointing to my latest builds (or private builds if it's a private beta), or I provide instructions on how to do this. Then whenever I fix a bug (or a couple of bugs in a session) I run a script that commits the source, pulls it out in a new location, builds, packages, and uploads everything to the beta site. I'll do this between 5 and 20 times a day.

I always want testers using the latest build unless I'm doing some major reshuffling.

. . .

So yea, I'm looking for testers for the next big release of Acorn. If you promise to keep secret about it (it's a private beta!), then send an email to gus@flyingmeat.com with your system info (Mac OS X version- 10.6 folks preferred), what your favorite color is (plaid doesn't count) and I'll send you the download links and release info.

Why a private beta? It's a lot more controlled for me, features will and do change before final release, and I can't handle a billion bug reports at once.