1ft Startup

So, you’ve organized a clever build system that can manage a thousand different SKUs of your mobile software with the greatest of ease, and now you have to stop putting off the fun part: testing. How the heck do you really test on dozens of devices?
Your first (and probably most important) stop should be a few things to consider before even loading up a phone:

1. Make sure you have all of the features and behavior you want in the app. If you make functional changes to your game/app halfway through porting, you’ve invalidated all the fun work you’ve done and you get to do it again. Think outside of just your own requirements! Be doubly sure that your app also has all the features required by the carrier/vendor/external test team you’re porting for.
2. Be very careful choosing the phones you want to port to. Each phone will have its own quirks and will be less easy than you think it will, so choose carefully. If you can, pick newer devices, and carefully screen external demands for discontinued models (requirements get old fast, so be sure you’re not just being handed last quarter’s list). While some consumers are likely have the discontinued handsets out in the wild, people are less likely to be buying content than on their brand new Nokia. Also, old devices are going to be a pain to find yourself for testing.

Once you’ve trimmed your handset list and settled all your features, you’ve got basically three levels of quality testing, each with their own tradeoffs:

1. Beta programs run by mobile communities. These programs allow you to upload your code to a select group of community members and allow them to try out your code. You get helpful feedback, they get early access to the coolest new stuff; everyone wins. Here are a couple of them:
   • GetJAR Beta Testing
   • Vodafone Betavine

   I’ve personally used GetJAR’s beta program and had surprisingly good results. One reason for this is that GetJAR is actually very strict about users who sign up, and ban users who download your game/app and don’t give any feedback. Some users were extremely helpful; GetJAR delivers a spreadsheet at the end of the test program with handsets and basic details about what worked and what didn’t. Occasionally, you’ll get the user’s contact info to follow up.
   
   This is a great way to go for free, coarse app testing. This is a terrible way to go for carrier-grade testing, because you won’t get enough detail, enough devices, you’re unlikely to get to repeat tests you need for bugfixes, and you have to entice users to download your app in the first place. But, if you’re a hobbyist with a budget of about $0, and are not taking this to a carrier/deck with serious porting requirements, it’s a very good (and free) first stab.

2. DeviceAnywhere. These guys have saved our low-end lives! Here’s how it works:
   
   MobileComplete (the company behind the operation) has a lab full of handsets that they let users lease remotely, by the hour. That is, they’ve hotwired a few hundred phones to the internet and let you control them remotely, and yes, these are the real devices being controlled over the real net! It’s a clever business and a neat trick — For those of us who can’t change even set our clocks for daylight savings time without electrocuting ourselves, it makes our head spin.
   
   I’ve found the testing process pretty smooth for most cases. Some devices aren’t always available (all their users are pulling from the same pool), but they open up pretty quickly. You will experience some problems with sound on some devices, and rapidly-updating features like video streaming will be laggy, but this will be a very good verification for 98% of what you’re up to.

3. Brute force. This means either:
   • Buying yourself every phone you need to test
   • Paying a porting house to do the work for you.

   Both of those options are things you want to give a lot of thought to before you head toward them. Porting shops will cost you at least a few hundred dollars per phone, and that can stack up after 50-60 handsets. My personal opinion is that if you’re in this business seriously, gaining the porting toolchain and expertise in-house will be very good for you in the long run.

All in all, porting is one thing that really hurts about mobile from an engineering point of view. It’s hard to estimate, resource-intensive, and very hard to manage and maintain. I’ve heard one developer recently estimate this as taking about 100%-150% of the initial development effort per project, which feels about right to me.

One last thing about this: Porting is a process, not a destination! You’ll inevitably be a huge success once you have widespread support for your snazzy new widget, and want to continue support for phones in the future months and years… So in spite of the urge to make spaghetti to get it to work, it’s even more critical than usual to keep the code and build process clean and repeatable!