1ft Startup

So, you’ve organized a clever build system that can manage a thousand different SKUs of your mobile software with the greatest of ease, and now you have to stop putting off the fun part: testing. How the heck do you really test on dozens of devices?
Your first (and probably most important) stop should be a few things to consider before even loading up a phone:

1. Make sure you have all of the features and behavior you want in the app. If you make functional changes to your game/app halfway through porting, you’ve invalidated all the fun work you’ve done and you get to do it again. Think outside of just your own requirements! Be doubly sure that your app also has all the features required by the carrier/vendor/external test team you’re porting for.
2. Be very careful choosing the phones you want to port to. Each phone will have its own quirks and will be less easy than you think it will, so choose carefully. If you can, pick newer devices, and carefully screen external demands for discontinued models (requirements get old fast, so be sure you’re not just being handed last quarter’s list). While some consumers are likely have the discontinued handsets out in the wild, people are less likely to be buying content than on their brand new Nokia. Also, old devices are going to be a pain to find yourself for testing.

Once you’ve trimmed your handset list and settled all your features, you’ve got basically three levels of quality testing, each with their own tradeoffs:

1. Beta programs run by mobile communities. These programs allow you to upload your code to a select group of community members and allow them to try out your code. You get helpful feedback, they get early access to the coolest new stuff; everyone wins. Here are a couple of them:
   • GetJAR Beta Testing
   • Vodafone Betavine

   I’ve personally used GetJAR’s beta program and had surprisingly good results. One reason for this is that GetJAR is actually very strict about users who sign up, and ban users who download your game/app and don’t give any feedback. Some users were extremely helpful; GetJAR delivers a spreadsheet at the end of the test program with handsets and basic details about what worked and what didn’t. Occasionally, you’ll get the user’s contact info to follow up.
   
   This is a great way to go for free, coarse app testing. This is a terrible way to go for carrier-grade testing, because you won’t get enough detail, enough devices, you’re unlikely to get to repeat tests you need for bugfixes, and you have to entice users to download your app in the first place. But, if you’re a hobbyist with a budget of about $0, and are not taking this to a carrier/deck with serious porting requirements, it’s a very good (and free) first stab.

2. DeviceAnywhere. These guys have saved our low-end lives! Here’s how it works:
   
   MobileComplete (the company behind the operation) has a lab full of handsets that they let users lease remotely, by the hour. That is, they’ve hotwired a few hundred phones to the internet and let you control them remotely, and yes, these are the real devices being controlled over the real net! It’s a clever business and a neat trick — For those of us who can’t change even set our clocks for daylight savings time without electrocuting ourselves, it makes our head spin.
   
   I’ve found the testing process pretty smooth for most cases. Some devices aren’t always available (all their users are pulling from the same pool), but they open up pretty quickly. You will experience some problems with sound on some devices, and rapidly-updating features like video streaming will be laggy, but this will be a very good verification for 98% of what you’re up to.

3. Brute force. This means either:
   • Buying yourself every phone you need to test
   • Paying a porting house to do the work for you.

   Both of those options are things you want to give a lot of thought to before you head toward them. Porting shops will cost you at least a few hundred dollars per phone, and that can stack up after 50-60 handsets. My personal opinion is that if you’re in this business seriously, gaining the porting toolchain and expertise in-house will be very good for you in the long run.

All in all, porting is one thing that really hurts about mobile from an engineering point of view. It’s hard to estimate, resource-intensive, and very hard to manage and maintain. I’ve heard one developer recently estimate this as taking about 100%-150% of the initial development effort per project, which feels about right to me.

One last thing about this: Porting is a process, not a destination! You’ll inevitably be a huge success once you have widespread support for your snazzy new widget, and want to continue support for phones in the future months and years… So in spite of the urge to make spaghetti to get it to work, it’s even more critical than usual to keep the code and build process clean and repeatable!

One of the major differences between mobile code and desktop code is the severe number of SKUs (that is, variations on the build) that you have to support to make a dent in the customer base. Device support is a truly a long tail; after a couple hundred devices you’re getting close to about 80% of the market.

The biggest problem with J2ME in this case is that, unlike the desktop/PC world of Java, the VMs are built by MANY groups with different interpretations of the spec. Even if that weren’t the case, you’re trying to support some very different hardware characteristics: memory, screen resolution, network limitations, storage, etc.

All this means is that if you want your app to be a success, you’re going to adapt your code to support hundreds of phones.
Ugh.

What’s a developer to do? Here are a few basic points that work for me:

• Stay Organized. This should go without saying, but make a concerted effort to keep your builds clean, repeatable, and stowed away well. Someone will be asking you to send them the latest working build for their Motorola C290, and you should always have it handy. Do one of these:
  • Keep a folder heirarchy for your compiled files. Mine looks like:

    ->Builds
       -><app name>
           ->Version (e.g. 0.9.4)
               ->Screen Size (e.g. 176x220)
                    <standard build>
                   ->Custom build 1 (e.g. Nokia Series 60)
                       <build>
                   ->Custom build 2
                    ...
    

    For a solution like this, you’ll need a handy way to look up capabilities to help you dig out the right build for a handset.

  • Build/adapt a simple CMS. If you’ve got some time, this is the best option you’ve got (we’re working on a new one we may be able to share with you, stay tuned). You’ll thank yourself later.
• Write reusable code. Write with the intention of building what you write into a framework. OK, this will probably net you a few more KB in code, but it will save you from hurting yourself over and over again. Write as few new lines as you possibly can in a new app. Every time you fix a handset-specific bug, you’ve fixed it for everything you ever write in the future. In fact, if you’re really pathological, you can extract basically everything out of most apps into your own scripting/markup language and build your apps without writing a single line of real J2ME code. Sounds like a lot of work, not not nearly as much work as supporting two dozen codebases.
• Don’t use the APIs. Really. Do it yourself. The reasons for this tend to fall into two categories:
  • Lack of control. Generally, more advanced apps don’t make use of the standard javax.microedition.lcdui.Form drawing mode for professional apps, and definitely not for work-for-hire (Try explaining to a client that there’s no way you can change the title’s background, see how they react!).
  • Lack of bug-freeness. Think it’s handy that you can change the color of a font and write it to a virtual buffer? The original Nokia Series 60s didn’t think it was useful enough to support. Get in a habit of wincing every time you take advantage of another cool GameCanvas feature to your code. Sorry!
• Use a preprocessor. Java/OO purists are going to hate this, but the Java purists must hate quite a lot about J2ME. Use a preprocessor (like the one included with antenna) to build code that works against different handset sizes and capabilities.
• Use the JAD properties to tweak your code. This is far easier to manage than a total rebuild. Take advantage of the JAD properties for simple differences in builds, such as softkey codes, sound characteristics, timers adjusted to the speed of the handset, etc. Again, you’ll be sacrificing a little code size (and maybe a tiny bit of speed, in some cases), but seriously.
  

Certain tedious tasks come up all the time that are unique to the mobile dev world. One of those things is digging for information about phone capabilities (screen dimensions, feature support, etc). There are plenty of sites that handle this in some capacity (my favorite is Phonescoop), but as with all sites that aren’t your own, sometimes they just take you 80% of the way there and leave you wishing you had just one more button in just the right place.

I needed something a little more efficient for quick, capability-based lookups, to I threw this tool together:

http://devices.theoreticlabs.com

This is my first Ruby on Rails project (running on mongrel, yeah!) so there’s bound to be some wackness, but it works for ma; actual mileage may vary. It’s built on top of the excellent-for-its-price open source handset capabilities database, WURFL. Enjoy!

Quick — your eyes are rolling into the back of your head in your Introduction to Physics lecture hall, and you want to open up a game on your phone to kill the last 20 minutes of class. What’s the first thing you do when you flip open your phone?

If your answer was “make sure the sound is turned off so I don’t get caught / disturb the class, and sound on cell phone games suck anyway,” you’ll be happy to know that our first in-house game release, Mitosis, comes with that audio feature built in just for you; it has no sound. Here are a couple reasons why:

• Sound compatibility across handsets is more bug-prone than nearly any other porting problem, with the possible exception of networking code. As a small shop, we really only have about a half dozen phones that we can test on, which would severely limit us if we weren’t able to make a few assumptions. So why bother, if:
• As mentioned above, people really don’t care about sound in J2ME games! Of course in a sweeping generalization like that, there are bound to be people who disagree, but if you take a look at the polls on Midlet Review, game reviews anywhere, posts on the J2ME forums, you’ll find that people only ever complain about sound, never consider it positively. It’s dead last on everyone’s list. Anecdotally, nearly all of the people I see playing games on the subway, standing in line, in waiting rooms, do have the sound turned off.

That’s not to say that we’re a shop made of mimes and we’ll never do sound, and we’ll never try to do a really high-quality job on sound effects or music! For one thing, carriers may mandate that we add sound, or we may just miss it or feel like it’s very important to some particular game/app. For the time being, though, mum’s the word.

Caveat: Does not apply to iPod games! Those gadgets are obviously made for music, plus they’ve got no external speaker, so it’s a different, uh, game altogether.

I just gave a J2ME Game API workshop to Michael Sharon’s excellent Mobile Application Design class over at NYU’s ITP program. We walked through the best practices behind creating an old-skool side-scroller game. Talk notes and source from the game we put together can be found here.

There are two major things that make mobile software development vastly different and more frustrating than development in other industries. These hindrances are holding the whole sector back, and really, it’s in everyone’s best interest to smooth the development path for mobile software, both for WAP and downloadable applications.

Carriers need to learn how to let go.

Remember Prodigy, and the good old days of AOL? The carriers do, and they’re totally in love with the “walled garden” style of trapping a user in their own universe. Carriers still want to control all of the mobile content that the user sees, buys, and uses. This goes both for the mobile web and for downloadble applications. This has a couple effects:

• Users are given limited choices. Obviously. But this means that it’s very hard for the industry to grow organically. You’ll hear over and over again that “nobody knows what works yet” in mobile, and this is largely due to the fact that it’s extremely difficult to get the audience to try new things. What the user sees and gets is dictated from the top down.
• Mobile software companies depend hugely on carrier relationships. The success of your game/application depends a lot less on free market capitalism and how good your product actually is, and much more on how strong your relationships with the big carriers are.

Handsets need to obey standards.

There’s a terrible tension between the need for software and browser compatibility across multiple handsets, and the need for handsets to distinguish themselves from others in the market with wacky features. That’s not to say that Nokia or Motorola should stop making phones with peculiar geometry or features nobody else has, but they need to do so without requiring developers to bend their code to fit into yet another sub-model. A single phone represents a fraction of a percent of a developer’s user base, so maintaining so many SKUs quickly becomes unmaintainable.

Java ME needs to stabilize. Porting games and apps — that is, making a mobile app work across multiple handsets — is an awesomely frustrating chore. There’s a special room in hell where you have to port Java ME apps to support handsets full of undocumented bugs. From this torture has risen a thriving cluster of small startups that specialize in porting, who have knowlege of the handsets’ different capabilities and bugs and retool code that works well for one phone to work against three dozen other phones. Those shops often rely on a third-party database of undocumented phone capabilities, another mini-industry built on the inefficiencies of the mobile space. A big part of the promise of Java is supposed to be that “write once, run anywhere” feel-good feeling, and maybe with the exception of varying graphics sizes for different screen sizes, it should. Full-time porting jobs should not exist! But they do, and they probably will continue to for a few years. It must be said that BREW is much better in this respect; it doesn’t hurt that Qualcomm is the tightly-controlled central source for the firmware.

All of this is sort of a crying shame for a technology that should be so much farther along than it is, but it also presents a neat, obvious opportunity to people willing to move it forward. The silver lining for the United States is that our backwardness and slowness has given us two crystal balls into the future of mobile: The internet, and the rest of the world’s mobile industry. You can guess some things about the future: Walled gardens will give way to third party portals, and as more site come online and cross-connect to each other, search engines will become more useful and the portals will become less so. The greater the connectivity capabilities, the better the apps — and we can watch the huge successes half a world away that will likely be repeated here in two more years. The industry is moving, fast, and everyone (even the carriers, in their own weird way) seem to recognize this situation and are working to move everyone out of the darkness, and soon, this mini-rant will be obsolete.

Recently, I’ve made a few embarrassingly obvious concurrent-programming faux pas, so as a means of retribution to the multi-threaded daemon, I’ll try to confess a couple of my own anti-patterns here.

1. Servlets are reusable.

Here’s a snip of sample code:

public class MyServlet extends HttpServlet
{
  private HttpServletRequest _request;
  private HttpServletResponse _response;

  public void doGet(HttpServletRequest req, HttpServletResponse res)
  {
    _request = req;
    _response = res;
    ...
    new ResponseHandler().handle(this);
    ...
  }

  public HttpServletRequest getRequest() { return _request; }
  public HttpServletResponse getResponse() { return _response; }
  ...
}

What’s the most awful thing about this code? The big catch is that the servlet engine (Tomcat in my case) will reuse the same MyServlet object for subsequent requests, and when you make it big (or when someone’s, ahem, excessively aggressive crawler finds you), the servlet engine will call the same doGet()/doPost() on request B before request A one finishes. This means that, in the above example, when the ResponseHandler object calls MyServlet.getRequest() or MyServlet.getResponse(), it could potentially return the wrong object. This means that requests A and B could end up with their schwartz twisted, and you’ll probably end up with something like both responses appended together in B’s output stream, with A’s response lost and gone forever.

The solution is pretty simple in this case: Don’t use member variables in your servlet. Here’s the same code fixed:

public class MyServlet extends HttpServlet
{

  public void doGet(HttpServletRequest req, HttpServletResponse res)
  {
    ...
    new ResponseHandler().handle(req, res);
    ...
  }

  ...
} 

You’ll probably find you don’t even need the servlet object anymore and can pass its parts in. That thing is dangerous anyway.

2. Protect your cron jobs

This you can file under the “like, duh” folder, just don’t file it under the “my script only takes a few seconds to run, it can’t happen to me” category, like someone writing this entry did. Long-running jobs can get backed up on each-other, as cron will obediently lob another program out of the gate, whether or not the first one finishes. If you’ve got a serious hang-up or delay, this backup can get pretty severe. Take a flip through David Pashley’s excellent article on writing robust shell scripts and you’ll find something along these lines for that you can pretty much rip off and use:

if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null;
then
  trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT

  [critical-section]

  rm -f "$lockfile"
  trap - INT TERM EXIT
else
  echo "Failed to acquire lockfile: $lockfile."
  echo "Held by $(cat $lockfile)"
fi

When you think you’ve fully redeemed yourself, and you can’t ever be totally sure, you absolutely must test your code. Testing for race conditions is notoriously difficult — for a start, I’d recommend some free load testing software like Apache JMeter, unless you have the megabucks for something like LoadRunner. JMeter seems like it can handle both web and non-web apps, including trying to injure JUnit tests with multithreaded calls. Very handy.

One quick concluding word about race conditions: You never think they’ll happen to you, and they don’t. Until they do, and when they hit, they’ll take down your entire site with hundreds of megabytes of logged stacktraces, at 3:30 in the morning, on your first day of vacation. This is not an exaggeration, it’s just life.