1ft Startup

GetJAR is well known by mobile developers the first stop in distributing a free mobile app. They have plenty of reach, worldwide traffic, a cool beta-testing community, and some (slightly questionable) statistics on performance of each app. The downside to GetJAR is that the bar is set very low for entrants, so there is a lot of stuff here. You’ll be pretty motivated by seeing that the top apps are getting tens of thousands of downloads a week, but do observe just how long that list is. As of this writing, even the relatively mundane “Times and Calendars” category is well over 300 apps. Nobody is going to review 300 time-keeping apps to find the perfect one, so rest assured that the ones at the back of the list can’t get no love.

What makes a popular app popular? On this site, it’s all about perpetual motion: Aside from promotional placement, apps are rated simply by number of downloads — and the ones at the front of the list, surprise!, get more visibility, and thus more downloads, and the trend continues. If you fall down the list, your fall may accelerate over time as new apps show up and get promoted.

You do have one simple grace period: From the time you first upload an app will also make an appearance on the first couple pages. This is your chance to make it a huge hit — once it’s gone, your chance isn’t likely to come back.

Here are three tips for getting the most out of your brief opportunity:

1. Be flashy. Use a bright animated GIF for your icon. Write the most seductive quip you can conjure in your summary. This is your chance to get users’ attention. This may sound obvious, but it is really helpful and something not to forget. When we switched from a static icon to an animated GIF, views on our app jumped 3x instantly.
2. Use a tracking image. How did I know that views on the app jumped 3x right away after putting an animated GIF in? You’re allowed some basic HTML in the description, so take advantage of it by letting yourself see a trackable hit on your server whenever someone checks out the details of your app.
3. Release to as many handsets as possible. Right away. GetJAR measures gross total downloads of your app, so don’t release for just your favorite Nokia N-series at first while you work out the kinks on the rest. Release everything at once — so everyone has an opportunity to download, and get your numbers up. This bit us the first time around since we were so psyched to release a few handsets the second they were ready. This is probably the single most important thing you can do to get good placement!

We originally launched a little game called Mitosis nearly a year ago, as a cool experiment in what it would take to get a basic, unbranded game from nothing all the way through the door and as far as we can take it. Like most geeks, we’ve released very few of our own projects to commercial success, spending our time on other peoples’ projects for the actual cash.

If we were going to, say, write up the process of how we got Mitosis launched, it would look something like this:

- Build a cool game. The fun part. When you think you’ve got something good, you should consider the option to…
- Take it to an agent. We happened to know a few good people who could represent us to the carriers, otherwise we’d have languished out on GetJAR (more on that next time). Your agent will entice you with eye-popping (but un-guaranteed, of course) numbers. Your agent will also give you feedback from carriers, middlemen, basically whoever happens to be nearby, and if you’re lucky, you’ll eventually….
- Get reviewed by the carriers for deck placement. It takes MUCH longer than you think. Dealing with carriers is extremely slow; typically they have review processes that aren’t constant, but periodic: monthly, or even quarterly. If you’re one of the lucky, chosen few, you get to spend about 60 days to….
- Port your app. Foolhardy and spendthrift people that we are, we decided to do this in-house with the help of Device Anywhere. This is a possible option, but porting is never enjoyable. Do not underestimate how much work this is. It’s really a tedious and ongoing process, and some problems you simply can’t get around. Definitely take the time to streamline your build tools as much as possible, and you can survive to see the day that you…
- Get your builds certified. Like Verizon’s relationship with NSTL, T-Mobile now has a for-pay certification program with a company called True North. In our experience, their testing has been fair, and their test cases are reasonable; the true downside is simply that it’s up to the developer/publisher (depending on your agreement) to pay for the testing, and re-testing if you fail. That’s life! And now, there’s not much left to do but….
- Wait for it to show up in the catalog. Maybe we got lucky, but we ended up in some sort of deck-placement purgatory for many months while there was some sort of internal reorganization. Recently, finally, we launched on T-Mobile. Yeah!

Our important take-homes from this whole process:
- Publishing through carriers puts you at the mercy of many, many forces beyond your control.
- Decide what you want your company to be good at and focus on improving it.
- Don’t underestimate the pain of porting.
- The last 5% of the process can will drag on for months.

And most importantly, don’t forget why you’re doing what you are doing. We set out to have some fun, take a little break from our day jobs, and make some great games. And to be honest, we started to get burnt out by the actual business process. Once your game is out the door and in the hands of pre-teens spending their parents’ second mortage, though, there’s really no feeling like it.. And we may be just foolish enough to do it again.

If you’re managing a large site, building the WAP version of your site in a completely isolated environment is an appealing idea. It’s a business experiment, it doesn’t impact the daily site or put it at any risk, and it’s a quick way to get it up and running. This worked extremely well when I built the mobile site for a large news agency — News agencies are in the business of syndication, so they make their data extremely accessible. One of their web developers put it: You turn the tap on, and news is supposed to flow out.

So we launched our mobile news site, it was a huge success, and traffic started flowing in. Eventually, though, we started getting a few complaints that went like this: People were emailing articles to each-other from the web, and when they tried to click the link on their BlackBerries, they were redirected to the front page of the mobile site! They’d get a link from their friend, to:

news-site.com/article/25798470

And when they hit the link on their phone, they’d be detected as mobile and redirected to:

mobile.news-site.com

Not good! The same thing was also happening from users’ RSS readers. We did eventually fix this, but we had to set up some crazy redirects to take care of it (and unify some back-end data more tightly, so we could refer to the same article IDs).

The much cleaner approach would have been an integrated one, that (assuming a MVC model) simply displayed a different View while using the same Model/Controller.

A huge part of WAP consulting these days lies around ‘making the mobile version of my site.’ This is a great way to get things going quickly, but please do this with an eye toward eventually providing airtight site integration!

So, you’ve organized a clever build system that can manage a thousand different SKUs of your mobile software with the greatest of ease, and now you have to stop putting off the fun part: testing. How the heck do you really test on dozens of devices?
Your first (and probably most important) stop should be a few things to consider before even loading up a phone:

1. Make sure you have all of the features and behavior you want in the app. If you make functional changes to your game/app halfway through porting, you’ve invalidated all the fun work you’ve done and you get to do it again. Think outside of just your own requirements! Be doubly sure that your app also has all the features required by the carrier/vendor/external test team you’re porting for.
2. Be very careful choosing the phones you want to port to. Each phone will have its own quirks and will be less easy than you think it will, so choose carefully. If you can, pick newer devices, and carefully screen external demands for discontinued models (requirements get old fast, so be sure you’re not just being handed last quarter’s list). While some consumers are likely have the discontinued handsets out in the wild, people are less likely to be buying content than on their brand new Nokia. Also, old devices are going to be a pain to find yourself for testing.

Once you’ve trimmed your handset list and settled all your features, you’ve got basically three levels of quality testing, each with their own tradeoffs:

1. Beta programs run by mobile communities. These programs allow you to upload your code to a select group of community members and allow them to try out your code. You get helpful feedback, they get early access to the coolest new stuff; everyone wins. Here are a couple of them:
   • GetJAR Beta Testing
   • Vodafone Betavine

   I’ve personally used GetJAR’s beta program and had surprisingly good results. One reason for this is that GetJAR is actually very strict about users who sign up, and ban users who download your game/app and don’t give any feedback. Some users were extremely helpful; GetJAR delivers a spreadsheet at the end of the test program with handsets and basic details about what worked and what didn’t. Occasionally, you’ll get the user’s contact info to follow up.
   
   This is a great way to go for free, coarse app testing. This is a terrible way to go for carrier-grade testing, because you won’t get enough detail, enough devices, you’re unlikely to get to repeat tests you need for bugfixes, and you have to entice users to download your app in the first place. But, if you’re a hobbyist with a budget of about $0, and are not taking this to a carrier/deck with serious porting requirements, it’s a very good (and free) first stab.

2. DeviceAnywhere. These guys have saved our low-end lives! Here’s how it works:
   
   MobileComplete (the company behind the operation) has a lab full of handsets that they let users lease remotely, by the hour. That is, they’ve hotwired a few hundred phones to the internet and let you control them remotely, and yes, these are the real devices being controlled over the real net! It’s a clever business and a neat trick — For those of us who can’t change even set our clocks for daylight savings time without electrocuting ourselves, it makes our head spin.
   
   I’ve found the testing process pretty smooth for most cases. Some devices aren’t always available (all their users are pulling from the same pool), but they open up pretty quickly. You will experience some problems with sound on some devices, and rapidly-updating features like video streaming will be laggy, but this will be a very good verification for 98% of what you’re up to.

3. Brute force. This means either:
   • Buying yourself every phone you need to test
   • Paying a porting house to do the work for you.

   Both of those options are things you want to give a lot of thought to before you head toward them. Porting shops will cost you at least a few hundred dollars per phone, and that can stack up after 50-60 handsets. My personal opinion is that if you’re in this business seriously, gaining the porting toolchain and expertise in-house will be very good for you in the long run.

All in all, porting is one thing that really hurts about mobile from an engineering point of view. It’s hard to estimate, resource-intensive, and very hard to manage and maintain. I’ve heard one developer recently estimate this as taking about 100%-150% of the initial development effort per project, which feels about right to me.

One last thing about this: Porting is a process, not a destination! You’ll inevitably be a huge success once you have widespread support for your snazzy new widget, and want to continue support for phones in the future months and years… So in spite of the urge to make spaghetti to get it to work, it’s even more critical than usual to keep the code and build process clean and repeatable!

One of the major differences between mobile code and desktop code is the severe number of SKUs (that is, variations on the build) that you have to support to make a dent in the customer base. Device support is a truly a long tail; after a couple hundred devices you’re getting close to about 80% of the market.

The biggest problem with J2ME in this case is that, unlike the desktop/PC world of Java, the VMs are built by MANY groups with different interpretations of the spec. Even if that weren’t the case, you’re trying to support some very different hardware characteristics: memory, screen resolution, network limitations, storage, etc.

All this means is that if you want your app to be a success, you’re going to adapt your code to support hundreds of phones.
Ugh.

What’s a developer to do? Here are a few basic points that work for me:

• Stay Organized. This should go without saying, but make a concerted effort to keep your builds clean, repeatable, and stowed away well. Someone will be asking you to send them the latest working build for their Motorola C290, and you should always have it handy. Do one of these:
  • Keep a folder heirarchy for your compiled files. Mine looks like:

    ->Builds
       -><app name>
           ->Version (e.g. 0.9.4)
               ->Screen Size (e.g. 176x220)
                    <standard build>
                   ->Custom build 1 (e.g. Nokia Series 60)
                       <build>
                   ->Custom build 2
                    ...
    

    For a solution like this, you’ll need a handy way to look up capabilities to help you dig out the right build for a handset.

  • Build/adapt a simple CMS. If you’ve got some time, this is the best option you’ve got (we’re working on a new one we may be able to share with you, stay tuned). You’ll thank yourself later.
• Write reusable code. Write with the intention of building what you write into a framework. OK, this will probably net you a few more KB in code, but it will save you from hurting yourself over and over again. Write as few new lines as you possibly can in a new app. Every time you fix a handset-specific bug, you’ve fixed it for everything you ever write in the future. In fact, if you’re really pathological, you can extract basically everything out of most apps into your own scripting/markup language and build your apps without writing a single line of real J2ME code. Sounds like a lot of work, not not nearly as much work as supporting two dozen codebases.
• Don’t use the APIs. Really. Do it yourself. The reasons for this tend to fall into two categories:
  • Lack of control. Generally, more advanced apps don’t make use of the standard javax.microedition.lcdui.Form drawing mode for professional apps, and definitely not for work-for-hire (Try explaining to a client that there’s no way you can change the title’s background, see how they react!).
  • Lack of bug-freeness. Think it’s handy that you can change the color of a font and write it to a virtual buffer? The original Nokia Series 60s didn’t think it was useful enough to support. Get in a habit of wincing every time you take advantage of another cool GameCanvas feature to your code. Sorry!
• Use a preprocessor. Java/OO purists are going to hate this, but the Java purists must hate quite a lot about J2ME. Use a preprocessor (like the one included with antenna) to build code that works against different handset sizes and capabilities.
• Use the JAD properties to tweak your code. This is far easier to manage than a total rebuild. Take advantage of the JAD properties for simple differences in builds, such as softkey codes, sound characteristics, timers adjusted to the speed of the handset, etc. Again, you’ll be sacrificing a little code size (and maybe a tiny bit of speed, in some cases), but seriously.
  

Quick — your eyes are rolling into the back of your head in your Introduction to Physics lecture hall, and you want to open up a game on your phone to kill the last 20 minutes of class. What’s the first thing you do when you flip open your phone?

If your answer was “make sure the sound is turned off so I don’t get caught / disturb the class, and sound on cell phone games suck anyway,” you’ll be happy to know that our first in-house game release, Mitosis, comes with that audio feature built in just for you; it has no sound. Here are a couple reasons why:

• Sound compatibility across handsets is more bug-prone than nearly any other porting problem, with the possible exception of networking code. As a small shop, we really only have about a half dozen phones that we can test on, which would severely limit us if we weren’t able to make a few assumptions. So why bother, if:
• As mentioned above, people really don’t care about sound in J2ME games! Of course in a sweeping generalization like that, there are bound to be people who disagree, but if you take a look at the polls on Midlet Review, game reviews anywhere, posts on the J2ME forums, you’ll find that people only ever complain about sound, never consider it positively. It’s dead last on everyone’s list. Anecdotally, nearly all of the people I see playing games on the subway, standing in line, in waiting rooms, do have the sound turned off.

That’s not to say that we’re a shop made of mimes and we’ll never do sound, and we’ll never try to do a really high-quality job on sound effects or music! For one thing, carriers may mandate that we add sound, or we may just miss it or feel like it’s very important to some particular game/app. For the time being, though, mum’s the word.

Caveat: Does not apply to iPod games! Those gadgets are obviously made for music, plus they’ve got no external speaker, so it’s a different, uh, game altogether.

Recently, I’ve made a few embarrassingly obvious concurrent-programming faux pas, so as a means of retribution to the multi-threaded daemon, I’ll try to confess a couple of my own anti-patterns here.

1. Servlets are reusable.

Here’s a snip of sample code:

public class MyServlet extends HttpServlet
{
  private HttpServletRequest _request;
  private HttpServletResponse _response;

  public void doGet(HttpServletRequest req, HttpServletResponse res)
  {
    _request = req;
    _response = res;
    ...
    new ResponseHandler().handle(this);
    ...
  }

  public HttpServletRequest getRequest() { return _request; }
  public HttpServletResponse getResponse() { return _response; }
  ...
}

What’s the most awful thing about this code? The big catch is that the servlet engine (Tomcat in my case) will reuse the same MyServlet object for subsequent requests, and when you make it big (or when someone’s, ahem, excessively aggressive crawler finds you), the servlet engine will call the same doGet()/doPost() on request B before request A one finishes. This means that, in the above example, when the ResponseHandler object calls MyServlet.getRequest() or MyServlet.getResponse(), it could potentially return the wrong object. This means that requests A and B could end up with their schwartz twisted, and you’ll probably end up with something like both responses appended together in B’s output stream, with A’s response lost and gone forever.

The solution is pretty simple in this case: Don’t use member variables in your servlet. Here’s the same code fixed:

public class MyServlet extends HttpServlet
{

  public void doGet(HttpServletRequest req, HttpServletResponse res)
  {
    ...
    new ResponseHandler().handle(req, res);
    ...
  }

  ...
} 

You’ll probably find you don’t even need the servlet object anymore and can pass its parts in. That thing is dangerous anyway.

2. Protect your cron jobs

This you can file under the “like, duh” folder, just don’t file it under the “my script only takes a few seconds to run, it can’t happen to me” category, like someone writing this entry did. Long-running jobs can get backed up on each-other, as cron will obediently lob another program out of the gate, whether or not the first one finishes. If you’ve got a serious hang-up or delay, this backup can get pretty severe. Take a flip through David Pashley’s excellent article on writing robust shell scripts and you’ll find something along these lines for that you can pretty much rip off and use:

if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null;
then
  trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT

  [critical-section]

  rm -f "$lockfile"
  trap - INT TERM EXIT
else
  echo "Failed to acquire lockfile: $lockfile."
  echo "Held by $(cat $lockfile)"
fi

When you think you’ve fully redeemed yourself, and you can’t ever be totally sure, you absolutely must test your code. Testing for race conditions is notoriously difficult — for a start, I’d recommend some free load testing software like Apache JMeter, unless you have the megabucks for something like LoadRunner. JMeter seems like it can handle both web and non-web apps, including trying to injure JUnit tests with multithreaded calls. Very handy.

One quick concluding word about race conditions: You never think they’ll happen to you, and they don’t. Until they do, and when they hit, they’ll take down your entire site with hundreds of megabytes of logged stacktraces, at 3:30 in the morning, on your first day of vacation. This is not an exaggeration, it’s just life.