Escaping The Two-Star Rating Pit
By Evan Miller
May 12, 2013
I sell two titles in the Mac App Store. My sales depend on my ratings, and my income depends on my sales, so I spend a lot of time thinking about good ratings and how to get them.
The first app I ever released (Magic Maps) averaged 2 star ratings for the first six months of sales. This was painful to witness, because even when I made improvements to the program, it meant that people were reluctant to buy it, which meant that new ratings trickled in slowly, which meant the average rating hardly budged and sales remained flat. I was stuck in a vicious ratings trap. Here are a few gems taken from the customer reviews that I received during my first year in business:
And my personal favorite, which I’m in the process of having framed:
Ouch! Slowly as I fixed some crashes and added some requested features, and customers amended their reviews, the ratings crawled up to 2.5 stars. Six months and countless development later, it crept up to 3 stars. It’s still at 3 stars, even though in my heart of hearts I’m pretty sure it’s a 3.5 – 4 star app. Ratings traps are terrible for sales, and a doozie for morale.
The second time around, I was smarter about ratings. The new title (Wizard) came in Pro and Standard flavors; from the beginning, they’ve consistently averaged 4.5 or 5 stars in the App Store. Despite having critical, show-stopping bugs in the initial releases, I managed to avoid the ratings trap and quickly began the path to a steady income and Mac App Store fame.
And now, dear reader, I am going to explain how you can avoid the mistakes of my youth and achieve riches or prominence or whatever it is you desire from the distribution of a highly regarded app on the Mac App Store.
You might be thinking, that’s easy: Just write a 5-star app and release it!
Wrong. There’s no such thing as a 5-star app. Star ratings encompass people’s subjective experience of your app, which often has very little to do with what the app actually is. Ratings are not an inherent property of the software; rather, they emerge from an interaction between the app itself, your customers’ expectation of the app, and your customers’ desire to leave a rating for the app. In order to consistently receive good ratings, you need to successfully manage all three.
Prevent Crashes Like It Is Your Mission In Life
If there were a Ten Commandments of app development, the first one would be: Thou shalt not crash! This would be repeated nine more times on both tablets.
Crashes are the absolute worst thing that can happen to your app. Not only does your customer fail to accomplish their goals, in a crash, you lose control of the customer’s experience.
This is important, because if a customer is pissed off in a non-crash scenario, there are ways of dealing with it that don’t result in you getting a 1-star rating. You can have the customer email you or call you and let them tell you how pissed they are, and then you have an opportunity to fix the situation (more on this below). But in a crash, the customer has no other option. The program has closed, and now they’re off to the App Store to tell everyone how your app is garbage and a complete waste of $$$.
So: how do you avoid crashes?
Use Automatic Reference Counting. This prevents a whole class of referenced-after-release crashes.
Run the static analyzer. In Xcode this is easy: Product > Analyze. Respect the output! The static analyzer is smarter than you are. Maybe not in the same way as you, but definitely in the same way that a computer will always beat you at chess.
Heed compiler warnings. Do I really need to explain?
Find willing beta testers. This can be difficult for niche products, but it’s worth going out of your way to get people to test. Make sure they’re motivated, too; either they need the program (which is a good omen for your future prospects in the market), or because you’ve offered a bounty for new crash reports. I’ve never offered a bounty myself, but in retrospect it would have been a good idea. One motivated tester is worth fifty casual users. I had a “closed beta” program with fifty people who signed up on my website, and I don’t think I got a decent good bug report from the lot of them. (Sorry, guys.)
Hire professional QA to test the program. With Magic Maps, I made the mistake of assuming that if my app worked on my computer with my files, it must work on all computers with everyone’s files. Boy was I wrong! So when it came time to release Wizard, I hired professional QA to test version 1.0 for 20 hours. This was by far the best investment I’ve made in my apps to date: he uncovered all kinds of bugs and crashes that helped Wizard dodge the ratings trap. (Thanks, Ben.)
Test on all supported platforms. This should be obvious, but if you’re going to support Snow Leopard, then test on Snow Leopard. Apple’s APIs change in subtle ways between OS releases, so it’s really important to test everywhere. For example, a critical bug that entered my program was that calling
nilon a certain widget would set a blank title in Lion (as expected), but caused an exception in Mountain Lion. D’oh!
Eat your own dog food. Find excuses to use your own app as frequently as possible. Import weird files that shouldn’t even work with the program. Do things that the user “shouldn’t” do.
Read your crash reports. It’s right there in iTunes Connect, although the geniuses who designed the interface make it appear that there are no crash reports the first time you load the page, even if there are hundreds waiting for your review. Click “Refresh Now”, and behold the variegated causes of your customers’ misery.
I can’t stress this enough: when you’re getting close to the release date, treat the elimination of crashes as a sacrosanct mission personally delivered to you from on high. If a single customer experiences a crash, and their day wasn’t going well to begin with, it is well within their power to cast you into a 1-star rating trap without you getting a word in edgewise. This could set your sales trajectory back by months. Spare no expense to ensure that crashes just don’t happen.
OK, glad we got that out of the way. Let’s move on to the more interesting stuff.
Add an “Email the Developer” Button
Once you have a non-crashing app, the simplest feature you can add to your app to improve ratings is a menu item called “Email the Developer,” like this:
I like making it explicit who they’re emailing. If the item said “Send Feedback” or “Service and Support,” it would be completely unclear what will happen and who will be listening. Send feedback where? Under the Support docks and into the Service abyss?
“Email the Developer,” on the other hand, tells the customer they’re getting a direct line to the person who makes the product, in fact they get my personal email address, not firstname.lastname@example.org.
The email button is important because it’s a first line of defense against customer dissatisfaction. If the customer has come across a bug that prevents them from getting their work done, or if they’re having trouble accomplishing something that they want to do with the program, they can just email you and explain the problem.
You then have an unique opportunity to make their day. If it’s a bug, you can fix it immediately and send them a new build; if they’re confused about a feature, you can explain it in detail; or if they’re asking for the impossible, you can suggest workarounds or alternative programs to help the customer achieve their goals.
This interaction is crucial because it helps you build an actual human relationship with the customer. They’ll feel comfortable emailing you with smaller suggestions, and they’ll also feel comfortable answering your questions about why they use their software; both kinds of interactions will ultimately help you better craft your product and message. And if the customer has expressed interest in a particular set of functionality, then congratulations — you just found yourself an eager guinea pig on which to test out new features!
And of course, after you’ve made their day, it’s completely appropriate to ask the customer for a rating in the App Store. You’ll note that many of the reviews for my apps emphasize how helpful and responsive “the developer” (that’s me) is. Those are all due to the “Email the Developer” button in the Help menu.
There’s another benefit to “Email the Developer”. If the customer absolutely hates your product and wants to blow off steam, they can email you instead of writing a supposedly “clever” take-down of your product in the form of a “hilarious” 1-star customer review.
I learned this the hard way dealing with 1-star “DO NOT BUY” Magic Maps reviews; after adding an “Email the Developer” button, angry customers started emailing me personally instead of ridiculing the product publicly. I then had a chance to make the situation right — either by fixing the maddening bug they’re experiencing or explaining to them how to get a refund.
So the email button is really a two-pronged strategy: it gives you more 5-star reviews, it also results in fewer 1-star reviews. In both cases it helps you strengthen the relationship with your customers. Even if the customer decides your product was not right for them, they’ll walk away from the experience with a positive impression and will feel disposed to recommend your products to friends and colleagues.
I also have a “Rate in App Store” button in my apps — this can cut both ways, so use it with care. I added this at a time when I knew customers loved the app and I needed ratings fast. (The app was being featured as “New and Noteworthy” in the App Store, but didn’t have enough ratings to display an average on the front page!) You can add a “Rate in App Store” button to your own app if you’d like, but keep in mind it may result in fewer customers emailing you, and hence, fewer customer relationships. I’ll probably remove this button from my apps in the future.
Pricing and Free Trials
There’s been a ton written about product pricing. Joel Spolsky has some excellent analysis of pricing software, but here I want to talk about some conditions peculiar to the Mac App Store and how they affect your ratings.
Free trials weed out bad customers
First off, you should offer a free trial of your app on your website if you can. Unless you are Apple, no one has ever heard of you, and unless your app is cheap or the customer is desperate, they’ll be hesitant to buy an app that has 0 ratings or reviews.
But the key part about a free trial is not the customers that it converts, but the customers it doesn’t convert. A free trial version ensures that people who would have left a 1-star rating don’t get mad at your app; instead, they get mad at the trial version! They can rage impotently at the free trial all day and night, but unless they are feeling outright sadistic, they won’t buy the full version and hence won’t affect your ratings. Think of a free trial as a barrier that lets the good ratings through and stops the bad ones in their tracks. Cunning, eh? Put less cynically, a free trial helps ensure there’s a good fit between what the customer wants and what the product is offering.
As a new app, your goal is not to maximize sales. It’s to maximize average customer happiness, which will translate into great ratings, which will get people talking, which will raise your brand awareness, which will eventually translate into sales. In the short term, you need to sacrifice sales volume in order to achieve a stellar reputation.
Anyway, back to free trials: the only problem with free trials is that sometimes they don’t work. That is to say, sometimes they fail to stop people from buying your app! If your app costs $50 or less, most people are just going to decide whether or not to buy based the product description. They won’t even check to see if there’s a free trial, let alone download it. They’ll buy your app without trying it, realize it’s not what they thought it was, and then leave you a potentially apocalyptic review.
People don’t read product descriptions
The cause of people’s failure to download your free trial is this: the amount of time people spend deciding whether to buy your app is roughly proportional to its price divided by their income. The cheaper your app is, the less time people will spend deciding whether to buy it. This can be terrible for ratings, because with a cheap app, there’s huge potential for a mismatch between the customer’s expectations and the product’s actual functionality. That’s why I personally will never sell another app for less than $50 if I can help it.
For example, many people will download a $20 app having barely read the product description. Go out and read customer reviews of apps that sell for $19.99 if you don’t believe me. Heck, read the Magic Maps reviews. If you sell software for under $30, people will look at a screenshot, read one sentence, and then fabricate a fictitious app description in their heads and assume they are buying the nonexistent software that they just imagined. If you’re going to sell software for under $30, your app had better be so incredibly simple that it’s absolutely impossible to misunderstand what it does.
After I had experienced the horrors of selling Magic Maps for only $20, I decided on a different strategy for my second app, Wizard. It would be expensive, but have a Trial version available. The initial release — a “Pro” version — cost $200, and to my amazement, people started buying it. The pricing was expensive enough to attract attention, and people would check the website for a trial version, download it, and see if it was really worth $200. I didn’t have a ton of buyers, but there was a steady trickle of customers, averaging about one a day. Almost all of them left a 5-star rating, because by the time the customers were ready to buy, they already knew the program was worth $200 to them.
Overcharge for crappy software
There’s a critical corollary to all this, which is that the solution to crappy software is not a low price. If your software leaves a lot to be desired, the worst thing you can do is charge a low price to try to “make up” for it. Now you have two problems, which is crappy software and people leaving you terrible ratings because they didn’t read the app description. Absolutely no one is going to leave a review saying “Crappy software, but at least it was cheap. Five stars.”
Go ahead and start charging the long-term price from the beginning, even if your software is missing key functionality. Explain what it’s missing and where the product is going. If you’re charging enough money for the software, people will read the entire description and will understand all of the product’s limitations by the time they press Buy.
This point is crucial. The first release of Wizard Pro had a bug that prevented Mountain Lion users from accessing key functionality in the program. A $200 program that barely worked on Apple’s newest operating system! I was absolutely sick to my stomach when I found out about the bug, and started imagining all the terrible things people would say about my program as I waited three harrowing weeks for Apple to approve the fix. In the meantime, I put a message in the app description and on the website telling people about the problem, and hoped to heaven that at least a few of them would notice it.
And guess what? Customers got the message. I didn’t receive a single negative rating the entire time the bug was out there — nearly a month! Lion and Snow Leopard users continued to buy the program, leaving excellent reviews, and Mountain Lion users held off. The product price is the most important tool you have for capturing people’s attention as they decide whether or not to buy your app. You can then use this attention to set proper expectations about what your app does and the direction that it is heading.
Standard and Pro
A key decision you’ll face in devising your product strategy is whether to offer a “Pro” version. There are financial pros (hyuk) and cons to the practice, which I might discuss in another post. Here I want to talk about the consequences that a Pro version has on your ratings.
The secret upgrade path
Because the App Store doesn’t offer paid upgrades, the existence of a Pro version is potentially an Achilles’ heel in your ratings. Go read customer reviews for OmniGraffle if you don’t believe me; about a quarter complain about having to pay twice if they want to unlock Pro features after buying the Standard version. I get this all the time with Wizard.
This problem is actually really easy to fix: tell your customers how to get an app refund from Apple. Basically they just have to click through a bunch of links and then talk to a customer service representative. Sure, it will take them 15 or 20 minutes, but they’ll be happy to have their money back, and then they’ll buy Pro with the proceeds. Problem solved. Hopefully if Apple gets enough refund demands like this they’ll get wise and offer a real paid upgrade path.
Pleasing Standard users is hard
There’s a related problem, which is sometimes people will buy the Standard version and not realize in advance which features it’s lacking compared to Pro, and then feel cheated — even if they wouldn’t have bought Pro, anyway. Their expectations get even more skewed because when most people download a free trial, they’ll download a trial of the most expensive version that they can find on the website. (Who wouldn’t want to test-drive a Lamborghini?) Only later after they’ve signed a lease for a Chevy Lumina do they realize the engine is not quite the same.
Averting this danger requires careful product planning. My philosophy of Pro versions is this: If you can’t explain the difference between the Pro version and the Standard version in one or two sentences, you shouldn’t be offering a Pro version. Every Pro feature that you add is one more thing that you have to communicate to your Standard users, lest they feel disappointed.
Pleasing Pro users is harder
The temptation to add Pro features is difficult to resist, especially because many Pro customers will email you with perfectly good feature suggestions that carry the insane caveat “This of course would be for the Pro version.” Don’t listen to this crap!
I have no idea why, but people have very strong opinions about what should be in Pro versus Standard, even if they already bought Pro and would get the new feature either way. My theory is that Pro users want to feel special and that their Pro Dollars are hard at work delivering new features only for elite Pro users such as themselves. This is nonsense, so resist the siren song of New Pro Feature wherever possible.
To appease Pro users, I will occasionally throw them a bone by introducing a New Pro Feature — but only if it fits in tightly with the product strategy. 95% of my development time goes into features that everyone can enjoy. If you choose to offer a Pro version, be aware that it comes with all kinds of psychological baggage that will manifest itself in subtle ways amongst your user base. Don’t let this baggage complicate your two-sentence description of how Pro differs from Standard.
In any event, I personally like to pack the Pro version with import and export features but leave the rest of the program untouched. I can dispense with the “What does Pro do?” question by saying “It imports files created with programs X, Y, and Z. That’s it!” I also don’t have to worry about what happens when documents that were saved with “Pro-only” features are opened by someone else who only has the Standard version. (Did you ever stop and think about that?)
Pricing of Pro versus Standard is a topic for another day. I think a Standard version should generally be between 25% and 40% of the Pro price. That way there’s not such a huge gap that people assume the Standard version must be sucky, and not such a small gap that the customer agonizes about which version to buy. Their budget will tell them, usually.
You could also cheat and pay your friends to leave great reviews… but where’s the fun in that?
Seriously, if you get caught leaving fake reviews you stand to be expelled from the App Store. Also, people tend to have a good sense of smell about whether reviews are genuine or not, and will raise an alert if they suspect that something is amiss. Plus, if you have a great app and follow the advice in the article carefully, the 5-star ratings will come flowing in from the beginning, and you’ll feel a sense of unmitigated pride when you go in the App Store and read your own reviews.
Speaking of which…
Read Your Customer Reviews
You can scratch your head and guess all day why people are leaving you the ratings they’re leaving, but why not get it straight from the horse’s mouth? Customer reviews are often extremely specific about the particular sources of joy and points of pain that the customer experienced while using your product.
Be sure to read all of your reviews. Even the 5-star reviews will contain clues about what people are feeling frustrated about, and potential reasons why other people aren’t buying the app. For example, one of my apps had the ability to import older binary Excel documents (XLS), but not the newer “open” format (XLSX). I didn’t think the absence of XLSX support was a big deal — I figured people could just choose a different save format in Excel. But then I noticed that several people were mentioning the XLSX issue in their otherwise pristine reviews.
Well it turns out that the older Excel format can’t save more than about 65,000 rows of data, and many customers wanted to import a lot more than that directly from Excel. I had no idea! So the people who could work around the issue (e.g. by exporting to CSV first) were buying the app and loving it, but I was leaving out a lot of people for whom lack of XLSX was a deal-breaker. I never got to hear their concerns because they never became customers.
The moral of the story is: listen carefully to customer concerns, because a small issue for a paying customer might be a huge issue for a non-customer.
By the way, you can read reviews from every country’s App Store — not just your own — in iTunes Connect. I actually first became aware of the XLSX issue from a review written in Chinese. When the letters “XLSX” stood out from the sea of Mandarin, I knew I had a problem.
I’ve covered a lot of ground in this article, so just to recapitulate, my advice boils down to:
Prevent crashes at all costs.
Add an “Email the Developer” button… no excuses!
Decide how much time you want customers to spend researching your app, and set the price accordingly. Time is money!
“Pro” features should be summarized in 1-2 sentences. Tell your Standard customers they can upgrade by getting a refund from Apple.
Scour your customer reviews for hints of dissatisfaction.
And now I’ll add one more to the list:
Go out there and write a 5-star app!
Kidding, of course! Those don’t exist, remember? If you take away one thing and only one thing away from this entire article, it should be this: There is no such thing as a 5-star app. There is only the perception of a 5-star app, which is completely dependent on how your customers experience your product.
This is great news, because it takes a lot of the pressure off of shipping a mythical “5-star app”. It’s okay if you think your app doesn’t deserve 5-star ratings, and it’s okay if you think your app isn’t worth the asking price. All you need to believe is that your product delivers real value to your customers, and that their experience will meet or exceed the expectations they had for the product when they purchased it. (So no crashing!!!!)
Because even if you wouldn’t pay the asking price, even if you think it’s currently just a 3-star app, there are people out there who need your software’s functionality, are more than willing to pay for it, and would love to build a relationship with a developer who’s actively working on their problems.
It might take some work to find those people, but believe me: they’re out there. Even if your software is not at all innovative, remember that 90% of desktop software companies develop for Windows first and Mac second (or never), so there’s almost always some existing functionality that a Mac user is missing and would love to have installed natively on their computer. And then if your product is truly innovative and better than anything on Windows, your customers will adore you forever because it means they get to feel all smug when they show off your awesome Mac software to their poor PC-using colleagues.
This is an exciting time for Mac developers. Apple’s hardware is second-to-none, OS X gets better each year, and the Mac App Store has revolutionized the distribution of desktop software. Being in the App Store will give you instant exposure to a large audience, and if you carefully follow the advice in this article, you’ll be well on your way to acquiring your first appreciative customers and solving their problems with software.
You’re reading evanmiller.org, a random collection of math, tech, and musings. If you liked this you might also enjoy:
Want to look for statistical patterns in your MySQL, PostgreSQL, or SQLite database? My desktop statistics software Wizard can help you analyze more data in less time and communicate discoveries visually without spending days struggling with pointless command syntax. Check it out!
Statistics the Mac way