Add-on and App Reviewer Meetup at MozFest 2014

Every app submitted to Firefox Marketplace and every add-on submitted to AMO is reviewed by a person; 60-80% of the time, that person is a volunteer.

Each year, the AMMO team endeavors to meet a few of the top volunteer reviewers in person to talk about the past year, get feedback, plan for the coming year, and have a pint or two. This year, we arranged a meetup at MozFest in London.

welcome dinner

We kicked things off with a welcome dinner and drinks, then spent the following day at the London MozSpace having more in-depth discussions. The group was extremely diverse—ten countries were represented among the 12 people who attended. Some reviewers joined in the past year, others have been reviewing for nearly a decade.

One piece of feedback that really stuck with me was that many people think being an app reviewer is the only way to contribute to Marketplace. This means we need to do more to get the word out about the myriad ways one can get involved. But it also means the reviewer program is strong, and widely known, and these are reasons themselves for celebration.

Reviewer Meetup

Notes from the meeting are available on this etherpad. We got some great feedback, and I am really looking forward to tackling some of the action items in the coming weeks.

Afterwards, we attended the MozFest science fair kick-off together, then dispersed over the weekend to explore the event, letting serendipity and our own unique interests guide us.

My interests and chance meetings guided me to make an LED robot, learn how to pick locks, and have fascinating discussions about communities and web literacy. Though I was weakened by flu at the conclusion of the weekend, I came away feeling invigorated by the spirit of Mozilla.

The AMO Reviewer Community Turns 10

A decade ago, Firefox introduced the world to a customizable web browser. For the first time, you could use add-ons to personalize your entire browsing experience—from the look and feel of buttons, to tab behaviors, to content filtering. Anyone with coding skills could create an add-on and submit it to addons.mozilla.org (AMO) for others to use. The idea that you could experience the web on your own terms was a powerful one, and today, add-ons have been downloaded close to 4 billion times.

Each add-on listed on AMO is thoroughly reviewed to ensure its privacy and safety, and volunteer reviewers have shouldered much of this effort. To properly inspect an add-on, a reviewer has to dig into the code—a taxing and often thankless chore. Nobody notices when an add-on works as expected, but everybody notices when an add-on with a security flaw gets through. These reviewers are truly unsung heroes.

From the beginning, volunteers recognized the importance of reviewing add-ons, and self-organized on wiki pages. As add-ons grew in popularity, it became necessary to hire a few people out of this community to keep it organized and nurtured. Ten years later, volunteers are still responsible for about half of all add-on reviews (about 150 per week). Our top volunteer reviewer is approaching 9,000 reviews.

As a community manager working with volunteer reviewers, I’m sometimes asked what the secret is behind this enduring and resilient community. The secret is there isn’t just one thing. Anyone who’s ever tried giving away free food and booze as their primary community-building strategy has learned how quickly the law of diminishing returns kicks in.

What’s In It For Me?

To understand why people get involved with reviewing add-ons, and why they stay involved, you only have to understand human nature. Altruism tells just part of the story. People are often surprised when I tell them that many reviewers began volunteering for selfish reasons. They are add-on developers themselves, and wanted their add-ons to be reviewed faster.

Some of these developers authored add-ons that are used by tens of thousands, sometimes millions of people, so it’s important to be able to push out updates quickly. Since reviewers are not allowed to review their own add-ons, the only way to speed things up is to help burn down the queue. (Reviewers can also request expedited reviews of their add-ons.) Also, they can learn how other people make add-ons, which in turn helps them improve their own.

Intrinsic Motivation

People who create add-ons are people who write code, so the code itself can be interesting and intrinsically motivating. In Drive: The Surprising Truth About What Motivates Us, Daniel Pink writes that self-motivated work tends to be creative, challenging, and non-routine, and add-on reviewing has it all: every piece of code is different (creative), security flaws can be cleverly concealed (challenging), and reviewers contribute at their own pace (non-routine).

Not Just Carrots and Sticks

A few years ago, we began awarding points for add-on reviews and introduced a leaderboard that lets reviewers see their progress against other reviewers. The points could also be redeemed for swag as part of an incentive program.

While this is admittedly a carrot-and-stick approach to engaging contributors, it serves a larger purpose. By devoting time and resources to sending handwritten notes and small tokens, we are also sending the message that reviewers are important and appreciated. When you open your mailbox and there’s a Fedex package containing a special-edition t-shirt in your size, you know your efforts haven’t gone unnoticed.

Community and Responsibility

AMO reviewers know that they play an important role in keeping Firefox extensible, and that their work directly impacts the experience people have installing add-ons. Since about half of the hundreds of millions of Firefox users have add-ons installed, that is no small feat. I’ve heard from reviewers that they stick around because they like being part of a community of awesome people who are responsible for keeping add-ons safe to use in Firefox.

The Magic Formula

Online communities are complex, their fabric woven from a mesh of intrinsic and extrinsic, selfish and altruistic motivations. A healthy, lasting community benefits from a combination of these factors, in varying proportions, some of them driven by the community and some by the attentive community-builders tasked with nurturing it. There isn’t a silver bullet; rather, it’s about finding your own magic formula and knowing that often, the secret ingredient is whatever it is that makes us human.

Happy 10th birthday, AMO reviewers.