Sunny Cha
March 22, 2017
Big data tools are now ubiquitous and, compared to years ago, cheap. But while these tools give you the “what” — fire hydrants of data — they don’t provide the “how” of using it.
A gaming app developer, for example, can set up easy, out of the box tools to collect millions of data points daily on users’ gameplay behaviors, viral activities, ad engagements and so on. But it is up to the developer to figure out which of this data is good, bad, or not useful to study in the first place.
Below we walk through a useful decision flow, using our fictitious game developer as an example.
Start with the right questions
Before any data is ever gathered, you can start asking questions about business performance. These questions act as blinders to help you focus on what’s important. While it’s also possible to decide what you want — 50 percent day 1 retention, for instance — and try to find clues toward getting there in the data, this approach lacks the focus of a question and risks confirmation bias.
For our example, let’s say our developer starts with asking: “Why don’t my users come back frequently?”
After you know where to start, dive deeper
For best results, be as specific with your questions as possible. “Why don’t my users come back frequently” offers too many potential directions for investigation. A specific question should use basic insights a developer already knows about.
Developer: “Why do my users open the app for long 5-10 minute sessions and engage in game play, but still don’t come back frequently?”
Formulate hypothesis statements that can answer your questions
After finalizing your questions, form assumptions that can be proven or disproven.
Developer’s starting hypothesis: “Users of my game don’t come back frequently because they have few connections on the app.”
Then attach an action for each outcome (true, false or other) to give yourself an immediate next step after validation.
Developer’s final hypothesis: “Players of my game don’t come back frequently because they don’t have enough active connections. If this is true, then we need a strategy to bring over their friends to the game. If my hypothesis is false, then I look at other possible reasons like poorly timed notifications.”
Segment your users when analyzing data
The smaller your cohort, the easier it is to formulate a plan of action based on their behaviour. Slice your data by signup region, age, or device type and pair with factors like number of connections or behaviour toward notifications.
Developer: “It will be a lot easier to create a referral campaign for 18-25 year old iOS users living in the US, vs just targeting all iOS users.”
Establish success metrics
Without success metrics to measure data against, it’s hard to decide if you should continue a strategy, or if it’s time to recalibrate. It’s also difficult to pinpoint which users fall under the cohort that you want to examine.
Developer: How do you differentiate well-engaged users from the rest? Is it when they come back ten times a week? Should you measure iOS and Android users against the same metric?
And don’t look at individual data points
Finally, remember that things change. For example, top developers often spend months perfecting a tutorial flow, only to throw it out and start from scratch. That’s because the users you get today may not be the same as 6 months ago. Don’t get too fixated on data points at the expense of a global trend.
Developer: “Now that we’ve achieved viral growth in the 18-35 segment, should we still pursue improvements for 36-50?”
Don’t forget that the results of this data shouldn’t be siloed in a business intelligence unit or executive meeting room. As you ask questions and reach conclusions, be sure that your process for communicating to other teams is working. For instance, a marketing team will benefit from knowing which segments to advertise to. Designers will need to know usage frequencies so they can design a smooth app experience. Programmers needs to know usage forecasts to scale updates appropriately. Data should be democratized.
And for the sake of sanity, it’s best if you use a single tool for all your data, rather than multiple silos. For more on picking a service, check out our recent article on business intelligence tools.
