Any business that wants to succeed must keep a close eye on its user engagement and retention. And of course, there are several reasons for that: the first one it's no secret that getting a new user for an app is much more difficult (and expensive as well) than keeping an existing one so it becomes quite obvious where to apply the best effort. And the second one: the more engaged your users are, the more chances they will interact with the content, which means possible revenue for a company.
Furthermore, according to a study conducted by Flurry in November 2012, there is a strong correlation between the average number of minutes per month users spend in apps and retention, and this correlation is statistically significant. This leads us to the thought that developers need to make retention their top focus.
Okay, we got it. Retention is important, but what's next? How to master it and after some changes (that we believe will help us with that) had been made how to measure the impact we've got? And most importantly: is there a way to improve retention almost immediately without months of development and difficult implementation of new features?
By the name of the article you’ve probably already guessed it - push notifications is the answer.
In nGrow, we dedicate the retention rate metric very special attention as it's crucial for our partners and for us in understanding the value of our service: if we uplift user retention for a given app, it means a company can save money on attracting new users and earn more from existing as they live longer and stay involved longer. That also means more opportunities to generate revenue from in-app purchases and advertising.
Let's take a look at how we measure retention in nGrow.
What is the retention rate?
Basically, retention rate is a number of people that app retains over a certain period and churn rate on a contrary shows how many people has abandoned your app over a certain period of time. Both of these metrics are usually calculated in percentages. Churn rate can be easily obtained if we deduct retention rate from 100% - which literally means that everyone who hasn’t retained - has abandoned the app.
It’s easier to understand on a simple example: lets say we have 100 new users on day 1 and 80 of them came back on day 7, this means that the 7 day retention rate for this cohort is 80/100*100% = 80%(which is pretty impressive)
Also, we need to define what is cohort analysis as we gonna use this type of segmentation on our users. Cohort analysis describes behaviors of various groups of users over time and splits users into “buckets”(cohorts) so each of these cohort will refer to one period of time (one day, one week, one month etc)
One more important detail: as we want to validate the impact that we have with our pushes, we need to set up a control group which we call a holdout. By default, we randomly take 10% of users that we label on our side so we never interact with them afterward, but sometimes, if users' traffic isn't that great, we can raise holdout up to 50%. After that move, we can finally run an experiment.
How do we measure retention?
Depending on what stage of incorporation with our partners we are on, we evaluate retention on a month, week, or day-period basis and sometimes we use several periods for a deeper level of understanding of what is going on.
Daily retention is used when we have just started working with an app and we want to have proofs that we have an uplift in retention and it meets our expectations but we don’t have enough data yet. To do that we break down our users by date they’ve installed the app and calculate short retention - it might be first-day retention, 3d day retention, or 7th-day retention, etc.
Let’s calculate for instance what is 3-d day retention rate. To do that we need to sum up all the users that came back on a 3-d day after installation(see pic 1) - for all cohorts that have 3 days in their timeline, and divide it by sum of people that were in this cohorts on a day 0 (installation day) - means that we sum the cohort sizes.
So we need to calculate it for nGrow cohorts and for holdout cohorts, and then we can get the difference (preferably an uplift) in absolute numbers and relative percentages.
But if cohorts are too small, the result and consequently the difference might not be statistically significant, and we see fluctuations in both lines then we can say that we need more data and so we use one week period to calculate the retention. It needs to be said that with the day period we can not only evaluate changes pretty fast and catch the slightest movement of our line graph but also see quite precisely when these changes took place in particular.
When we run an experiment for about a month and we finally aggregate enough data we can start analyzing the impact of push notifications without weekly volatility. It helps us understand what changes we cause in the mid-term.
But surely, it's not the max scale. We can zoom out a bit more after we prove that we have an uplift on week-period retention and we've cumulated data for several months now - we can take a look at the big picture here.
That is actually what the business part wants to know: if we use push notifications, how much more are we gonna make every month, and will it cover the cost of notifications? To measure that, we need to build a graph with a one-month bucket split.
it needs to be said that usually, at some point retention rate stops its natural descending and stays at pretty much the same level. So if pushes work, this line at the end of the graph should be higher than the holdout line - that's the result we are seeking for.
Although monthly retention is useful in long-term vision it doesn’t allow us to monitor changes fast enough to react and implement amendments in our experiment. So it is used more for retrospective analysis and evaluation of overall impact.
Speaking about overall impact and business team that wants to have just 1 number to make a decision - difference between the experimental group retention rate and control group. So how do we measure that?
Well, to calculate that, we take the sum of retentions for the taken period (in the example above, the whole period is 3 weeks) for grow cohorts, and holdout cohorts, then subtract the first from the second and divide it by the sum of holdout retention. In the end, we will get an average relative difference which is, in our example 20%.
We broke down some steps we take in nGrow to measure the impact we make with our push notifications. Hopefully, this article will be useful for those of you who decided to improve your app's retention rate.