min read
February 21, 2022

How to measure retention impact from push notifications

It’s no secret that any company that wishes to succeed in the digital space needs to keep a close eye on its user engagement and customer retention. 

There are several reasons for this: 

Firstly, acquiring a new user for an app is much more difficult (and much more expensive) than keeping an existing one. So, it becomes quite obvious where to apply the most effort. Secondly, the more engaged your users are, the more they will interact with your content, and the more revenue they will  generate for your company. According to a study conducted by Flurry in November 2012, there is a strong correlation between the average number of minutes a user spends each month in an app, and the retention rates of those users. Therefore it follows logically, that retention should be given top priority by developers. The question then becomes: How? 

The answer: Through Push Notifications. 

Push notifications are quite simply messages that are sent from apps to their users, that appear on the screen of the user's smartphone. It is estimated that pushes can increase in-app engagement by up to 88%, and that 65% of users will return to an app within 30 days if their notifications are turned on. 

At nGrow, we place great emphasis on the retention rate metric, as we believe it is the single most valuable outcome of our service offering. If we increase user retention for a given app, the company saves money on attracting new users,  earns more from their existing users, and opens more opportunities to generate revenue from in-app purchases and advertising.

Let's take a look at how we measure retention in nGrow.

What is the retention rate?

Basically, the retention rate is the number of active users that the app retains over a certain period of time, and is generally expressed as a percentage. The difference between the number of active users at the beginning of the period and the number of active users at the end of the period is known as the churn rate. 

For example: let's say that we have 100 new users on day 1 (The beginning of the defined time period) and 80 on day 7(the end of the defined time period), this means that the 7-day retention rate for this cohort is 80/100*100% = 80%. A cohort is a clearly defined group of users selected for analysis over the given time period. 

pic 1

In order to validate the impact that we achieve with our pushes, we generally set up a control group which we call a holdout (a randomly selected percentage of the cohort) and send them no push notifications during the testing period. The difference between the retention rate of the cohort and that of the holdout, indicates the effectiveness of our service.  

How do we measure retention?

We generally measure retention on a daily, weekly or monthly rate, depending on the stage of incorporation we are in with our client. Occasionally we will group several periods together in order to gain a deeper understanding into the situation. 

Daily retention is used when we have just started working with a client and we want to have proof that we have experienced an increase in retention but have very little data to analyze. To do this we break down our users by the date they installed the app and calculate a short term retention rate - This could be 1st-day retention, 3rd-day retention, or 7th-day retention, etc.

pic 2

For example: A 3rd-day retention rate calculation on the above picture would go as follows:

Number of users that came back on day-3 after installation divided by the number of users that were in this cohort on day-0 (installation day).

If the cohorts are too small, the result and consequently the difference might not be statistically significant. We would therefore see fluctuations in both lines and would require more data.

We would then move to a 7-day retention calculation.

It is important to note however, that a 1-day retention calculation is extremely valuable as it not only allows you to evaluate changes quickly by detailing the slightest movements in the line graph, but to also understand precisely when these changes occurred, and pinpoint why. 

After a full month we will finally be able to aggregate enough data to start analyzing the impact of push notifications without weekly volatility. This then helps to detail the impact our push notifications are having on the app's user base. 

Once operational for several months, we are able to see the bigger picture and add real value to the business decision-makers in the form of understanding the real increase in revenue and what this increase is in relation to the cost of the service. 

It is important to note that at some point the retention rate stops declining and levels out. This means that if the pushes are effective, the blue line on the graph will be higher than the red (holdout) and that is precisely the objective from the start. 

Although monthly retention is useful in understanding the long-term trend, it doesn’t allow us to monitor changes rapidly enough to react and implement amendments in our experiment. So, it is generally used more for retrospective analysis and evaluation of overall impact.

From the perspective of impact, and delivering one metric for decision making, the difference between the retention rates of the cohort and the holdout is the key figure needed, and is calculated as follows:

We take the sum of retention rates for the cohort in a given time period (in the example above the whole period is 3 weeks) and divide it by the sum of the holdout retention rate. 

This gives us an average relative difference. In the example this difference is 20%.

Hopefully, this article serves to create an understanding of how we, at NGrow.ai, calculate our retention rates, and  we hope it will be useful to those of you who would like to improve your app's retention rate accordingly.