Aussie Web Analyst

So you have set up some weekly reports, focusing on key metrics – whatever is most important to your business/website in understanding its performance. Monday morning, you crawl out of bed, into work and update your reports so you can check out this performance (hopefully at the touch of a button). But something has happened, a key metric has dropped by 10% against the previous week. What should your first step be towards understanding what has happened?

For the sake of this example, let’s assume that it is the number of visits to your site which has declined over the previous week. And knowing you would have quickly scanned your other key metrics for more information, let’s assume that they don’t provide helpful information (unique visitors and pageviews also dropped 10%, no change in the proportion of traffic from different traffic sources).

Given this, a good place to look for an explanation of a change in weekly data is at your daily data. This may quite quickly show the exact day that values changed (whether this is up or down) providing insight into the cause of the change.

I am an analyst with Excel consistly open. Whenever I go to manipulate data, I transfer it from whatever data source it can be found in (Sitestat these days of course) into Excel where I can easily play with it. I have created some basic examples in Excel using simple numbers to highlight alternative situations.

Your Basic Data

The first step is to run your visits data at daily level so that they can be examined. By themselves, the numbers don’t mean much, they need to have some context added to them. I have added additional data showing the week on week change for each day during the last week as well as charting the two weeks data. If nothing had changed, you would get data and a chart as per below, with this sample data following a fairly standard internet day of the week pattern.

Change 1 – Trending Down (or Up)

But for these examples, visits had dropped, down by 10% against the previous week. You need to know why visits have dropped from 877,000 two weeks ago to 789,300 last week, a decline of 87,700 visits or (in useful terms) a decline of 10.0%. In this first example, there has been a consistent decline during the course of the week. Every day was down 10% against the equivalent day in the previous week. In reality of course, there will be day to day variations but you can always see a fairly similar level of change during the week.

Change 2 – The Step Change

A trend like that above won’t often be seen for a big change, say > 5%. It is more common when you are looking at a 1% to 3% change. While these are small numbers, over a number of weeks they do add it. I have seen that trend most often leading into Summer or Winter as days get longer/shorter and hotter/cooler, with these conditions impacting on overall internet usage.

Instead what you will see is a sudden step change where a metric suddenly changes to a new level. If you are looking at absolute numbers, there will be a six to eight consecutive days where each day has a similar week on week change until the numbers settle down at their new level. If you are looking at a metric that is a ratio or percentage with a less obvious day of week pattern, the change can be seen even more clearly, for example the frequency may change from being constantly around 2.50 to being constantly around 2.85.

Note: I will show in a future post a simple way to remove the day of week seasonality from metrics like visits so that trends can be more easily seen.

In the example on the left, there was a clear step change on the Thurs (day 11) with visits dropping by 20% every day from there. There is commonly only a partial change on the first day of a step change with the factor that caused the change commencing from part way through a day (change to website or online marketing).

The beauty of a step change is that it can allow you to pinpoint the exact day (and sometimes the approximate time of day) that something happened that is impacting the performance of your website. Through this, it is easier to identify what was the cause of the change. Additionally, if a business case is required, it is not difficult to calculate the impact of this change.

It should be kept in mind though that if a permanant step change occurs part way through a week, the weekly numbers will decline (or improve) further the next week when they have a complete week at this new level. I used the term permanant step change as the most common step change is temporary, occuring every school holidays.

Change 3 – The Spike

Another common cause of a weekly change in a metric is a sudden one to three day spike in the metric, either up or down. In the example on the right, there was a one day negative spike in visits on the Thursday which resulted in total visits being down 10% week on week.

In a way though, despite their impact, spikes are less important. As be seen, every other day of the week had exactly the same visits as the previous week. There is no reason to believe that the big drop in visits will be repeated. The cause of this drop in visits should be investigated but as it has already corrected itself, there is no reason to believe it will happen again.

Potential causes of spikes are servers crashes where the website either goes down or everything is fine but data wasn’t being captured, spam producing fake traffic, online marketing dropping off possibly due to budget temporarily running out, public holidays, etc.

Summary

A vital part of understanding what could have caused a change in a metric for a week (or a month) is looking at the data on a daily basis. By comparing each day against the same day in the previous week (or if need be, an alternative baseline week) and by looking at the data visually, it is possible to get a feel for when the change occured. This aids in identifying the cause of the change as well as understanding whether it is something to be concerned for the future or not.

Tags: Performance Reporting, Web Analytics