Three lessons learned from 5 years analytics experience in a Fortune 100 company

 

I have recently wrapped up 5 years (4 years + 9 months to be exact) of analytics career in a Fortune 100 company. In this precious in-between moments, I’d like to take the time to reflect on 3 lessons learned as an analytics professional in the digital marketing industry.

“Correlation” versus “Causation”.

Correlations (A happened, along with B) is arguably the most commonly applied data and analytic method, and we have a tendency to mix it up with causation(A happened, cause B to happen).  This BBC story on Kylie Jenner costing Snapchat $1.4B is is a typical example.

For the data available from the marketing world, the traditional marketing channels(TV, Radio, Brick and Mortar Point of Sales) can be exclusively measured by “correlation” to be best of my knowledge.  The most commonly used method is sales lift – “With this TV campaign/off-shelf display/temporary price reduction, the POS is up by 50% “. We are almost certain to jump into conclusion by adding promotion cost and incremental gross margin to calculate ROI.

In reality, the +50% lift can be driven by many different other reasons outside the promotion – weather, special events, retail product availability, packaging, consumer confidence shift, just to name a few.  We need a rigidly controlled test isolating those factors, in order to make the connection between correlation to causation. It didn’t happen very often. And the marketing department ended up settling with a sales lift measurement, usually in its favor, and fool the novice but never convince engineering driven IT department.

On the flip side, digital marketing is born with the edge of finding out “causation” by tracking visitor’s online clicking behavior.  Web analytics tool like Google Analytics and Adobe Analytics can track every step of the funnel toward the final check out (eCommerce) or lead form submission(lead-gen), and pinpoint back to visitor(browser cookies) level.  In other words, finding out what is driving the sales in the online world(causation factor) is possible – it is not solved yet in my opinion.  Tim Wilson(Sr Director of Search Discovery) wrote in this article explain well the concept of “Attribution” – It will take hundred thousands of dollars for organizations to figure out its own path. But it is doable, and that is one of the reasons why the digital world attracts me, as a proud analyst.

Bottom line: correlation analysis is still important and much better than a gut check, but understanding the causation takes much harder work and it provides much higher confidence for effective decision making.

“Analysts’ own biases”.

Fundamentally the job of an “analyst” is to JUDGE: analyze the situation using facts(data) to tell stakeholders – how are we doing, and why.  By the nature of the work, we(as analysts) are living and breathing in the data world and will have a biased tendency toward purely using data to interpret, without getting enough non-factual context. Twice I was embarrassed by it and I can still remember vividly today:

• At the beginning of my digital journey, I was gathering information on the social media pages of one of the consumer-facing brands I support.  Majority of the recent feedbacks and reviews were negative – claiming products difficult to use, lackluster packages etc.  My conclusion was “damn it, our customer hates our products.  We gotta do something about it”.   I shared the “insights” with internal business stakeholders – who politely listened but did nothing.  Not long after that, I was told this brand just had one of the best sales records in this company’s history.  I was preoccupied with biased thinking social media respondents are a good representation of our customer base. The reality is they tend to surface more motivated outliers instead.
• I was relatively experienced at this stage and got a chance to present to a healthcare business on its newly launched website performance.  Without a reasonable baseline to compare, I decided to use traffic from another safety business, and subsequently made a bold claim that “We have a lot to catch up, as we are only receiving 4% traffic compared to XXX”. The whole room is dead silent. After a while, the highest paid person in the room said: “We are just building our own baseline and we gotta be patient”.  I walked out that meeting feeling awful.  Later upon my own reflection, I realized I picked the comparable business because it is the one I am most familiar with, an unconscious and biased decision. The healthcare website is targeting to professionals like nurses and doctors, while the safety website could be talking to a much wider audience. They are simply not comparable.

Being tricked by the motivated outliners, or unconsciously taking the shortcut, are the two examples I surrendered to my own biases.  We can get better at dealing with them thru experience, but it can’t be completely eliminated, as new challenges are always rising in this ever-changing industry.

Bottom line –  We need to be conscious of our own bias, and make it up by adding more context to validate our original judgments. A simple phone call, a 15 minute sit down with marketers could easily avoid my embarrassment. Otherwise, artificial intelligence analytics are coming after us. Or maybe it is inevitable.

 

“Audience-based data visualization”

After my 1st year, my deliverables to the organization had shifted gradually from reports and analysis (Excel Spreadsheet) to data visualization products(PowerPoint slides, PowerBI Dashboard) over the next 4 years.

One way to look at it – I was moving up the food chain and other more junior analysts are handling the “dirty” data wrangling work, while I can focus my energy to deliver more “value” to my stakeholders. The “value” is the influence in decision making – buy more products and services(increased sales), and spend budget more wisely(reduced cost). In the modern world, decision making requires numbers, nice looking number in particular.

Part of me was convinced, and I became passionate about pushing my own and the organization’s boundary of data visualization. The effectiveness of it always comes down to the receiving end:

• Mass audience:  Everybody loves simplicity.  A giant number with well-crafted percentage change to benchmark is a low-hanging fruit visualization for communicating numbers.  Bar chart and line chart is one step further, as long as it is following the same simplicity principle.  I usually include the source of my data to improve credibility as well.
• Executive/Business Leaders: Dashboard is their favorite. Start with the basic 3-color-coded numbers, different executives might prefer a different level of detailed data. Drill down feature is super handy for those are hands-on and like to explore the data by themselves.  The hard truth is it is exponentially more work to have drill down available, while not many of the audience are actually using it (almost all of them will claim it is needed)
• Front line marketers: They also like “dashboard”, but it is a totally different type – data heavy, drill down is a must.  The best medium to carry this type of “dashboard” is Excel’s Pivot table, and I wrote in this blog post on the advanced PowerPivot feature.

Part of me was in doubt for the shift of separation between data acquisition and data visualization/storytelling.  It’s hard to admit, but the truth is the anxiety builds up quickly when someone else (that I don’t know if I can trust 100%) is tasked to prepare the data for me.

Bottom line – Know your audience first to plan for the visualization, and have someone you trust to prepare the data(if you have the luxury to have one)