It's Soooo Easy to Misinterpret Data

Recently The Wall Street Journal poll held their annual Chief Information Officer conference, and polled the participants. Here are their top priorities, as voted on by the CIOs.

It’s important to read this graph correctly—each CIO chose their top priority, and that’s what this graph shows. It doesn’t show how CIOs ranked each possible priority. In other words, just because 'business intelligence and analytics' is first, and mobile is the lowest priority, it doesn’t mean that mobile is a low priority for CIOs—it just means that it isn’t most people’s first priority. It could easily be, however, many CIO’s second or third priority.

Incidentally, the ease with which this graph can be misread illustrates not just the importance of Business Intelligence (BI) and Analytics, but also the necessity of clarity in such information. An assumption that Mobile is a low priority, as opposed to just not everyone’s top priority, can lead to a dramatically incorrect business strategy.


 
Since the gist of the source WSJ article is the expanded role of the CIO in running a business, the top priorities of CIOs may also be top priorities for executive management:

Business Intelligence and Analytics: it’s great to see this as #1, because it shows how much IT has moved from just being technology infrastructure and making sure the base business software works, to taking an active role in running a successful business. From personal experience I can testify to how important it is to have a proactive IT department that provides impartial information on various business transactions and events--the CIO’s role in this is to make sure the interpretation of data is clear and correct.

Oftentimes executives don’t realize how many different ways data can be interpreted, and think just because it came out of a computer it is impartial. When marketing gives you a fancy report in Excel that shows that a particular mobile ad campaign resulted in a 2% click through rate and 5% conversion rate, what exactly does that mean? You can’t tell without knowing the assumptions and criteria that went into the numbers. For instance, if you click on a mobile ad, you typically go to a web page. Do you measure how long the person stays on that page? If you don’t, you should, because somewhere around 99% of clicks on mobile ads are ‘fat finger’ mistakes……here’s what happens to me—I’m scrolling through a news story, and this ad pops up and rolls along with the text, and on the next swipe I hit it by mistake. I have no idea what I’ve hit, and spend the next 3 seconds figuring out how to get back to the news story. The little ‘X’ in the corner comes up and I’m back to reading. That’s not a real click. You need to see how many people stayed on that website long enough to at least see what the ad was for.

And what’s a conversion---just because someone bought something? If they clicked through from the ad and made a purchase, great, but you can’t attribute all sales to an advertisement. I worked for a company that sold to public schools, and every year they’d mail to all the teachers, and if that school then bought the company’s books, they’d attribute it to the campaign. As might be expected, these campaigns were remarkably successful, but were they?? This particular marketing group considered sales from repeat customers, sales before the samples shipped, etc. as coming from the campaign.

A good IT group can make sure that executives know how to interpret the data they receive, and what assumptions went into it. Many things are tough to interpret, but they are impossible to interpret without knowing the whole story, and that’s where the CIO role comes in—making the IT function impartial, collaborative, and accessible.

One last note on business intelligence—this information also needs interpreted, and for this you really need a business analyst. BI tools are like Excel, but on steroids. Not only can the data be presented in different ways, but the data that goes into the BI tool that you can then massage on your PC also has been massaged before it got to you. That’s what the BI developer does. You are limited to what you can see by what the developer has put into the report. Here again, know the criteria, or ask your CIO!

Cybersecurity: This one isn’t as much fun, because you’re playing defense, not offense, but as we saw in the Super Bowl, defense wins football games…..further illustrating the pitfalls of analytics. File that last analogy under ‘misinterpretation of data’, because defense in football has nothing in common with cybersecurity.

I’m not going to tell you how to avoid a security breach like Target just experienced, because I don’t know all the answers, but here are some suggestions.

•    Take security seriously.
•    Do the obvious things well—password complexity, firewalls current, good antivirus that’s always up-to-date, etc.
•    Run reports that monitor activity, and LOOK AT THEM. Make it someone’s job.
•    Keep sensitive data offsite as much as possible. If you are a small company that accepts credit cards, don’t store them in your database, even if encrypted. Use a service to process the cards. You can link to these services via a key, so that your customer service people can see that last 4 digits of the credit card#, but that number is associated with a long key which is then transmitted to the processing service, which then gets the actual credit card number and processes the payment. The same concept applies to payroll info and the like…if you don’t need it at your business, don't keep it there. Keep it where it’s needed.
•    Hire real security experts to review your infrastructure and security, and I’m not talking about engineers who know how to set up a firewall properly. Do like the federal government does—hire reformed hackers to catch hackers. No one can help you to stop an attack as well as someone who knows how to launch a cyber attack. This is especially important if you have extremely sensitive or valuable records. Think of the cost of bringing in security experts like business life insurance.

Related reading by Jay Shenk:

A Layman’s guide to the cloud
The Deep Internet: What and where is it? (It's where the Target hackers bought the code)
Musings on Technology