There is a lot of hype in the world of reporting and analytics at the moment around Big Data. Is Big Data the silver bullet for all BI problems or is it just a passing fad?
Traditional definitions of Big Data refer to it in the context of Volume, Variety and Velocity. In this sense, it is about more than just size, it also about different types of information such as structured and unstructured, and the speed with which this is generated and changes. For example, retail giant Wal-Mart feeds more than 1million transactions an hour into databases estimated at more than 2.5 petabytes. Facebook’s estimated 750 million users create an average of 90 pieces of content each month and an average of 294 billion e-mails are sent every day.
Despite this, Big Data is in some ways not a new concept. Organisations have always been adept at generating large amounts of data and the explosion in variety and velocity through things such as social media and the incessant chatter of the ” internet of things” threatens to be the proverbial straw that breaks the camel’s back.
The real challenge with Big Data lies in creating meaning through insight and it is here that technology enablement is at an inflection point. Advancements in analytical capabilities are now at a point where the creation of meaning from mass is achievable. Economist Intelligence Unit research indicates however, that most businesses have made slow progress in extracting value from big data and some companies attempt to use traditional data management practices on big data, only to learn that the old rules no longer apply.
Technology is however only part of the solution and having a technology centric approach will likely result in organisations being overwhelmed by information irrelevance. Data does not make decisions, people do, and more data does not automatically result in better decisions.
Miller’s Law suggests that the number of objects an average person can hold in working memory at a point in time is 7 plus or minus 2. While the ability of technology to store, process and sort data has grown exponentially over time, human cognitive ability has arguably not grown at the same rate. The challenge for Big Data is therefore one of relevance – how do you synthesize the masses of data into the 7 plus or minus 2 objects to support effective decision making?
Keeping this in mind Big Data does offer tremendous opportunity to drive value. A recent study by McKinsey has suggested US retailers could realise in excess of a 60% improvement in net margin and consumers globally 0 billion in value through the creative and innovative use of Big Data. They have further indicated that Big Data will become a key basis for competition and growth.
While another global survey In June 2011 by the Economist Intelligence Unit of 586 senior executives, looked at the state of big data, the organisational characteristics of companies that are adept at extracting value from data and the most challenging aspects of data management. The research found:
- Extracting value from big data remains elusive for many organisations. For most companies today, data is abundant and readily available, but not well used. Nearly one in four said the vast majority of its company’s data is untapped. Another 53% say they only use about half of their valuable data. Yet 73% say that data collection in their organisation has increased over the last year. These figures indicate that organisations are still learning how to manage Big Data.
- Many companies struggle with the most basic aspects of data management, such as cleaning, verifying or reconciling data across the organisation. Nearly one-third of respondents admit their data governance practices are insufficient. Many struggle to deliver important data to the right people within an acceptable timeframe. And there is also a depth of workforce skills required to sift through, analyse and develop insights from Big Data. The lack of the right skills to manage data effectively is among the top two challenges cited by CIO’s. While the most difficult process right now is reconciling disparate data sources.
- Companies that are furthest along the data management competency continuum provide a useful model for how organisations will need to evolve if they are to extract and utilise valuable data-driven insights. Strategic data managers use data to first identify specific measurements and data points that align closely with corporate strategic goals. They select the most appropriate data to make decisions, and put a high percentage of the data they collect to use. They are also more likely to assign a C-level executive to manage data strategy, and they continue to explore emerging sources of data for potential value.
Of course, Big Data will be but one of the tools that companies use to inform decisions. But companies if they are going to effectively harness Big Data need to start with the end in mind and have a razor-sharp focus on outcomes. It also requires some unique people skills in terms of data management and the ability to harness data as a critical factor of production on a par with land, labour and capital.
