There is a lot of comment on Big Data and what it is, but according to the Telsyte Big Data market study, one in four Australian businesses lack a Big Data strategy as the main blocker for their IoT adoption, despite the fact 83% of these CIO’s plan to invest more in Big Data platforms. Here we’ll provide some useful tips on how an Australian company can start analysing Big Data to gain a competitive advantage using technologies such as the Qlik business intelligence (BI) software platform.
Businesses are now able to capture, store, and process an incredible amount of data about almost everything. But a Big Data repository has no inherent value, and many companies that have invested heavily in Big Data technologies are still learning how to leverage it.
According to a recent Telsyte Australian Big Data & Analytics Market Study in 2017, more than half of Australian CIOs expect a fivefold or higher increase in the number of connected devices in their enterprise within the next five years, but one in four organisations note that a lack of a Big Data strategy is a blocker for their IoT adoption. Despite this, 83% of Australian CIO's plan to invest more on Big Data platforms.
83% of Australian CIO's plan to invest more on Big Data platforms.
A Quick Recap On What Big Data Is
Big data is the term that describes the large volume of data - both structured and unstructured - that inundates a business on a day-to-day basis. Usually characterised by analysts around:
- Volume - the amount of data to collect and store from varying sources such as business transactions, social media and information from sensor or machine-to-machine data, IoT.
- Velocity - The speed at which data streams. RFID tags, sensors and smart metering are driving the need to deal with torrents of data in near real-time.
- Variety - The different types of formats data comes in - from structured numeric data in traditional databases to unstructured text documents, email, video, audio, stock ticker data and financial transactions.
Big Data flows can be highly inconsistent with periodic peaks which makes data loads challenging to manage, especially with unstructured data. The fact that data also comes from multiple sources, makes it difficult to link, match, cleanse and transform across systems. It’s necessary to connect and correlate relationships, hierarchies and multiple data linkages or your data can quickly get out of control.
It’s important to remember that the primary value from Big Data comes not from the data in its raw form, but from the processing and analysis of it and the insights, products, and services that emerge from analysis. The sweeping changes in Big Data technologies and management approaches need to be accompanied by similarly dramatic shifts in how data supports decisions and product/service innovation.
According to the Telsyte study, 30% of Australian enterprises who are using or planning to use, Big Data, are looking at predictive analytics.
What Type Of Big Data Applications Are Australian Businesses Using?
Telsyte found the intention to use Big Data analytics are high across a range of applications for Australian businesses, including financial modelling, customer interaction, security and fraud detection, retail sales and ecommerce, and IoT and machine to machine infrastructure.
Sales and marketing applications are one of the top three line of business use cases flagged by CIOs – one third of whom say they are looking to use Big Data analytics for this application.
However uptake is lagging for marketing teams, with just 15% of marketing departments having implemented Big Data analytics. So an opportunity exists here for the marketers among us to lead in this area as a means to differentiate.
The benefits Australian CIO’s are seeking from their Big Data and analytics strategies are to improve productivity and decision making, with better product and service development now the number one business priority for Australian IT leaders.
However, the main barriers to adoption are factors such as software costs, lack of in-house skills and IT infrastructure requirements.
Companies like Woolworths are taking a big leap into the Big Data game, investing some years ago a quarter of a billion dollars into its Big Data endeavours in order to better analyse its consumers’ online and in-store spending habits. While Telstra uses Big Data to improve its marketing strategies and customer service by analysing massive amounts of consumer data in real-time. One of our long-standing clients, Qantas, has deployed Qlik Sense to reduce the impact of flight disruptions and maximise occupancy on planes by coupling data from its business applications with external weather and flight path data.
Where Do I Start?
The first question you need to ask before diving into Big Data analysis is what problem are you trying to solve? Having now looked at what others are doing, are you interested in predicting customer behaviour to prevent churn? Do you want to analyse the driving patterns of your customers for insurance premium purposes? Are you interested in looking at your system log data to ultimately predict when problems may occur? The kind of high level problem is going to drive the analytics you use. If you’re not sure, you can always look at what areas of the business that need improvement.
Once you’ve chosen one simple, well-defined issue, explore it and demonstrate a little value. Repeat those steps, over and over. Begin by taking a manageable sample of your data that’s suitable for the analysis you want to perform - this might prove difficult if your data is spread across disparate data sources, or might be in one database that requires complex queries to get the sample you’re after. You need to think about the data you need and how it needs to be organised. Identify any special requirements (such as legal) for handling the data and ensure compliance.
The sources for Big Data generally fall into one of three categories:
- Streaming Data - this is data that reaches your IT systems from a web of connected devices. You can analyse this information as it arrives.
- Social media data - this is data on social interactions, particularly relevant for marketing, sales and support functions. It’s often unstructured or semi structured so often poses a challenge.
- Publicly available sources - Massive amounts of data are available through open data sources.
After you’ve identified the potential sources of data, you’ll need to consider how to store and manage that data, how much of it to analyse and how you’ll use the insights you uncover. Once you’ve got your sample you can perform data exploration, analysis and modelling. Try and keep your models simple to start with and rely on the same analytic methods you would use for any other data source.
Make Big Data Accessible To the Whole Business
Real value from Big Data is created when businesses can bring together data - big or traditional - from multiple sources or locations, and present that information in a way that encourages exploration and insight.
The ability to make Big Data accessible to the entire business drives value in two important ways. First, it means organisations are including Big Data sources into standard business analysis, thus gaining more detailed insight into key aspects of its operation. Second, it fosters a culture of inquiry in which experience and gut feel is supplemented with the power of Big Data. One of the significant issues expressed by IT leaders around the topic of Big Data is the struggle to find the ‘needle in the haystack’. By making Big Data accessible, more people will experiment with ideas around their data, eventually leading to greater business value.
Getting Insight Is As Much About Relating Data As It Is Collecting It
There are many tools focused on improving the ability for data scientists to perform analysis on massive amounts of data. But it’s important to go beyond the data scientist and empower all business users to perform analysis, regardless of technical skill. If we look at Qlik BI software as a case in point, it does this through:
- A complete view of information - Qlik’s data integration tools bring together multiple disparate data sources to provide a comprehensive picture of the business. Qlik can connect to virtually any data source - including file-based sources like Excel and XML web content, application specific data like Salesforce, ERP etc and Big Data sources like Hadoop, Teradata and Cloudera.
- Interactive, free-form exploration & analysis - Qlik’s patented, Associative Engine ensures that every piece of data is dynamically associated with every other piece of data, across all data sources. These associations can be extremely useful if there are hundreds or thousands of products, customers, geographies, etc. Such extremely large datasets can be sliced with a few clicks rather than scrolling through thousands of items. Context and relevance go hand in hand and quickly take what seemed to be a Big Data problem down to something that is quite manageable without any programming or advanced visualisation skills
- Multiple methods to support Big Data - Because Big Data use cases and infrastructure differ in every organisation, Qlik offers multiple techniques - which can be used individually or in combination - to best meet your Big Data needs. Such as:
- In-memory - Utilising the Qlik Indexing engine (QIX) optimisation of in-memory storage to compress data down to 10% of its original size.
- Segmentation and chaining - Sectioning multifaceted data views into subject-specific views and then chaining these separate views with each other.
- On Demand App Generation - Empowering the user to automatically create a purpose-built analysis app every time they select a slice of a very large data source.
- Other methods - A robust set of Qlik APIs as well as a variety of partner technologies can be used for situations like a custom User Interface.
Australian businesses are gearing up to continue to invest in Big Data analytics as a source of differentiation, particularly to improve products and services, however the key is to bring together data - big or traditional - from multiple sources or locations, and present that information in a way that encourages exploration and insight. If you’d like to know more about how Inside Info and Qlik can help you with Big Data analytics, check out this ebook on 10 Ways To Transform Big Data Into Big Value.
