Guest post by Amanda Greenwood.
It’s been nearly a decade since interest in big data began taking off. Yet because the term is used in so many different contexts—from artificial intelligence (AI) applications to business intelligence to ecommerce to the Internet of Things (IoT)—there’s still a lot of confusion around it.
What is big data & where did it come from?
Stripped to its essence, big data is a volume of data so large that traditional data processing tools and methods such as Excel spreadsheets or text processors can’t handle it.
“The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.” – Forbes
Due to advances in technology, 90% of the data in the world was generated within the last two years.
We, as individuals, create 1.7 megabytes (one million bytes) of data every second, which equates to 1.145 trillion megabytes per day.
And, by the end of 2025, we’re likely to be generating around 463 exabytes of data every day.
To put that into perspective, most smartphones now come with 64 gigabytes (one thousand million bytes) of storage. 64 gigabytes is only .0000000064 of an exabyte.
Image credit: Backblaze
That’s a lot of data. Which is why it’s called “big data.”
But big data isn’t only about volume…
Big data and the three V’s
Big data is a relatively new term, but the concept of big data—the gathering, storing, and handling of data —is an age-old one (well, two decades old).
It began in the 2000s when industry analyst Doug Laney characterized the collection and harnessing of extremely large amounts of data into three V’s:
- Volume: Refers to the amount of data being processed.
- Velocity: Represents how fast the data can be processed.
- Variety: States the different types of data that are being processed.
These three properties define big data.
“Big data is data that contains greater variety arriving in increasing volumes and with ever-higher velocity.” – Oracle
That’s what big data is, but…
Where does all this data come from?
Most businesses are inundated with big data every day, and most of it comes from three primary sources:
- Social data: This data is collected from things like web searches and the likes, tweets, comments, shares, videos, and photos across all social media platforms.
For example, over 500 terabytes of data get ingested into the databases of Facebook every day. - Machine data: This is data that’s gathered from machine learning. So, things like web tracking tools, image processing, smart meters, road cameras, satellites, games, and IoT / IIoT devices. For example, Rolls Royce generates 10 terabytes of data for just one simulation of a jet engine.
- Transactional data: This is the everyday transactional data businesses collect. Invoices, payment orders, storage records, and delivery receipts are a few examples.
For example, the New York Stock Exchange generates one terabyte (one trillion bytes) of new trade data per day.
That’s a lot of data. But how can we use it?
Why is big data useful?
“While many organizations understand the importance of data, very few are yet seeing the impact of it.” – CloudMoyo
As traditional data processing methods can’t handle the volume, velocity, and variety of big data, managing big data is a challenge requiring new tools and techniques.
These massive volumes of data are incredibly useful. Using advanced software systems, analysis of big data can uncover patterns, correlations, and insights that can inform big business decisions quickly, helping companies stay lean and agile. It can be used to build relationships, remain competitive, and establish opportunities they wouldn’t have otherwise been able to uncover.
“[Big data] can help companies make sense out of random information, become proactive and start setting the pace instead of continuously putting out fires and following the competition.” – WishWorks
What can big data be used for?
Netflix harnesses big data to offer its subscribers personal recommendations for what film or TV series to watch next. With over 100 million subscribers, they have plenty of data to work with.
Analyzing your past search and watch data gives them deep insights into what interests you the most and allows them to show appropriate programs. This has helped them save around $1 billion per year on customer retention.
Walmart has capitalized on big data to become one of the largest retailers in the world, with over two million employees and over 11,500 stores worldwide. The company relies on big data to monitor how efficient its workflows are in its distribution centers and stores.
It uses this data to make continuous improvements to its distribution processes to increase efficiency and improve productivity.
Ever booked an Uber and had to pay twice the price you were expecting to? Uber uses big data to formulate its surge price strategies. It uses passenger data to analyze usage patterns, determine the supply and demand of drivers, and alter its prices accordingly.
These are examples of how some companies are using, and benefiting from, big data to improve productivity, target their marketing efforts, and inform effective pricing strategies.
But big data can also be used for:
- Location tracking. With big data, logistic companies can collect and use data about traffic and weather conditions to determine the best routes for transport. This helps them mitigate risks in transport and improve the speed and reliability of delivery.
- Fraud detection. Banks and financial institutions use big data to prevent cyber crimes and fraud. By analyzing historical customer and cyber-attack data, banks can predict future attempts and put measures in place to stop them.
- Precision medicine. There are many variables that can impact the manufacture of certain biopharmaceuticals, such as insulin. Using big data, biopharmaceutical companies are able to identify risks to production and take steps to eliminate them so they can produce safe and effective drugs.
“Big data can be analyzed for insights that lead to better decisions and strategic business moves” – RDA
To fully capitalize on all the opportunities presented by big data still requires talented (and expensive) analysts using sophisticated tools.
But as this technology continues to get simultaneously more powerful, easier to use, and more affordable, the benefits of harnessing big data are becoming available even to midsized and small businesses. Perhaps even to your company.
