Market research and advisory firm Ovum estimates the big data market will grow from $1.7 billion in 2016 to $9.4 billion by 2020. As the market grows, enterprise challenges are shifting, skills requirements are changing, and the vendor landscape is morphing. The coming year promises to be a busy one for big data pros. Here are some predictions from industry watchers and technology players.
1. Data scientist demand will wane
Demand for data scientists is softening, suggests Ovum in its report on big data trends. The research firm cites data from Indeed.com that shows flat demand for data scientists over the past four years. At the same time, colleges and universities are turning out a greater number of graduates with data science credentials.
“Who is recruiting these prospects? In all likelihood, excluding online digital businesses, relatively few enterprises outside the Global 2000 are absorbing them, and few would have any idea of how to use data scientists,” Ovum writes. “For the mass of organizations that rely on packaged analytics, the need is not for data scientists per se, but applications or tools that apply data science under the hood.”
2. Making data science a team sport will become a top priority
Data scientists and data engineers play different roles in the enterprise: the data scientists form and test hypotheses, while the data engineers select data sets, provision clusters, and optimize their algorithms for production. Without collaboration, the models and hypotheses that data scientists develop run the risk of getting stuck on their desktops, Ovum warns.
“The real need is getting data scientists and data engineers better connected to ensure that the models that the data scientist has written and tested on his or her laptop gets deployed properly with the right data sets on the cluster (which is the data engineer's expertise),” Ovum writes.
At the same time, machine learning is becoming embedded in enterprise software and tooling for integrating and preparing data, Ovum notes, and that’s putting pressure on enterprises to make sure that their data scientists and business analysts are working closely together. “Enterprises will not gain the full value of machine learning if the models remain inside the heads of data scientists,” Ovum writes. “The overlying trend will be toward collaboration environments where business analysts and data scientists can share workflows in the planning, deployment, and execution of machine-learning models.”
3. There will be more pressure to keep data local
Global law firm Morrison & Foerster predicts an increase in privacy laws that aim to keep data in country.
“Expect more data localization laws following recent developments such as the first enforcement of Russia’s data localization law being upheld by Russian courts, and China recently passing its own data localization law. Other countries will follow in the year ahead,” say Miriam Wugmeister and Andrew Serwin, co-chairs of the global privacy + data security group at Morrison & Foerster.
4. Enterprises will struggle to monetize data
Businesses will have many options for productizing data, but it won’t be easy to do and many enterprises will miss opportunities, warns research firm IDC.
“Despite the wishes of business leaders, businesses will struggle to succeed in creating meaningful products and revenue streams. Those that do succeed will be underpinned by solid IT strategies and data-oriented services spanning: data acquisition; transportation, transformation, and storage; analytics and dashboards; data as a product/service; and security and access control,” the firm writes in its report, IDC FutureScape: Worldwide CIO Agenda 2017 Predictions.
Among the guidance IDC offers is a suggestion that IT leaders “set up an innovation team consisting of IT and business personnel that reviews existing and future applications/systems for possible monetization of resulting data.”
5. Data lakes will finally become useful
“Many companies who took the data lake plunge in the early days have spent a significant amount of money not only buying into the promise of low cost storage and process, but a plethora of services in order to aggregate and make available significant pools of big data to be correlated and uncovered for better insights,” says Ramon Chen, chief marketing officer of data management vendor Reltio. The challenges have been finding people with the skills to make sense of the information; enabling data lakes to provide input into and receive real-time updates from operational applications; and bridging the gaps between master data management and operational applications, analytical data warehouses and data lakes.
“With existing big data projects recognizing the need for a reliable data foundation, and new projects being combined into a holistic data management strategy, data lakes may finally fulfill their promise in 2017,” Chen predicts .
6. M&A activity will accelerate
“There’s no doubt that there’s a massive land grab for anything AI, machine learning or deep learning,” Reltio’s Chen notes. A key driver of all the deals has been demand for IA experts: “Due to the short operating history of most of the startups being acquired, these moves are as much about acquiring the limited number of AI experts on the planet as the value of what each company has produced to date,” says Chen, who predicts even more aggressive M&A activity in the coming year.
7. Demand for IoT architects will soar
The Internet of Things (IoT) is projected to become a $1.46 trillion market by 2020, according to IDC. As it surges, so will demand for skilled IoT experts.
“The Internet of Things Architect role will eclipse the data scientist as the most valuable unicorn for HR departments. The surge in IoT will produce a surge in edge computing and IoT operational design. Thousands of resumes will be updated overnight,” predicts Dan Graham, Internet of Things technical marketing specialist at data and analytics vendor Teradata. “Additionally, fewer than 10% of companies realize they need an IoT Analytics Architect, a distinct species from IoT System Architect. Software architects who can design both distributed and central analytics for IoT will soar in value.”
8. Streaming analytics will be reborn
“Analyzing data in motion is nothing new – event-processing programs have been around for nearly 20 years,” Ovum says. But there are a number of factors that are transforming real-time streaming from a niche technology to one that’s more broadly appealing. Open source technology, for example, makes real-time streaming more accessible, as does the availability of scalable commodity infrastructure, the firm points out. On the demand side, IoT is fueling interest in streaming applications that can sense, analyze and respond in real time.
What won’t happen right away is market consolidation: “Today, there is a growing array of choices that are competing for new workloads. Eventually, we expect that the market will winnow down to three to four streaming platforms,” Ovum predicts.
But platforms such as Spark Streaming and Amazon Kinesis Analytics – and their competitors – are still works in progress. “Given the early state of the market, we do not expect the market to winnow down in 2017; we expect it will take 24–36 months for streaming engines to mature and IoT implementations to attain critical mass,” Ovum says.