As data science evolves, key challenges are driving organizations to seek innovative solutions to compete in the new AI-driven economy.

copyright by www.informationweek.com

In late 2018, a survey by Univa found that 96% of respondents expected an “explosion in machine learning projects” in production by 2020. International Data Corp. forecasts that spending on artificial intelligence and machine learning will grow to $57.6B by 2020 . Fraud detection, customer analysis, churn prediction, and numerous other applications are driving the rapid growth of AI and ML. The world of AI, however, has a problem. A 2019 survey by Dimensional Research found that 80% of companies reported stalled AI and ML projects.

The following five factors are causing these slowdowns, each with its own set of challenges and opportunities:

1. Making data actionable for data science

The same Dimensional Research article found that 96% of respondents cited data quality and data labeling as crucial problems slowing their AI and ML adoption. Data silos are especially troublesome for data science.

Businesses store vast amounts of data, but often in different lines of business, across disparate systems and with varying levels of leadership and governance. Whether through manual processes or by leveraging automation, the first struggle for data science teams is to access and collect relevant data from different sources. Chief information officers and chief data officers must lead the charge to make data actionable for data science. Mitigating challenges related to data integration, ETL , security, and data privacy will drive faster turnaround of data science projects and making data science quicker and more efficient.

2. Shortage of data science talent

A 2018 LinkedIn survey found a lack of over 150,000 people in the U.S. with data science skills. The rapid adoption of ML and AI and the shortage of labor are likely to exacerbate the talent problem. To produce meaningful results, organizations must leverage statistical knowledge, data management, engineering, and subject matter expertise to tackle data quality, architecture design, and model production. Finding this multi-talented unicorn is impossible. Given the complexity of data science, it’s no wonder that 88% of data science graduates have a master’s degree and 46% a Ph.D. Addressing this problem requires expanded education as well as continued investments in growing the talent pool at a corporate as well as governmental level. New technologies to automate and accelerate the data science process also promise to reduce talent constraints.

3. Time-to-value must accelerate

The plodding pace of development also slows data science. Data science projects are iterative in nature due to the uncertainty of data and require a deep understanding of underlying business problems. Data scientists create a series of hypotheses to be tested and validated with actual business data by wrangling, cleansing, joining, combining, and aggregating data to identify data relationships and extract relevant patterns to build ML models. This process requires a rigorous trial and error approach to find the right answers, often involving multiple exchanges between business and data science teams prolonging projects.[…]

read more – copyright by www.informationweek.com

Image: Dmitry – stock.adobe.com