The Time to Change is Now

The world is speeding ahead at a significant pace towards a major revolution—the data-driven economy.  Several data-driven start-ups in the last decade have become large corporations (Google, Facebook, Twitter), with billions of people reached and influenced by their innovations. Here is a list of the hottest start-ups that are looking to mature to the big league.

As the momentum is picking up, major organizations from different industry verticals are in a quest to exploit the opportunities that have arisen from the humongous amount of data that their business generates, directly and indirectly. Philip Evans, Senior Partner, Boston Consulting Group, discussed in his TED talk what businesses would look like in the future, and the impact that Big Data will have on business strategies.

Whether businesses want to use data to make the world a better place, to understand the wishes of customers before they’re expressed, to be more proactive than reactive in decision making based on predictive technologies, or something else, there are several challenges that we must all face.

These challenges include the following.

1. Data volumes are ever-increasing. Most of the data is unstructured (either textual, videos, graphs and so on) rather than transactional and structured.
2. The decision cycles are becoming shorter. We expect millisecond response times from the systems we interact with. And with mission-critical applications, the response time could be even shorter.
3. Thousands of predictive models are required to get coverage of all the predictive scenarios that an application can create.
4. Traditional methods of modelling are very time-consuming. The quest to find a perfect model drains valuable time and money before it can be put to business use.
5. The knowledge workers who understand data science, and who could mine useful actionable nuggets from the data, are rare. The demand for such skilled workers is ever-increasing and their lack of availability is causing a massive skills gap.

With Challenges Comes Opportunities

However, with challenges come opportunities.

Consider the Industrial Revolution. As we know, at that point in history the move was to automate processes that were repetitive or required more manual effort, and find ways to free valuable resources—the brain and imagination—that we use to focus on even larger problems. The result is the modern world we now live in.

Now the data revolution is demanding a new change. That is, the way in which we work with data. We must find ways and means to automate most of the repetitive workflows and modelling processes that are applicable industry-wide. This way, we can free the very valuable time of the data scientist to focus on tough problems that cannot be solved without human intervention.

With several thousand models that enable a data-driven company to run, it’s also important to have capabilities that enable the company to monitor the performance of these models in real time. This means decommissioning the models that exhibit significant deviation in performance, as compared to when they were deployed on production systems.

This paves the way for the need of a Massive Predictive Factory, a single source of truth and heart-beat monitor for the entire organization.