Practical DevOps for Big Data/Conclusion
In July 2014, the EU Commission outlined a new strategy on Big Data, supporting an accelerated transition towards a data-driven EU economy. According to the "Worldwide Big Data Technology and Services, 2012–2015 Forecast" conducted by IDC1, Big Data services are expected to grow worldwide at an annual growth rate of 40% – about seven times that of the ICT market overall. Another recent study on "Big Data Analytics: An assessment of demand for labour and skills, 2012-2017"2, conducted by e-skills UK and SAS, predicts that in the UK alone, the number of Big Data specialists will increase by 240% over the next five years. DICE is a response to increasing needs of society to develop Big Data applications.
Today, most organizations face high market pressure, and their supporting ICT departments are struggling to accelerate the delivery of applications and services while preserving production and operations stability. On the one hand, ICT operators lack understanding of the application internals including system architecture and the design decisions behind architectural components. On the other hand, development teams are not aware of operation details including the infrastructure and its limitations and benefits. These issues are even magnified for Big Data systems. For these, recent years have seen the rapid growth of interest for the use of data-intensive technologies, such as Hadoop/MapReduce, NoSQL, and stream processing. These technologies are important in many application domains, from predictive analytics to environmental monitoring, to e-government and smart cities. However, the time to market and the cost of ownership of such applications are considerably high. We argue that, while typically considered a niche approach for experts, model-driven approaches have the potential to automate several development activities thus helping to reduce the time to market. Also, operational data can be exploited using automated tools to give feedback to the development to accelerate the development activities.
In this context, DevOps practices address the common isolation between Development and Operations. The main goal of DevOps is to achieve a better and continuous delivery application lifecycle by eliminating silos and integrating ‘Dev’ and ‘Ops’ activities, with people sharing the same goals and working closely with automated tools and services that enables such interactions. DICE aims at incorporating within the DevOps movement a model-driven approach that facilitates flow of information from Dev to Ops and enables operations monitoring and anomaly detection capabilities to facilitate flow of information from Ops to Dev. Moreover, DICE offers the following tenets in the context of DevOps: