The world is already into the information age. The huge growth of digital data has overwhelmed the traditional systems and approaches. Big data is touching almost all aspects of our life and the data-driven discovery approach is an emerging paradigm for computing. The ever-growing data provides a tidal wave of opportunities and challenges in terms of data capture, storage, manipulation, management, analysis, knowledge extraction, security, privacy and visualisation. Though the promise of big data seems to be genuine, still a wide gap exists between its potential and realisation. In last few years, there is a huge surge in research efforts in academia as well as industry to have a better understanding of big data. This article discusses the following: (1) big data evolution including a bibliometric study of academic and industry publications pertaining to big data during the period 2000–2017, (2) popular open-source big data stream processing frameworks and (3) prevalent research challenges which must be addressed to realise the true potential of big data.
Towards a Data Warehouse Architecture for Managing Big Data Evolution