Definition of Big Data
Since the beginning of IT, the amount of data that is managed has grown. This growth has accelerated with the advent of IoT and social media. These data sources often produce less structured data (events, chats, streaming video, …). Data is often more real-time (e.g., current spindle rotation) with real-time processing needs.
Traditional databases are less suitable to handle data with this level of volume, variety, and velocity. Big Data describes the handling of such data, according to what is considered the original definition by Gartner 2001. (Later, veracity, value, and more dimensions were added.)
‘Handling’ includes storage, querying, transfer, and analytics.
An important requirement is analytics of ‘data in motion‘, directly in memory, like streaming analytics does. Storing data is still important, but storing first and analyzing ‘data at rest‘ later is not always a suitable approach for Big Data.
“Data is the new oil” (Clive Humby, 2006). Companies that adopted early and are able to manage Big Data are among the most profitable on the globe. The impact of Big Data on the economy, but also on society in general, can hardly be overstated.
Relevance for IoT and Integration
The tight relationship to IoT is obvious as IoT is one of the main sources of Big Data.