‫Welcome back to another lecture.
‫In this lecture,
‫we're going to get an introduction and motivation
‫for big data processing.
‫So let's start with describing what big data means.
‫Big data is a term used to describe datasets
‫that are either too large in size, too complex in structure,
‫or come to our system at such a high rate
‫that exceeds the capacity of a traditional application
‫to process it fast enough to provide any value.
‫There are many characteristics of big data,
‫but three of them are the most prominent.
‫The first characteristic of big data is volume.
‫Volume refers to the quantity of the data
‫that we need to process, store, and analyze.
‫When we talk about big data,
‫we're talking about large quantities of data,
‫typically in the order of magnitude of terabytes, petabytes,
‫or even more per day.
‫A few technology fields where we have a high volume of data
‫include internet search companies
‫that have to analyze the entire internet
‫and provide instant search capabilities for their users.
‫Another example is medical software systems
‫that collect, analyze, and store a lot of information
‫about patients in hospitals or clinics
‫and can help in preventing or detecting diseases.
‫Then we have real-time security systems
‫that analyze multiple video streams coming from cameras
‫located throughout certain neighborhoods, cities,
‫or high security facilities.
‫The purpose of those systems is to help combat crime.
‫Finally, we have the weather prediction systems
‫that analyze a lot of data
‫from different sensors located on satellites,
‫as well as in different locations
‫throughout the large geographical region.
‫Those systems can help us predict the weather,
‫as well as alert us about upcoming storms or tsunamis.
‫The second characteristic of big data is variety.
‫In traditional non-big data systems,
‫we typically work with a limited number of structured
‫and well-defined types of data.
‫However, when we move to the field of big data,
‫we can have a large variety of potentially unstructured data
‫that we collect from multiple sources.
‫Our goal is to process all that data and combine it together
‫through a process called data fusion.
‫This can help us find hidden patterns
‫or provide business insights for our organization
‫that aren't obvious if we analyze only one data source.
‫An example of that are social media services or apps
‫that collect a lot of different types of data
‫about the behavior of their user base in real time.
‫For instance, they can collect information
‫about the user's clicks, likes, shares, or posts,
‫as well as capture the amount of time
‫a user spent watching a particular video
‫or even hovering over an article or a digital ad.
‫All those types of seemingly unrelated activities
‫can be combined together and build models
‫that can predict the behavior and response of each user
‫to future ads of particular products.
‫But also on an aggregate level,
‫if we combine all that data from multiple users,
‫we can detect internet trends,
‫as well as clusters of interest.
‫Now finally, the third characteristic
‫of big data is velocity.
‫When we deal with big data,
‫we normally have a continuous stream of data
‫that comes to our system at a very high rate.
‫The high rate of incoming data
‫can be either due to the large scale of our system
‫or simply the high frequency of events.
‫For example, if we have an online store
‫that operates on a global scale
‫with millions of users visiting our website every day,
‫browsing and purchasing our products,
‫then the higher rate of events simply comes from the fact
‫that we have a very large number of users.
‫On the other hand, we have the field of internet of things
‫which deals with connecting multiple devices
‫and getting analytics from their sensors.
‫In this case, we can have a relatively small fleet of buses,
‫trains, or autonomous cars,
‫but all those vehicles can generate
‫lots and lots of data points from their sensors
‫about their location, speed, surrounding objects and so on.
‫Similarly, we can have a food or clothing production factory
‫which is full of robots, assembly lines, and other machinery
‫that constantly generate data about their production quality
‫and speed from their sensors.
‫In both those use cases,
‫the number of devices may not be large.
‫However, each sensor on a robot, a piece of machinery,
‫or autonomous car can generate
‫a continuous stream of data points
‫that we have to ingest very quickly.
‫Now, it's important to point out that storing
‫and processing big data is pretty complex,
‫as well as very expensive,
‫but the value we get from it
‫usually outweighs the cost associated with it.
‫The insights we get from analyzing big data
‫can provide a significant competitive advantage
‫over our competitors.
‫Those insights can come in a form of visualization,
‫querying capabilities or predictive analysis.
‫Visualization is a very powerful tool that can allow humans
‫to make sense of otherwise meaningless data
‫stored in some file system or database.
‫In many cases, after we collect a lot of data,
‫we don't necessarily know what to do with it
‫or how we can benefit from it right away,
‫so querying capabilities allow us to run
‫ad hoc analysis on that data,
‫which eventually helps us find those insights or patterns
‫that were not obvious before.
‫And finally, on the predictive analysis front,
‫we can go as fancy as building algorithms
‫or machine learning models
‫to predict the behavior of our users
‫and suggest products that they will more likely purchase.
‫But we can also go as simple as detecting anomalies
‫in our system by analyzing logs coming from our servers
‫and automatically roll back a new release
‫or alert the engineers on-call.
‫So now that we have the understanding and the motivation
‫behind big data processing,
‫let's proceed and learn about a few architectural styles
‫that help us in processing and analyzing big data.