‫Hey, welcome back.
‫Now that we got a good introduction
‫to the field of Big Data in general,
‫let's learn about two architectural patterns
‫for processing big data using event driven architecture.
‫But before we talk about big data processing,
‫let's clearly define the problem we're trying to solve.
‫Let's assume we're getting a continuous stream of data
‫coming from different sources into our system.
‫This may be user interactions
‫on our social media platform, logs or metrics
‫coming from production application instances,
‫or data coming from transportation devices
‫such as planes, trains, or autonomous cars.
‫All this large volume of raw data
‫is coming to us in real time.
‫And we want to process it so we can analyze it
‫and provide insights, visualization, or predictions.
‫Now how do we go about processing this data?
‫Well, there are generally two strategies
‫or patterns we can use.
‫The first strategy is called Batch Processing.
‫When we use batch processing,
‫we normally store the incoming data
‫either in a distributed database
‫or more commonly directly on a distributed file system.
‫That data is never modified
‫and we only append to it as more data arrives.
‫Now the key principle in batch processing
‫is we do not process each piece of data
‫that comes in our system individually.
‫Instead, we process that data in batches of records
‫and we typically perform this processing
‫on a fixed schedule.
‫Though we can also run it based on a fixed number of records
‫that we want to process.
‫This batch processing can run on any schedule
‫that fits our use case,
‫which can be once a month, once a day,
‫once an hour, and so on.
‫The logic that goes into that job
‫that processes this data is written by us
‫and we can easily update it whenever we want to.
‫Every time this batch processing job runs,
‫it picks up the new raw data
‫that came since the last time it ran,
‫then it analyzes it and produces an up-to-date view
‫of all the data we currently have.
‫This view can be stored in a well structured
‫and indexed database that we can easily query
‫to get the desired insights.
‫It's important to note
‫that this view we generate should reflect the knowledge
‫we have about our entire data set.
‫And depending on the use case,
‫this batch processing job can pick up only the data
‫that arrived recently
‫or it can process the entire dataset
‫from scratch to provide us with a new view of the data
‫we collected.
‫Let's look at two perfect use cases
‫for batch processing to illustrate the process.
‫Let's assume we have
‫an online learning subscription platform
‫where we offer thousands of video courses
‫to millions of students around the world.
‫While students watch those video courses,
‫they can leave a review and rate the course
‫that they are currently watching
‫to reflect their satisfaction from the course
‫at that point in time.
‫On our end what we get is a stream of two types of events.
‫The first type of event indicates the progress
‫a student is making by watching a video course.
‫So for example, every additional minute
‫that a student watches, we get an event into our system.
‫We can roughly estimate
‫that if we have a hundred million students on our platform,
‫even if only 10% of them are active on our platform
‫at a given moment,
‫we can expect about a million events per minute.
‫So this definitely falls within the scope of big data.
‫Another type of event we get is a review and a star rating
‫for a particular course.
‫Now we do not need to process any of this data in real time.
‫However, if we process it in batches
‫and take into account historical data for each course,
‫we can provide a lot of mission critical insights
‫for our business.
‫For example, we can use the data
‫about the content consumption for each course
‫to compensate our instructors based on the percentage
‫of the content that was consumed from their courses
‫out of all the available content on the platform.
‫Also, based the ratings we get for each course,
‫we can recalculate the average rating for each course daily.
‫However, we can go much farther
‫and fuse the data from those two types of events.
‫For example, when we calculate the average rating
‫for each course, we can provide higher weight
‫to ratings that came from students
‫who watched a higher percentage of the course
‫and give lower weight to ratings that came from students
‫who barely watched just a few lectures.
‫This way if we have 10 students that left a one star rating
‫and watched less than 10% of the course,
‫and we have another 10 students who left a five star rating
‫and watch the entire course,
‫we can rate the course much closer to five stars
‫with more confidence that this reflects the quality
‫of the course more accurately.
‫Another example of data fusion
‫we can have from those two types of events
‫is to rank the courses in each category
‫based on the rating, as well as the overall engagement
‫of the students in each course.
‫Finally, we can combine all that data
‫to build a machine learning model
‫and try to predict what type of students can benefit
‫from which courses and send them push notifications
‫or display those courses at the top of the page.
‫Another very common use case for batch processing
‫is search engine services.
‫When we have large amounts of data
‫such as websites, articles, or images
‫that we want to provide search capabilities for,
‫if we try to scan all of it
‫every time a user perform the search,
‫it would take us hours to provide a result.
‫So what search companies typically do
‫is crawl the entire data set periodically and organize index
‫and store it in a way that is very easy and fast
‫to search for.
‫So because this crawling process
‫takes a very long time anyway and there is no real need
‫or expectation that every new website or article
‫would appear in the search results immediately,
‫this is also a perfect use case for batch processing.
‫So now that we got some intuition for batch processing
‫from real use cases, let's talk about the advantages
‫and disadvantages of this processing model.
‫The first advantage of batch processing
‫is that it's very easy to implement
‫because we don't have to worry about latency.
‫Batch processing also provides us with high availability
‫because until a currently running job
‫is done analyzing the entire dataset
‫and providing the new view,
‫the old view is still available for us for querying.
‫So there's essentially no downtime for the users.
‫Another advantage of batch processing is efficiency.
‫Processing data in batches is usually a lot more efficient
‫than processing each incoming record individually.
‫With batch processing
‫we also have a much higher fault tolerance
‫towards human error.
‫For example, if we push some bad code to our processing job,
‫there is not much harm done
‫because we still have our original data.
‫So we can fix our bug, redeploy the processing job,
‫and run it again to analyze our entire data set
‫which would produce a new and correct view of our data.
‫And finally, batch processing can perform very complex
‫and deep analysis of large data sets
‫that can span years of data points.
‫This can provide us with very advanced prediction models
‫and insights.
‫On the other hand, batch processing has one major drawback
‫that makes it not suitable for many use cases.
‫This drawback is the long delay we have
‫between the data coming into our system
‫and the result we get from the processing job.
‫If we run our batch processing job every day or every hour,
‫we don't get a real time view of the data coming in.
‫And if we don't get a real view of what's happening
‫on our platform, we can't respond to it fast enough,
‫which is extremely important in many use cases.
‫This also forces our users to wait a long time
‫before they can get feedback on actions
‫they take on our system.
‫This in turn may cause some confusion
‫if they're not aware of the fact
‫that we analyze our data in batches.
‫For example, if we have an online store
‫that uses batch processing to index different products
‫on our platform for search purposes,
‫a merchant who just added
‫or updated their product description may be surprised
‫why their product doesn't show up in the search results
‫on our online store.
‫In this particular case, we can simply inform them
‫that those changes may take effect
‫only after one business day.
‫And they may be okay with that.
‫But in other cases, batch processing just won't work.
‫For example, if we have a log and metrics analysis system
‫that collects streams of log files and data points
‫from thousands of production application instances,
‫we have to be able to ingest and visualize all this data
‫in real time.
‫Otherwise, if there's a production issue in our data center,
‫the engineers on call have to have those logs
‫and metric graphs available immediately
‫so they can figure out how to fix the issue.
‫Similarly, if we have an online stock trading system,
‫we have to be able to take in both bids and asks,
‫match them together so people can trade on our platform
‫and also update the price on each stock in real time.
‫So the second way we can process big data
‫is real time processing.
‫In real time processing, we put each new event
‫that comes to our system in a queue or a message broker.
‫And on the other end, we have a processing job
‫that processes each individual data record as it comes.
‫After that record is processed,
‫the processing job updates the database
‫that provides querying capabilities
‫for real time visualization and analysis.
‫The obvious advantage of the real time processing model
‫is we can analyze and respond to data
‫as it comes to our system immediately
‫without having to wait hours for a job to process it.
‫The drawback for real time processing
‫is that it's very hard to do any complex analysis
‫in real time.
‫So we don't get as much value or insight into our system
‫as we get if we process it using batch processing.
‫Additionally, doing data fusion from different events
‫that happened at different time points
‫or analyzing historic data is also nearly impossible.
‫So we're limited to the recent data
‫that we currently have to provide those insights
‫or predictions.
‫In this lecture, we'll learn about two different strategies
‫for processing large volumes of data into our system.
‫Those two strategies are batch processing
‫and real time processing.
‫We covered quite a few use cases
‫that fit each of those models
‫and also compared the two strategies
‫by listing their advantages and disadvantages.
‫I will see you soon in the next lecture.