‫Welcome back to another lecture.
‫In this lecture, we're going to continue our discussion about big data processing.
‫We will build upon the knowledge from the previous lecture and learn about the very powerful modern
‫architecture for processing big data.
‫So if we remember from the previous lecture, we had two strategies that we can use to process large
‫volumes of data coming into our system.
‫We had the batch processing strategy which provided us with a lot of benefits.
‫In particular, since our batch processing job operated on our entire data set, we could get deep insight
‫into historic data as well as the ability to fuse data that came from different sources and build prediction
‫models that could benefit our users as well as our business.
‫However, when we chose batch processing, we had to make a tradeoff and settle for high delay between
‫the data coming into our system and the time we can query that data from the database.
‫On the other hand, we had the real time processing strategy, which gave us only visibility into recent
‫data.
‫However, it allowed us to query and respond to data in real time as it came into our system.
‫So for each use case, we had to choose and sacrifice either the fast response time or deep analysis
‫into historic data.
‫And there are indeed use cases that are perfect for just batch processing where the delay is not that
‫important.
‫And there are other use cases that are perfect for real time processing that do not really need any
‫deep analysis that involves historic data.
‫However, in many cases, making this decision between real time processing and batch processing is
‫very hard because we need the properties of both strategies.
‫Let's look at a few examples to get some motivation.
‫For example, we have a system that aggregates and analyzes logs and performance metrics from hundreds
‫and thousands of production servers.
‫As we already know from the previous lecture, that this is a use case for real time processing.
‫We absolutely have to have visibility into those logs and metrics as they arrive at our system.
‫So if something happens, engineers can see those logs and metrics in real time and figure out what
‫happened and how to respond.
‫However, if we deploy a new feature to production that may have some performance impact, we want to
‫be able to look back and compare the current performance we see in real time to historic performance
‫based on the logs and metrics that we collected throughout the week or month.
‫Similarly, if we want to implement automatic anomaly detection, this type of system would have to
‫have access to both real time data and historic data so it can compare the two and find anomalies in
‫the current behavior of our system.
‫Another great example of a system that requires the properties of both batch processing and real time
‫processing is ride sharing service.
‫In this case, the events we are getting are the current location of both the drivers and the clients
‫that need a ride of course, to provide the service.
‫We need to know where each driver and client is located in real time so we can match them together and
‫also show their location on the map.
‫On the other hand, we want to look at historic data and detect patterns such as busy times or days
‫when a lot of clients need a ride.
‫Alternatively, we can detect certain locations that typically have more drivers than clients or vice
‫versa.
‫If we can analyze this data and find those patterns, we can share this information with our drivers
‫and make sure that at any given moment we have enough drivers in each location to provide them with
‫steady income and our riders with good writing experience.
‫So in this case, again, it's not that easy to choose one data processing strategy over the other.
‫For those use cases.
‫We have the Lambda architecture, which takes advantage of both batch and real time processing models
‫to provide us with the best of both worlds.
‫The first person who came up with the Lambda architecture term is Nathan Mars.
‫Based on his experience working on big data at Beck type and Twitter, Lambda Architecture attempts
‫to find a balance between the high fault tolerance and comprehensive analysis of the data that we get
‫from batch processing and the low latency that we get from real time processing.
‫In Lambda architecture, the infrastructure is divided into three layers the batch layer, the speed
‫layer, and the serving layer.
‫Data that comes into our system is dispatched both into the batch layer and the speed layer simultaneously.
‫The batch layer is very similar to what we had before when we talked about batch processing and lambda
‫architecture.
‫The batch layer serves two purposes.
‫The first purpose of the batch layer is to manage our data set and be the system of records.
‫So just like before, the data in our master set is immutable and new data is only appended but is never
‫modified.
‫This data is usually stored on a distributed file system, specifically designed and optimized for storing
‫big files containing massive amounts of data.
‫The second purpose of the batch layer is to pre compute our batch views.
‫Every time we run our batch processing job, it processes all the data that we have in our master data
‫set, which is also a perfect point to make corrections or duplicate the otherwise raw data that we
‫have in our master set.
‫Once the processing job is complete, it indexes and stores the result in a read only database.
‫Typically, this overrides the existing pre computed views that were created the previous time we ran
‫the processing job.
‫It's important to note that the batch layer aims at perfect accuracy and operates on our entire data
‫set.
‫In parallel to the batch layer.
‫The incoming data.
‫Goes to the speed layer, which uses the real time processing strategy that we learned in the previous
‫lecture.
‫All the data goes into a queue or a message broker and is then picked up as it arrives by the processing
‫job.
‫The processing job quickly analyzes the new event and adds the processed record to the real time view
‫so it can be available for anyone to query.
‫The speed layer essentially compensates for the high latency that we have in our batch layer.
‫But unlike the batch layer, it operates only on the most recent data and doesn't attempt to provide
‫a complete view or make any complex data corrections.
‫In other words, the speed layer is there to close the gap that we have between the present moment and
‫the last time our batch job ran and provided a result.
‫Now, the serving layer's purpose is to respond to ad hoc queries and merge the data from both the batch
‫and their real time views.
‫This way, any query can take all the data that we have into account, both historic data that was processed
‫by the batch layer and the most up to date real time data we just received and processed in the speed
‫layer.
‫Now let's explore the Lambda architecture in action using a real life practical example they can from
‫the ADTECH industry.
‫In the edtech industry.
‫On one end, we have advertisers.
‫Those can be companies that sell tickets to concerts or different events and would like to advertise
‫those events to potential customers.
‫Or those companies can be physical or online stores that would like to advertise their existing or upcoming
‫products.
‫On the other end, we have the online content producers such as bloggers or news agencies that provide
‫content on their websites to the general public free of charge.
‫The way those content producers make their money is by leaving empty slots on their websites where advertisers
‫can place their ads.
‫And any time a user either sees or clicks on an ad, the advertiser pays the content producer a small
‫fee.
‫Our system acts as a middleman.
‫On the one hand, our system provides the advertisers with websites that they can advertise on.
‫On the other hand, our system provides content producers with advertisers that are willing to provide
‫ads and pay them money.
‫So when a user opens their favorite blog or news website, this website sends a request to our system
‫requesting an add.
‫Then we find the best and most relevant ad for the user from all the pool of advertisers that we have,
‫and then send that ad back to the website so the user will see both the content and the ad all on one
‫page when the page loads in the user's browser.
‫So now that was the synchronous part of our system.
‫The asynchronous big data processing part kicks in when we start getting events about each ad, add
‫the three types of events we're going to process are as follows.
‫The first type of event happens when the user scrolls through the page and gets to the part where they
‫see the ad on the website.
‫The second type of event happens when a user clicks on a particular ad and is then redirected to the
‫advertiser's website.
‫The third type of event happens when the user makes a purchase on the advertiser's website, which is
‫the ultimate goal of each advertiser.
‫To process this massive stream of those three types of events.
‫We're going to use the Lambda architecture.
‫Each one of those events is going to be processed by both the batch layer and the speed layer, and
‫eventually it will be ready to be queried by the serving layer.
‫The serving layer may get different types of queries, which will require different parts of our big
‫data processing system.
‫Let's see a few concrete examples to understand how Lambda architecture works in practice.
‫For example, an advertiser can send us a query asking about the number of users that are currently
‫viewing their ads across all the different websites.
‫This can easily be satisfied by the data that we have from the speed layer.
‫On the other hand, the advertiser can send us a query asking about how many ads were shown to users
‫in the last 24 hours.
‫The purpose of this query may be to make sure that the advertiser stays within their budget and doesn't
‫spend more money than they intended to.
‫Now, if we assume that our batch layer processes incoming events every 2 hours, for example, this
‫query will have to merge data from both the batch layer and the speed layer, taking the last 2 hours
‫from the speed layer in the previous 22 hours from the batch layer.
‫Also, at some point, the advertiser will want to know the return on their investment and may ask us
‫how much money they need to spend on advertising before a user makes a purchase.
‫Now, it's important to understand that a user may see the same ad on multiple websites sometimes in
‫the course of a month until they finally click on it.
‫And maybe only a few weeks later they decide to make a purchase.
‫So in this case, we would have to rely on the deep analytics and data fusion that we have in the batch
‫layer so we can provide this data to the advertiser.
‫In this lecture, we'll learn about a very important architecture for big data processing, the Lambda
‫Architecture.
‫This architecture is a hybrid between the batch processing model and the real time processing model,
‫which allows us to enjoy the benefits of both worlds for use cases when choosing only one of those strategies
‫is not an option.
‫And finally, we explore the real life practical scenario where Lambda architecture could be used to
‫satisfy a wide range of queries that involve both real time and historic data.