‫Welcome back.
‫In the previous lecture
‫we talked about availability in general
‫and in particular the importance of high availability
‫for both users and our business.
‫In this lecture, we're going to focus on what prevents us
‫from getting high availability out of the box
‫and what measures and tactics we can take
‫to improve the availability of our system.
‫So generally there are three sources
‫or categories of failures.
‫The first and most common source
‫of failures are human error,
‫which includes pushing a faulty config to production,
‫running the wrong command or the wrong script,
‫or deploying a new version of our software
‫that hasn't been thoroughly tested.
‫The second category of failures, is software errors.
‫Those include exceptionally long garbage collections
‫and crashes such as out-of-memory exceptions,
‫null pointer exceptions, segmentation faults, and so on.
‫Lastly, we have hardware failures
‫such as servers, routers
‫or storage devices breaking down due
‫to made shelf life,
‫power outages due to natural disasters
‫or network failures because of infrastructure issues
‫or general congestion.
‫Or, how can we deal with all those failures
‫and still achieve high availability?
‫Despite the obvious improvements
‫that we can make to our code review,
‫testing and release processes
‫as well as performing ongoing maintenance to our hardware,
‫inevitably failures will happen.
‫So the best way to achieve high availability in our system
‫is through fault tolerance.
‫Fault tolerance enables our system to remain operational
‫and available to the users despite failures
‫within one or multiple of its components.
‫When those failures happen,
‫a fault tolerance system may continue operating
‫at the same level of performance
‫or a reduced level of performance,
‫but it will prevent the system
‫from becoming entirely unavailable.
‫Generally,
‫fault tolerance revolves around three major tactic,
‫which are failure prevention, failure detection
‫and isolation and recovery.
‫The first thing we can do to prevent our system
‫from going down is to eliminate any single point
‫of failure in our system.
‫A few examples of a single point of failure
‫can be that one server where we are running our application
‫or storing all our data on one instance of a database
‫that runs on a single computer.
‫The best way to eliminate a single point of failure
‫is through replication and redundancy.
‫For example, instead of running our service
‫as a single instance on one server,
‫we can run multiple copies
‫or instances of our application on multiple servers.
‫So if one of those servers goes down, for whatever reason,
‫we can potentially direct all the network traffic
‫to the other servers instead.
‫Similarly, when it comes to databases,
‫instead of storing all the information on one computer
‫we can run multiple replicas
‫of our database containing the same data
‫on multiple computers.
‫So losing any of those replicas will not cause us
‫to lose any data.
‫When it comes to computations,
‫besides the spatial redundancy,
‫which comes in the form of running replicas
‫of our application on different computers,
‫we can also have, what's referred to as time redundancy.
‫Time redundancy essentially means repeating
‫the same operation or the same request multiple times,
‫until we either succeed or give up.
‫Now, when it comes to redundancy and replication
‫we have two strategies, both of which are extensively used
‫in the industry in different systems.
‫The first strategy is called active-active architecture.
‫If we take the database example,
‫it means that requests go to all the replicas,
‫which forces them to all be in sync with each other.
‫So if one of the replicas goes down,
‫then the other replicas
‫can step in and take all the requests immediately.
‫The second strategy is called active-passive architecture,
‫which implies that we have one primary instance or replica
‫that takes all the requests,
‫and the other passive replica
‫or replicas follow the primary replica
‫by taking periodic snapshots of its state.
‫The advantage of the active-active architecture
‫is that essentially we can spread
‫the load among all the replicas,
‫which is identical to horizontal scalability.
‫This allows us to take more traffic
‫and provide better performance.
‫The disadvantage of this approach
‫is that all the replicas are taking requests.
‫So keeping all those active replicas in sync with each other
‫is not trivial
‫and it requires additional coordination and overhead.
‫Similarly,
‫if we take the active-passive architecture approach,
‫then we lose the ability to scale our system,
‫because all the requests still go to one machine.
‫But the implementation of this approach is a lot easier
‫because there is a clear leader
‫that has the most up-to-date data
‫and the rest of the replicas are just followers.
‫The second and tactic
‫for achieving fault tolerance in our system
‫is failure detection and isolation.
‫If we go back to this picture of our application running
‫as multiple instances on different computers,
‫if one of the instances crashes
‫because of software or hardware issues
‫then we need the ability
‫to detect that this instance is faulty
‫and isolated from the rest of the group.
‫This way we can take actions like stopping requests
‫for going to this particular server.
‫Typically, to detect such failures
‫we need a separate system that acts as a monitoring service.
‫The monitoring service
‫can monitor the health of our instances
‫by simply sending it period health check messages.
‫Or alternatively,
‫it can listen to periodic messages called heartbeat
‫that should come periodically
‫from our healthy application instances.
‫In either of those strategies,
‫if the monitoring service does not hear
‫from a particular server for a predefined duration of time
‫it can assume that that server is no longer available.
‫Of course, if the issue was in the network
‫or that host simply had a temporarily on garbage collection
‫the monitoring service is going to have a false positive
‫and assume that a healthy host is faulty.
‫But that's okay,
‫we don't need the perfect monitoring service
‫as long as it doesn't have false negatives.
‫Because false negatives
‫would mean that some of our servers may have crashed
‫but our monitoring system did not detect that.
‫Now, besides the simple exchange of messages
‫in the form of pings and heartbeats,
‫our monitoring system can be more complex
‫and monitor for certain conditions.
‫For example, our monitoring system can collect data
‫about the number of exceptions
‫or errors each host gets per minute.
‫And if the error rate
‫in one of those hosts becomes exceptionally high,
‫our monitoring system can interpret
‫that as an overall failure on that host.
‫Similarly, it can collect information
‫about how long each host takes to respond
‫to incoming requests.
‫And if that time becomes too long the monitoring system
‫can decide that something must be wrong with that server
‫because it became exceptionally slow.
‫Now, let's talk about the third tactic
‫in achieving fault tolerance,
‫which is recovery from a failure.
‫If we recall the second formula of availability
‫that we learned in the previous lecture,
‫we can see that regardless of our system failure rate
‫if we can detect and recover
‫from each failure faster than the user can notice
‫then our system will have high availability.
‫So once we detect and isolate the faulty instance or server
‫we can take several actions.
‫The obvious one is to stop sending any traffic
‫or any workload to that particular host.
‫The second action we can take
‫is to attempt to restart it,
‫with the assumption that after the restart
‫the problem may go away.
‫Another strategy which is commonly used
‫is called a rollback.
‫A rollback essentially means going back to a version
‫that was still considered stable and correct.
‫Rollbacks are very common in databases.
‫If for some reason,
‫we get to a state that violates some condition
‫or the integrity of the data,
‫we simply rolled back to the last state in the past
‫which is considered to be correct.
‫Similarly,
‫if we're rolling out a new version of our software
‫to all the servers, and we detect that all the servers
‫with the new version are getting a lot of errors,
‫we can automatically roll back to the previous version
‫of our software to prevent our system
‫from becoming entirely unavailable.
‫In this lecture, we learned about fault tolerance
‫which is how we achieve high availability
‫in large-scale systems.
‫We first discussed the different sources of failures
‫we can have in our system,
‫which are human error, software crashes,
‫and hardware failures.
‫Later we defined fault tolerance
‫as the ability for our system to remain operational
‫and available to the users despite failures
‫to one or some of its components.
‫And finally, we discussed in detail the three tactics
‫we use to achieve fault tolerance,
‫which are failure prevention,
‫detection and isolation and recovery.
‫I will see you soon in the next lecture.