‫In the previous lectures,
‫we talked extensively about system requirements
‫and their importance.
‫As part of those discussions,
‫we also talked in general about quality attributes,
‫as well as their impact on the software architecture
‫of our system.
‫We also mentioned that the list
‫of possible quality attributes is very long,
‫but not all quality attributes are equally important
‫for every system.
‫So in this and the following lectures,
‫we're going to focus on a small subset
‫of quality attributes that are the most important
‫in any large scale system, regardless of the use case.
‫So in this lecture,
‫we're going to start talking about
‫the first quality attribute, which is performance.
‫We're going to define a few types of performance metrics.
‫And later, we'll talk about important considerations
‫when setting goals, measuring,
‫and analyzing the performance of our system.
‫So let's start talking about the first performance metric,
‫the response time.
‫Frequently, when we talk about performance of a system
‫or an application,
‫we immediately think about speed.
‫Intuitively, when we interact with a system
‫that responds fast to our actions,
‫we associate that system with a high performance.
‫So let's define formally one
‫of the most common performance metrics,
‫the response time,
‫as the time between a client sending a request
‫and receiving a response.
‫The response time is usually broken into two parts.
‫The first and most obvious part is the processing time,
‫which is the time it takes our system to process the request
‫and send the response.
‫This is essentially the amount of time spent in our code,
‫in our databases
‫or other parts of our system actively working
‫on processing the request, applying business logic,
‫and building and sending the response.
‫The second, usually less obvious
‫and frequently forgotten part of the response time,
‫is the waiting time.
‫The waiting time is the duration of time
‫that the request or the response spends inactively
‫in our system.
‫Usually that time is spent in transit in our network wires,
‫switches, and gateways,
‫or in our network or software queues,
‫waiting to be handled or sent to its destination.
‫As a side note, in some textbooks or resources,
‫the waiting time is referred to as latency,
‫while in other resources,
‫the response time and latency are used interchangeably.
‫The reason this term is so overloaded
‫is latency simply means delay.
‫To make sure those two are not confused,
‫it's common to call a response time as end to end latency.
‫The response time is an important performance metric
‫when we're designing a system or a subsystem
‫that is in the critical path of a user interaction.
‫The reason for that is users have very little patience
‫for long response times and get quickly frustrated
‫when there is no immediate feedback to their actions.
‫Many engineers falsely assume that response time
‫is either the only or most important performance metric,
‫and some go as far as thinking that response time
‫is the definition of performance,
‫but the response time
‫is not the ultimate performance metric,
‫and for some systems,
‫it's not as important
‫as the next performance metric we're going to talk about.
‫For example,
‫if we're designing a distributed logging system
‫that aggregates and analyzes
‫a constant stream of logs coming from hundreds
‫and maybe thousands of servers,
‫the most important performance metric
‫is the ability to ingest
‫and analyze large amounts of data at a given period of time.
‫And the more data our system can ingest
‫and analyze per unit of time,
‫the higher the performance of our system is.
‫The term we use for this type of performance metric
‫is throughput.
‫The general definition of throughput
‫is either the amount of work performed by our system
‫per unit of time,
‫in which case the throughput is measured
‫as the number of tasks accomplished per second,
‫or alternatively, depending on the use case,
‫throughput can also be defined
‫as the amount of data processed by our system
‫per unit of time,
‫in which case it's going to be measured in bits, bytes,
‫or megabytes per second.
‫Now let's talk about a few important considerations
‫when we measure and analyze the performance of our system.
‫The first consideration
‫is going to be in regards to measuring the response time
‫as perceived by the client correctly.
‫Very often engineers make the mistake
‫of measuring only the processing time
‫by measuring the time spent in our code
‫and mistaking it for measuring the response time.
‫In such cases,
‫the user might experience very long response times,
‫while our performance graphs are showing
‫that everything is normal.
‫To illustrate the situation,
‫let's consider two requests
‫that come to our system pretty much simultaneously,
‫and let's assume that our system
‫can process only one request at a time.
‫While the first request is being processed by our system,
‫the second request is waiting in our network
‫or software queues,
‫and only once the response for the first request
‫is constructed and sent back to the client,
‫our system proceeds to process the second request
‫and send the response.
‫Now, if we forget about the waiting time
‫and measure only the time spent in our code or our system,
‫then we get the processing time and not the response time,
‫which in this case shows
‫that each request took 10 milliseconds to complete.
‫But from the user perspective,
‫who sees the overall response time,
‫the first request took 10 milliseconds
‫while the second request took 20 milliseconds
‫because of that additional wait time
‫for the first request to be completed.
‫And if we take the average of that,
‫we get 15 milliseconds as the average response time.
‫Now, this may happen not only
‫if we can process only one request at a time,
‫but anytime our system is a bit overloaded
‫with other requests.
‫The second consideration is the response time distribution.
‫Let's say we have hundreds
‫of servers running our application
‫and in each minute we collect the response times
‫from each server,
‫where each response time corresponds to a different request
‫from a different user.
‫Now, ideally, we would like all those samples
‫of response times to be the same,
‫which means all our clients are getting
‫the exact same experience.
‫However, in practice,
‫we will always get a distribution of response times.
‫Some will be very good
‫and some will be surprisingly very bad.
‫So knowing that we will always get such a distribution,
‫what is the metric
‫that we should set our goals around and measure?
‫Should this be the average response time,
‫the median, or maybe the maximum response time?
‫To answer this question,
‫first, we need to learn how to organize
‫and analyze all those samples.
‫The first way to start analyzing such a distribution
‫is to sort the response time samples
‫and bucket them into a histogram.
‫This histogram shows the frequency of each response time,
‫which gives us a better picture of the distribution
‫of our response times.
‫For example, by looking at this histogram,
‫we can see that three requests had a response time
‫of 10 milliseconds,
‫and only one request had a response time
‫of 40 milliseconds.
‫From that histogram,
‫it's easy to create a percentile distribution chart
‫or table.
‫And in that table, the xth percentile
‫is the value below which x percent of the values
‫can be found.
‫For example, by looking at this chart,
‫we see that the 20th percentile is 10 milliseconds,
‫which means that the response time of 20% of the requests
‫was under 10 milliseconds.
‫Equally, we can look at the 50th percentile,
‫which is also called the median,
‫and this tells us that the response time
‫of 50% of the requests was lower than 20 milliseconds.
‫And the 90th percentile tells us that the response time
‫of 90% of the requests was 52 milliseconds at most.
‫If we collect enough samples,
‫we can see that about 20% of our users experience
‫an excessively long response time,
‫which we can see only,
‫thanks to this percentile distribution,
‫while both the median and the average completely hide
‫the story of those 20% of users.
‫The term we use for this section
‫of the percentile distribution graph is the tail latency.
‫More formally, we define the tail latency
‫as the small percentage of response times from our system
‫that take the longest in comparison
‫to the rest of the values.
‫Needless to say,
‫we want this tail latency to be as short as possible.
‫So when we define the response time goals for our system,
‫we need to do it in terms of percentiles and tail latency,
‫and not in terms of average.
‫And later, when we launch our system,
‫we need to measure and validate our response times
‫by analyzing the percentile distribution and comparing them
‫to the goals we set for ourselves ahead of time.
‫So for example, we can set our goal
‫to be that the 95th percentile of our response times
‫would be less than 30 milliseconds,
‫which means that we allow our system
‫to exceed that response time only for 5% of our requests.
‫Alternatively, we can be more aggressive
‫and set our goal to be 30 milliseconds
‫at the 99th percentile.
‫This still leaves our system some room
‫to exceed that number due to unexpected
‫and transient system jitter.
‫Now, the third consideration
‫when it comes to measuring the performance of our system,
‫and this time it refers to both response time
‫and throughput,
‫is identifying the performance degradation point
‫and how fast or how steep this degradation is.
‫The degradation point is that point
‫in our performance graph where the performance
‫is starting to get significantly worse
‫as the load increases.
‫If we see that our system's performance
‫degrades very quickly,
‫it likely means that one or some of our resources
‫are fully utilized.
‫That resource can be the CPU,
‫in which case the CPU utilization
‫of one or some of our servers in our system
‫is very close to 100%.
‫It can be a high memory consumption
‫of one of our processes.
‫It can be that we have too many connections
‫and our operating system
‫or our network simply can't handle so much IO.
‫And it can also mean that one of our software queues
‫is at capacity and just can't keep up
‫with the number of messages coming into our system.
‫In this lecture,
‫we talked about a very important quality attribute,
‫the performance.
‫We learned about the two most common performance metrics,
‫the response time,
‫which is defined as the time
‫between the client sending a request
‫and receiving a response, and the throughput,
‫which is the amount of work or data processed by our system
‫per unit of time.
‫Later, we talked about three important considerations
‫when setting goals and analyzing the performance
‫of our system.
‫The first consideration was the correct measurement
‫of the response time,
‫which takes into account the waiting time
‫in our network and software queues.
‫The second consideration was both reasoning
‫and analyzing the response time
‫in terms of percentile distribution,
‫and not just simple averages.
‫And finally, we talked about performance degradation
‫in response to an increasing load on our system,
‫which would be a perfect segue
‫to our next quality attribute,
‫which we will talk about in the next lecture.
‫See you guys soon.