‫Now let's put everything we've learned so far together and architect a large scale system from the ground
‫up.
‫Throughout the process.
‫We'll practice everything we learned throughout the course.
‫And more importantly, we will boost our confidence in our ability to take on an ambiguous system design
‫problem and carry it all the way to a complete software architecture diagram.
‫To achieve this, we will follow the following process.
‫First, we will ask questions to gather functional, non-functional and system constraints requirements.
‫Then we will define our API.
‫After that, we will create an architecture diagram to fulfill the functional requirements.
‫And finally, we will refine that diagram to address all the non-functional requirements.
‫So before we begin, I want to remind you that this is system design, which means there is no one correct
‫way to solve a problem.
‫And the important thing is to make the correct trade offs depending on the requirements we gather.
‫So without further ado, let's introduce our first design problem.
‫We need to design and architect a highly scalable public discussion forum where people can post questions
‫or news to the public comment on existing posts, upvote or downvote posts or comments, and also get
‫a feed of the most popular posts on their home page.
‫I'm sure you've seen or even used such forums before, so it shouldn't be something new or foreign to
‫you.
‫But designing a system like this from zero, especially one that needs to scale to millions of users
‫worldwide, can be a bit daunting.
‫So let's stay organized and take it step by step, just like we learned throughout the course.
‫The first step would be to ask questions and capture all the functional requirements.
‫For example, can anyone post or view new posts and comments?
‫Or only logged in users?
‫What can a post contain?
‫Is it text?
‫Is it images, videos?
‫And so on.
‫What does it mean to show the most popular posts to the user?
‫How do we define popular?
‫And finally, what is the structure of the comments in a post?
‫Is it a flat list of comments or are they organized in a tree?
‫So let's assume that after asking those questions, we have the following functional requirements.
‫A user should be able to sign up and log in into our system to post, vote or comment.
‫A user should be able to create a new post containing a title, a set of topic tags and a body with
‫text and images that a user uploads to our system.
‫A user should be able to comment on any existing post and the comments should be ordered chronologically
‫as a flat list.
‫We also want to allow users to delete their own post or comment any time.
‫Next, we want to allow a logged in user to upvote or downvote an existing post or comment.
‫And finally, when any user logged in or not navigates to our home page, they should be presented with
‫the top most popular posts in the last 24 hours.
‫And let's define popularity as the total number of upvotes minus the number of downvotes for a post
‫that happened only in the last 24 hours.
‫Now for the non-functional requirements.
‫First and foremost, we have scalability.
‫Our system may start with very low traffic, but can grow to millions of users over time.
‫Also based on different factors such as world events or posts that go viral, we may get sudden traffic
‫spikes that we need to be able to handle.
‫Next, we have performance.
‫Since our clients are end users, we want to ensure that every interaction gets a response within a
‫few hundred milliseconds.
‫After that, we have fault tolerance and high availability.
‫But since it's not a bank or some other mission critical system, let's say that three nines of availability
‫is enough for our purposes.
‫Now we have a trade off to make between consistency and availability.
‫Regarding posts, comments, and votes.
‫Since we don't care so much about every user in the world seeing the most up to date posts, comments,
‫or the number of votes, we will prioritize availability over consistency.
‫This will improve our performance and user experience.
‫Lastly, we need your ability, which means new users posts, comments or votes should never disappear
‫from our system unless they were explicitly deleted.
‫Also, we may have system constraints like certain technologies or cloud vendors that we are bound to
‫or are excluded from in a real system, but we will ignore those in our design.
‫So now that we captured all the functional and non-functional requirements, let's briefly talk about
‫our Systems API.
‫Since this is a very typical web based system called by the client side code that runs on the browser,
‫we will use a rest API over HTTP.
‫So to define our rest API, we will follow the four step process we learned earlier.
‫First, we're going to identify the entities in our system.
‫In this case it will be users, posts, images, comments and votes.
‫Next, let's map those entities to Uris.
‫Taking the entities we gathered from the previous step users and posts are independent collections.
‫But comments and images are a sub resource of a given post and the votes are a sub resource of a post
‫or a comment.
‫In the third step, we need to define the representation of each resource.
‫For the sake of time, let's look at only two resources posts and comments.
‫A post can be represented as a simple JSON object with fields like ID title, the user who posted it
‫an array of topics or tags, its total upwards and downwards and a body that can be a JSON encoded markdown
‫for example.
‫Since it can contain both text and links to images.
‫A collection of comments for a post would simply contain the post ID and an array of comment objects,
‫and each comment would have its ID text.
‫The user who posted it and also its votes.
‫Finally, in the last step, we need to assign HTTP methods to operations on resources.
‫First, let's start with the user resource to sign up a new user.
‫We will of course use the HTTP post method.
‫And to log in.
‫We will also use the post method.
‫If you're wondering why we're using a post method and not a get method to log in a user, the reason
‫is twofold.
‫First of all, we need to send the user's username and password to the server, but if we use the get
‫method, we can't use the HTTP message body, which leaves us with sending the password as part of the
‫URL, which is publicly visible.
‫And of course we don't want that.
‫On the other hand, when we put the credentials inside the request message body, it's automatically
‫encoded if we use HTTPS.
‫The second reason has more to do with the semantics of the login operation.
‫We can think of the login operation as a create session operation or a create authentication token operation
‫where the response can contain a newly created authentication token, which then can be used with every
‫subsequent request without needing to re authenticate.
‫And since the rest create operation translates to a post http method.
‫It's only natural to use the post method for the user login.
‫Now let's move on to creating a post, which would be a post HTTP method.
‫Requesting to view a post or posts would be a get operation and deleting a post would be an HTTP delete
‫method.
‫The same goes for comments.
‫Creating a comment for a post is a post method.
‫Getting a list of comments for a post is a get operation and deleting an existing post is a delete operation.
‫And to upvote or downvote a post or a comment.
‫We will also use the post http method.
‫Finally, we will also use the post http method with the image in the message body to upload an image
‫and we can use the get method to get the actual image given an image URL.
‫Before we move on to the final stage, which is defining the architecture diagram, there are a few
‫important things to notice while designing our API.
‫First of all, if we remember our functional requirements when users navigate to the home page, they
‫should be presented with the most popular posts in the last 24 hours.
‫Now, the number of active posts in our system can be very big in the thousands or hundreds of thousands,
‫and we definitely don't want to load all of them to the user's browser.
‫So we will use API pagination and limit the number of posts, let's say to 20 per page.
‫In other words, when the user navigates to the home page, we will send back only the top 20 posts.
‫And if the user wants the next batch of 20, they will simply click on the appropriate button in the
‫UI, which will send a request for the next batch of posts.
‫The same goes for comments.
‫When a user clicks to view a very popular post, this post may have thousands of comments which can
‫take minutes to load.
‫So in this case we will also similarly use API pagination.
‫Now, on the client side, our front end engineers may hide this pagination behind a feature called
‫Infinite Scrolling, which I'm sure you've seen before.
‫This way, when a user scrolls to the end of the page, the front end code sends a request for the next
‫batch of comments that magically appear on the page.
‫So now before we move on to the exciting stage of defining our system architecture, let's quickly summarize
‫what we achieved so far.
‫After getting the initial vague and ambiguous requirements to build a highly scalable public discussion
‫forum, we ask questions and clarified our system requirements, which should always be the first step
‫in every system design.
‫After getting a very clear picture of our functional requirements, we moved on and defined our desired
‫quality attributes while making a trade off, for example, between consistency and availability.
‫After that, we defined our Rest API by following a step by step process where we identified our entities,
‫map those entities to Uris and define the representation of each of our resources.
‫And finally, and the last step, we define the operations on those resources and map them to HTTP methods.
‫Finally, we made an important observation regarding our API, which came from the strict performance
‫requirement we gathered earlier.
‫This observation was the need to use API pagination to minimize the amount of data we send to the user
‫at a given time.
‫I'll see you guys in the next lecture where we'll continue our design.