‫Welcome back.
‫Now that we extensively covered the topic of databases, we have all the knowledge on how to store and
‫access structured data effectively.
‫However, there is one more type of data that we haven't discussed yet, which is equally important,
‫and that is unstructured data.
‫So in this lecture, we'll first define what we mean by unstructured data.
‫Then we'll talk about some use cases where unstructured data is used.
‫And finally, we'll talk about two different solutions with very different properties a distributed
‫file system and an object store.
‫So first of all, what is unstructured data?
‫Unstructured data is data that doesn't follow a particular structure, schema or model.
‫If we look at the data we stored in a non relational database, although the structure of each set of
‫documents or key value pairs within the same collection did not have to match.
‫Each record still had a well defined structure.
‫On the other hand, when we talk about binary files like audio, video images, or even PDF documents,
‫without a special tool that can decode the file's content, all we have is a blob which stands for binary
‫large object.
‫Of course, some databases allow storing blobs in them, but typically databases are optimized for structured
‫data and not for unstructured data, and most of them impose strict size limits on storing such binary
‫objects, typically in the order of megabytes, because otherwise those databases would suffer from
‫performance and scalability problems.
‫So before we talk about other solutions for storing unstructured data, let's talk about a few common
‫use cases for storing and accessing such data.
‫The first use case is allowing users to upload data to our system for further processing.
‫We're talking about raw or uncompressed images, video files, audio files, or even documents that
‫users upload to our system.
‫This data can be compressed, trans coded and moved to a different location so it can be either shared,
‫such as in the case of social media or streaming platforms, or used for backup purposes such as in
‫the case of file hosting services.
‫Another use case is database backup and archiving.
‫In this case, we can take periodic snapshots of the state of our relational or non relational database
‫used in our system.
‫Those snapshots are organized in a proprietary database, specific binary formats.
‫We can then treat those snapshots as unstructured data, which can be stored elsewhere for two purposes.
‫One purpose is disaster recovery.
‫In case there is an unfortunate event where we lose the primary or even the replicas of our database,
‫we can always recover our data from that backup.
‫The second purpose is archiving backups of transactions, emails or documents which can later be used
‫for auditing.
‫This archiving is required by law in certain industries like financial institutions or health care.
‫The third common use case is web hosting.
‫Any web content like images, thumbnails, digital downloads or other media we need to display on our
‫website is unstructured data that we need to store somewhere, preferably with the ability to update
‫frequently.
‫And finally, we have data points collected for analytics or machine learning purposes.
‫Those data sets can be huge, and they also can contain binary data, such as sensor measurements taken
‫from Iot devices or images taken by satellites or surveillance cameras.
‫The main feature of those use cases is that those data sets can be very big, which means we need our
‫storage system to be able to scale to terabytes or even petabytes of data.
‫And also every single object, like a video file, audio file or document on its own can be very big,
‫way too big to be stored in any database.
‫So now that we understand the need for unstructured data, let's look at two scalable solutions for
‫storing and accessing such data.
‫The first solution is the file system, or more specifically, a distributed file system.
‫A distributed file system provides us with the same obstruction as if we store the data on our local
‫hard drive, except instead of using a single storage device.
‫We have a network of storage devices connected to each other through the network with different distributed
‫file systems.
‫We can get different replication consistency and auto healing guarantees depending on the type of that
‫file system.
‫But the main feature is that our binary objects are stored in a familiar way as files within folders
‫in a tree like structure.
‫One benefit of storing our unstructured data in a distributed file system is that we don't need any
‫special API to access those files.
‫Another benefit is we can easily modify those files if we need to.
‫For example, if we want to modify a document or append data to the end of a log file or a video file,
‫we can easily do that.
‫The last benefit is that performance intensive operations such as big data analysis or transformations
‫on big data sets are very efficient and fast When we do it directly on the distributed file system.
‫This is particularly useful for machine learning projects or Iot.
‫On the other hand, a distributed file system has limitations that make it suboptimal for certain scenarios.
‫For instance, most distributed file systems are limited in the number of files we can create, which
‫is a big scalability issue, especially if we have many relatively small files like images.
‫Another problem is that we can't easily allow access to those files from a web API, like in the case
‫of serving web content to the browser.
‫And we would have to build additional abstractions on top of that file system to provide that access.
‫So the second storage solution is using an object store.
‫An object store is a scalable storage solution, specifically designed for storing unstructured data
‫at internet scale.
‫An object store can scale linearly like a distributed file system by adding more storage devices.
‫However, unlike a distributed file system, we have virtually no limitation on the number of binary
‫objects we can store in it.
‫Also, object stores usually have a very high limit on the size of a single object, generally an order
‫of magnitude of several terabytes.
‫This makes object stores a perfect solution for database backups and archiving, but by far the best
‫feature of a typical object store is that they expose a very easy HTTP rest API that makes them perfect
‫for storing web content such as images, thumbnails, articles, digital downloads, or anything we
‫can link to on our web page.
‫The last major benefit of using an object store is object versioning.
‫If we wanted to use object versioning in a file system, we would have to use an external version control
‫system.
‫However, when we use an object store, we get object versioning out of the box, which allows us to
‫revert changes and undo delete operations very easily.
‫In contrast to a file system, files are not stored in a directory hierarchy.
‫Instead, they are stored in a flat structure organized in containers commonly called buckets.
‫Generally, those buckets don't have a limit on the number of objects we can store in them.
‫The only limit is our storage budget.
‫The main obstruction and an object store is not surprisingly, an object.
‫An object consists of a unique name or identifier and a value which is its content.
‫In addition, it typically also contains metadata, which is a list of key value pairs that provide
‫additional information about the object, like its size type or format.
‫And each object also contains an access control list for managing permissions on who can read or override
‫that object.
‫Nowadays, pretty much every cloud vendor offers object store services on top of their storage infrastructure
‫at very reasonable costs that are typically broken into tiers or storage classes.
‫Each storage class has different pricing and SLA guarantees.
‫The names of the tiers vary by cloud vendor, but usually at the top tier we have the most expensive
‫option that provides the highest availability, usually around 4/9.
‫It also offers the lowest latency, highest throughput and durability guarantees of above ten nines.
‫This is the best option for frequent access and user facing content like video or images.
‫In the middle tiers, we typically have a range of offerings with lower availability guarantees, typically
‫around 30 nines as well as cheaper pricing.
‫Those options usually have more limited performance and sometimes even have limits on the frequency
‫with which we can access data stored in those storage classes.
‫Those storage classes are perfect for data backups that aren't very frequently needed.
‫Finally, at the lowest tier, we have the optimal storage classes for long term archiving purposes.
‫Those storage classes are cheap and typically used by law firms, health care companies or financial
‫institutions.
‫Those companies use that storage to archive data for many years for legal purposes, but that data is
‫rarely used.
‫Now, in some cases, using a cloud based object store is not an option because of budget, legal or
‫performance constraints.
‫For that, there are quite a few open source and third party managed solutions for running an object
‫store using on premise storage devices.
‫And some of those solutions follow the same API as some cloud vendors, so you can easily use them in
‫a hybrid cloud deployment where part of our system runs in the cloud and the other part runs on our
‫private data center.
‫And we can use the same object store API to store and access data in both locations.
‫Finally, just like in a distributed file system, typically object stores use data replication under
‫the hood.
‫This ensures that losing physical storage will never result in actual loss of data.
‫Now, despite all those benefits of using an object store, it has a few downsides to it that are worth
‫mentioning.
‫The first drawback is objects in an object store are immutable.
‫In other words, unlike in a file system where we can always open a file and modify its content, we
‫can't do it in an object store.
‫The only thing we can do is replace an existing object in place with a new version.
‫This, of course, has performance implications.
‫If we want to store large documents for collaboration purposes and use cases like storing log files
‫where we need to append new data are out of the question completely.
‫Another downside is for creating or reading data to or from an object store.
‫We either need to use a special API or a rest API and we can't access that data as easily as in a regular
‫file system.
‫Finally, distributed file systems are usually the preferred storage option over object stores for very
‫high throughput operations.
‫In this lecture, we'll learn about unstructured data and the issues we have in storing such data in
‫data stores like relational or non relational databases.
‫Later, we talked about some very common use cases for unstructured data, such as storing raw user
‫uploaded data, backup and archiving, web hosting and machine learning.
‫And finally, we'll learn about two solutions for storing such data, where the first one was a distributed
‫file system, which is a more scalable and available version of the local file system we use in our
‫computers.
‫And the second option was an object store.
‫This option may not be as performant as a distributed file system, but it is a much better suitable
‫choice for web content thanks to its HTTP based rest API and ability to store many small and large objects.