In studying distributed systems, I've come across systems like Spanner, which incurs additional latency for strong consistency, and DynamoDB, which sacrifices strong consistency for low latency in responding to requests. This latency vs consistency tradeoff is one that many systems have to make, including COPS — this lecture's focus — but it offers a unique balance.

mit-6.824 distributed-systems learning-diary

This lecture is about building systems at scale. I'll start this post by describing how a website's architecture might evolve to cope with increasing load, before highlighting Facebook's use of memcached to support the world's largest social network.

mit-6.824 distributed-systems learning-diary

In the first lecture of this series, I wrote about MapReduce as a distributed computation framework. MapReduce partitions the input data across worker nodes, which process data in two stages: map and reduce. While MapReduce was innovative, it was inefficient for iterative and more complex computations. Researchers at UC Berkeley invented Spark to deal with these limitations.

mit-6.824 distributed-systems learning-diary

This lecture on optimistic concurrency control is based on a system called FaRM. FaRM is a main memory distributed computing platform that provides distributed transactions with strict serializability, high performance, durability and high availability by taking advantage of two hardware trends. Here, I explain how FaRM uses these techniques to perform faster and yield far greater throughput than Spanner for simple transactions.

mit-6.824 distributed-systems learning-diary

Spanner is a rare example of a distributed database that supports externally consistent distributed transactions. Many other databases either choose not to implement distributed transactions at all, or opt for weaker consistency models because of the performance cost involved. In this post, we'll learn how Google's TrueTime API enables it to provide this guarantee at a good performance.

mit-6.824 distributed-systems learning-diary

Distributed databases typically divide their tables into partitions spread across different servers which get accessed by many clients. In these databases, client transactions often span the different servers, as the transactions may need to read from various partitions. A distributed transaction is a database transaction which spans multiple servers. This post will detail how databases guarantee some ACID properties when executing distributed transactions.

mit-6.824 distributed-systems learning-diary

The ideal distributed file system would guarantee that all its users have coherent access to a shared set of files and be easily scalable. It would also be fault-tolerant and require minimal human administration. This post will cover how Frangipani approximates this ideal, with a focus on how it provides a consistent view of shared files while maintaining a cache for each user.

mit-6.824 distributed-systems learning-diary

Amazon Aurora is a distributed database service provided by AWS. The paper describes the considerations in building a database for the cloud and details how Aurora's architecture differs from many traditional databases today. This post will explain how traditional databases work and then highlight how Aurora provides great performance through quorum writes and by building a database around the log.

mit-6.824 distributed-systems learning-diary

Many distributed systems today sacrifice stronger consistency guarantees for the sake of greater availability and higher throughput. CRAQ, which stands for Chain Replication with Apportioned Queries, is a system designed to challenge this trade-off. CRAQ's approach differs from existing replication techniques we have seen so far, like in Raft. It improves on the original form of Chain Replication. This post will start by presenting the Chain Replication approach, before describing how CRAQ improves on it.

mit-6.824 distributed-systems learning-diary

Can the coordination of distributed systems be handled by a stand-alone general-purpose service? If so, what should the API of that service look like? In addition, can we improve the performance of a system by N times if we add N times replica servers? This post will focus on answering these questions using the ZooKeeper system as a case study.

mit-6.824 distributed-systems learning-diary