chore: remove sd

2026-11-02 21:22:43 +00:00 · 2026-10-13 14:29:10 -04:00
1 parent 4cf976f683
commit bf56e170a4
@@ -48,7 +48,7 @@ flowchart LR
 |---|---|
 | `work/<1\|3\|7>/<Difficulty>/<Category>/<num>.<slug>.{js,py}` | Solutions, e.g. `work/1/Easy/Array/1.two-sum.py`. The top directory is the spaced-repetition bucket: `1` = first solve, `3`/`7` = the blind re-solves of that problem 3 and 7 days later. Machine-parsed header comment (title / `Difficulty:` / URL / `─` rule / statement) — preserve its exact shape |
 | `apps/cli/` | Flat Bun TS workspace: automation entries (shebang + top-level await) and libraries (no shebang, side-effect-free on import). Root `bun run` scripts delegate here |
-| `apps/docs/` | Blume site (`blume.config.ts`, `content/`, `pages/`, `components/` + `components.ts`, `islands/`, `public/`). `content/(0N-<phase>)/<topic>/(problems)/<num>_<slug>.mdx` plus one `meta.ts` per phase and per topic folder, all generated by sync (a phase's once, then hand-owned): the phase folder is a `(group)` (sidebar section, no route segment), the topic folder is the README campaign topic slug (`tree-dfs/`, `heap-priority-queue/`) and a real route segment, and `(problems)` is a second `(group)` inside it, so `tree-dfs/(problems)/226_invert-binary-tree.mdx` routes to `/tree-dfs/invert-binary-tree` (Blume sorts by the `<num>_` prefix and strips it from the route). The `meta.ts` files carry the exact labels, phase/topic numbers and icons so the sidebar reads in curriculum order; every topic meta is `display: "page"`, so a topic is one sidebar row whose panel holds its landing page and a "Problems" section. A problem page's badge row (between the description and the body) is rendered, not written: the MDX `wrapper` (`components/ProblemBadges.astro`, registered as `mdx.wrapper` in `components.ts` — Blume's page template renders the title and description itself with no slot before the body, so the wrapper, which MDX hands no route, reads it off `Astro.url`) reads the page's `topics:` (LeetCode's topic tags, from sync) and `video:` (NeetCode YouTube id, from `bun run videos`) frontmatter — both declared via `frontmatter.extend` in `blume.config.ts`, hand-rolled Standard Schemas, no zod — and links a tag the campaign teaches to its topic landing page (filled), any other tag to its card on the Appendix glossary `/appendix#<slug>` (outlined; LeetCode's problem list if the tag has no term, gray and iconless), and the video as a red `Explanation` badge. Every badge in the row is a pill. A tag badge carries the icon its target shows (a landing's: the topic `meta.ts` icon read off `data.navigation`; an Appendix term's: its `icon` in `appendix.ts`) and the tag's own colour from `lib/tag-colors.ts` — unique per tag site-wide, assigned greedily over every tag with a landing or a term so tags that share problem pages sit furthest apart in OKLab, in a lightness band that keeps ≥ 3:1 contrast in both themes (Blume's Badge takes one hex for both). The reverse direction is the `PageFooter` slot (`components/BacklinksList.astro`): an Obsidian-style `<num> <name>` list under every topic landing page of the problems whose badges link there, plus a topic's own folder members (so a topic no tag names, `hybrid-structures`, still lists its problems), in a native `<details>` open on load whose top-right chevron collapses it (no JS); problem pages are recognised by their `<num>. <name>` title and render no list. The sidebar is the `Sidebar` slot (`components/SourceSidebar.astro`, also the mobile drawer's nav): a Fumadocs-style content-source selector (a `<details>` on Blume's `data-blume-dropdown` light dismiss), then Blume's own `NavTree` scoped to the active source. The sources are ONE table, `apps/docs/sources.ts` (`SOURCES`: `{ label, description, icon, path }`; the active source is the deepest `path` the route sits under; the `/` source — the LeetCode campaign — owns every node no other source claims, any other shows its folder's children as the whole tree), so a new source (system design: `content/system-design/`) is a content folder with an `index.mdx` plus one row there. The custom pages `/progress`, `/changelog` and `/appendix` are header `navigation.tabs` (on mobile, Blume lists them at the top of the drawer); never `navigation.featured`, which Blume renders above the slot, over the selector. Which tags the campaign teaches is ONE map, `apps/docs/topic-landings.ts` (tag slug → landing routes), and where a badge points is ONE function beside it, `tagTarget()` — read by both components and the Appendix page (so badges, backlinks and term cards cannot disagree) and the map by sync. Every other tag a problem carries is ONE hand-written table, `apps/docs/appendix.ts` (`APPENDIX`: slug → `{ name, icon, category, definition }`), rendered by ONE custom page, `pages/appendix.astro` — a glossary with an A–Z letter index and a filter box (an `<appendix-filter>` custom element over static markup; no JS, the full list), one card per term (icon tinted the tag's colour, category, definition, LeetCode list link, and a `<details>` "Solved" list of the problem pages whose badge `tagTarget()` sends there, paged `PAGE_SIZE` (5) rows at a time by numbered buttons under it — a `<solved-pages>` custom element; without JS every row shows and the buttons stay hidden), each card `id`'d by slug and highlighted as `:target`. It is no content folder, so it enters neither the sidebar tree nor prev/next: its header tab is its link. Sync writes nothing for it — it warns about a carried tag with no term, a term whose tag a topic now teaches (delete it), and a term `icon` that is a `TOPIC_ICONS` value or another terLine truncated
+| `apps/docs/` | Blume site (`blume.config.ts`, `content/`, `pages/`, `components/` + `components.ts`, `islands/`, `public/`). `content/(0N-<phase>)/<topic>/(problems)/<num>_<slug>.mdx` plus one `meta.ts` per phase and per topic folder, all generated by sync (a phase's once, then hand-owned): the phase folder is a `(group)` (sidebar section, no route segment), the topic folder is the README campaign topic slug (`tree-dfs/`, `heap-priority-queue/`) and a real route segment, and `(problems)` is a second `(group)` inside it, so `tree-dfs/(problems)/226_invert-binary-tree.mdx` routes to `/tree-dfs/invert-binary-tree` (Blume sorts by the `<num>_` prefix and strips it from the route). The `meta.ts` files carry the exact labels, phase/topic numbers and icons so the sidebar reads in curriculum order; every topic meta is `display: "page"`, so a topic is one sidebar row whose panel holds its landing page and a "Problems" section. A problem page's badge row (between the description and the body) is rendered, not written: the MDX `wrapper` (`components/ProblemBadges.astro`, registered as `mdx.wrapper` in `components.ts` — Blume's page template renders the title and description itself with no slot before the body, so the wrapper, which MDX hands no route, reads it off `Astro.url`) reads the page's `topics:` (LeetCode's topic tags, from sync) and `video:` (NeetCode YouTube id, from `bun run videos`) frontmatter — both declared via `frontmatter.extend` in `blume.config.ts`, hand-rolled Standard Schemas, no zod — and links a tag the campaign teaches to its topic landing page (filled), any other tag to its card on the Appendix glossary `/appendix#<slug>` (outlined; LeetCode's problem list if the tag has no term, gray and iconless), and the video as a red `Explanation` badge. Every badge in the row is a pill. A tag badge carries the icon its target shows (a landing's: the topic `meta.ts` icon read off `data.navigation`; an Appendix term's: its `icon` in `appendix.ts`) and the tag's own colour from `lib/tag-colors.ts` — unique per tag site-wide, assigned greedily over every tag with a landing or a term so tags that share problem pages sit furthest apart in OKLab, in a lightness band that keeps ≥ 3:1 contrast in both themes (Blume's Badge takes one hex for both). The reverse direction is the `PageFooter` slot (`components/BacklinksList.astro`): an Obsidian-style `<num> <name>` list under every topic landing page of the problems whose badges link there, plus a topic's own folder members (so a topic no tag names, `hybrid-structures`, still lists its problems), in a native `<details>` open on load whose top-right chevron collapses it (no JS); problem pages are recognised by their `<num>. <name>` title and render no list. The sidebar is Blume's default nav tree (no `Sidebar` slot override). The custom pages `/progress`, `/changelog` and `/appendix` are header `navigation.tabs` (on mobile, Blume lists them at the top of the drawer); never `navigation.featured`, which Blume renders above the nav tree. Which tags the campaign teaches is ONE map, `apps/docs/topic-landings.ts` (tag slug → landing routes), and where a badge points is ONE function beside it, `tagTarget()` — read by both components and the Appendix page (so badges, backlinks and term cards cannot disagree) and the map by sync. Every other tag a problem carries is ONE hand-written table, `apps/docs/appendix.ts` (`APPENDIX`: slug → `{ name, icon, category, definition }`), rendered by ONE custom page, `pages/appendix.astro` — a glossary with an A–Z letter index and a filter box (an `<appendix-filter>` custom element over static markup; no JS, the full list), one card per term (icon tinted the tag's colour, category, definition, LeetCode list link, and a `<details>` "Solved" list of the problem pages whose badge `tagTarget()` sends there, paged `PAGE_SIZE` (5) rows at a time by numbered buttons under it — a `<solved-pages>` custom element; without JS every row shows and the buttons stay hidden), each card `id`'d by slug and highlighted as `:target`. It is no content folder, so it enters neither the sidebar tree nor prev/next: its header tab is its link. Sync writes nothing for it — it warns about a carried tag with no term, a term whose tag a topic now teaches (delete it), and a term `icon` that is a `TOPIC_ICONS` value or another term's. The former per-term pages `/appendix/<slug>` are exact redirects to `/appendix#<slug>` in `blume.config.ts`, generated from `APPENDIX`'s keys. A hand-written `(0N-<phase>)/<topic>/index.mdx` (`hidden: true`) is that topic's landing page: Blume ≥ 1.7.3 links the sidebar group row to it (`sliding-window/index.mdx` → `/sliding-window`) and the hidden flag drops the duplicate row. Two kinds of custom MDX component, both import-free on every page: `islands/<Name>.tsx` is hydrated React (auto-discovered; `DependencySpine` — the README topic map as data, zoom/pan, nodes link to whatever route Blume's `/api/docs/navigaLine truncated
 | `apps/api/` | Cloudflare Worker workspace: `src/` modules, `data/schedule.json` (the plan: `date → { topics | problems | review | gate | sweep }`, human-edited, bundled at deploy), `migrations/` (append-only; `0002` is the +3/+7 ladder rebuild) |
 | `.github/workflows/` | deploy (docs → Pages with `GITHUB_TOKEN` for the releases-backed changelog, Worker → Cloudflare with the `OPENROUTER_API_KEY` Actions secret put into it first, on every main push), close-solved (issues + `/admin/solved` + Project `Set`/`Difficulty`, needs `SRS_ADMIN_KEY` and `PROJECT_PAT`; then the `release` job: timever tag + release per finished topic, `contents: write`), close-topics, sync-d1 (issue edits → `/admin/reconcile`), spaced-repetition (daily 06:00 ET cron → today's review issue; `GITHUB_TOKEN` only, no secrets) |
 | `.githooks/` | Tracked git hooks; `bun install` points `core.hooksPath` here (root `prepare` script, no husky). `pre-commit` runs `bun run sync` and stages only what sync changed (pages, `apps/docs/redirects.ts`), so a solution, its page and its `/<num>` short link land in one commit; it refuses to run while `work/1` has anything unstaged (sync reads the tree, so the page would not match the commit), and leaves pre-existing dirt under `apps/docs/content` (in-progress `## Approach` prose) unstaged. `git commit --no-verify` for a deliberate partial commit |
@@ -16,8 +16,5 @@ export default defineComponents({
  layout: {
    // Topic landing pages' "what links here": every problem pointing at the page.
    PageFooter: "./components/BacklinksList.astro",
    // Content-source selector + pinned links above the nav tree, scoped to the
    // active source (sources.ts).
    Sidebar: "./components/SourceSidebar.astro",
  },
 });
@@ -1,90 +0,0 @@
 ---
 /**
 * The `Sidebar` slot (also the mobile drawer's nav, which defaults to it): a
 * Fumadocs-style content-source selector, then Blume's own nav tree scoped to
 * the active source (`sources.ts`).
 *
 * The selector is a `<details>` with Blume's `data-blume-dropdown` light
 * dismiss (outside press, Escape, focus leaving), like its own NavSelector.
 */
 import Icon from "blume/components/Icon.astro";
 import { withMountedBase } from "blume/components/islands/base-path.ts";
 import NavTree from "blume/components/layout/NavTree.astro";
 import type { UIStrings } from "blume/core/i18n-ui.ts";
 import { isUnderPath } from "blume/core/navigation.ts";
 import type { NavNode } from "blume/core/types.ts";
 import { SOURCES } from "../sources.ts";
 interface Props {
  items: NavNode[];
  currentRoute: string;
  strings?: UIStrings["nav"];
  ids?: Map<NavNode, string>;
  fragmentBase?: string;
 }
 const { items, currentRoute, ...tree } = Astro.props;
 const active =
  SOURCES.filter((s) => isUnderPath(currentRoute, s.path)).sort(
    (a, b) => b.path.length - a.path.length,
  )[0] ?? SOURCES[0]!;
 const claims = (node: NavNode, path: string) => {
  const at = node.kind === "page" ? node.route : node.path;
  return at !== undefined && isUnderPath(at, path);
 };
 // The root source keeps every node no other source claims; any other source
 // shows its folder's children, as if that folder were the content root.
 const scoped =
  active.path === "/"
    ? items.filter(
        (node) => !SOURCES.some((s) => s.path !== "/" && claims(node, s.path)),
      )
    : items.flatMap((node) =>
        node.kind === "group" && node.path === active.path ? node.children : [],
      );
 ---
 <details class="group relative mb-3" data-blume-dropdown>
  <summary
    aria-label={`Content source: ${active.label}`}
    class="flex w-full cursor-pointer list-none items-center gap-2.5 rounded-xl border border-border bg-muted/50 p-1.5 pe-2.5 text-foreground text-sm transition-colors hover:bg-muted group-open:bg-muted [&::-webkit-details-marker]:hidden"
  >
    <span class="flex size-7 shrink-0 items-center justify-center rounded-lg border border-border bg-background">
      <Icon name={active.icon} size={15} />
    </span>
    <span class="min-w-0 flex-1 truncate font-medium">{active.label}</span>
    <Icon class="shrink-0 text-muted-foreground" name="chevrons-up-down" size={14} />
  </summary>
  <div
    class="absolute inset-x-0 z-50 mt-1.5 rounded-xl border border-border bg-background p-1 shadow-xl"
    data-blume-dropdown-panel
  >
    {
      SOURCES.map((source) => (
        <a
          aria-current={source === active ? "true" : undefined}
          class="flex items-start gap-2.5 rounded-lg p-2 text-muted-foreground text-sm transition-colors hover:bg-muted hover:text-foreground aria-[current=true]:text-foreground"
          href={withMountedBase(source.path)}
        >
          <Icon class="mt-0.5 shrink-0" name={source.icon} size={16} />
          <span class="min-w-0 flex-1">
            <span class="block font-medium">{source.label}</span>
            <span class="block text-muted-foreground text-xs">{source.description}</span>
          </span>
          {source === active && <Icon class="mt-0.5 shrink-0" name="check" size={14} />}
        </a>
      ))
    }
  </div>
 </details>
 <NavTree {...tree} currentRoute={currentRoute} items={scoped} />
 <script>
  import { installDropdownDismiss } from "blume/components/dropdown-dismiss.ts";
  installDropdownDismiss();
 </script>
@@ -1,265 +0,0 @@
 ---
 title: "Scale from Zero to Millions of Users"
 description: "Grow a single server into a multi-data-center system: load balancers, replication, caching, CDNs, a stateless web tier, queues and sharding."
 icon: trending-up
 sidebar:
  label: "Scaling"
 ---
 ## Introduction
 Scaling a system to support millions of users is a complex, iterative journey requiring refinement and optimization. This chapter outlines how to begin with a single server setup and scale the architecture step by step to handle millions of users.
 ---
 ## Single Server Setup
 Initially, all components (web app, database, cache) run on a single server.
 <Frame caption="Single server">
 ![Single server](./images/scaling/single-server.png)
 </Frame>
 ### Request Flow
 1. Users access the application via domain names (e.g., `api.mysite.com`), resolved to IP addresses using DNS.
 2. IP address of the web-server is returned to the browser or mobile app.
 3. HTTP requests are sent to the web server, which returns HTML or JSON responses.
 ### Traffic Sources
 1. **Web Applications:** Use server-side languages (e.g., Python, Java) for business logic and client-side languages (e.g., JavaScript, HTML) for presentation.
 2. **Mobile Applications:** Communicate with the web server using HTTP and JSON for lightweight data exchange.
 ---
 ## Database Separation
 As the user base grows, the database is moved to a dedicated server to allow independent scaling of web and database tiers.
 <Frame caption="Database">
 ![Database](./images/scaling/database.png)
 </Frame>
 ### Database Choices
 1. **Relational Databases (SQL):** Structured data stored in tables. Examples: MySQL, PostgreSQL.
 2. **Non-Relational Databases (NoSQL):** Suitable for unstructured data or low-latency requirements. Categories include:
   - Key-Value Stores
   - Graph Databases
   - Column Stores
   - Document Stores
 - Non-relational databases might be the right choice if:
   - application requires super-low latency.
   - data is unstructured, or  there is no relational data.
   - only need to serialize and deserialize data (JSON, XML, YAML, etc.).
   - need to store a massive amount of data.
 ---
 ## Vertical vs Horizontal Scaling
 ### Vertical Scaling
 - Adds more resources (CPU, RAM) to existing servers.
 - Limited by hardware constraints and lacks redundancy.
 ### Horizontal Scaling
 - Adds more servers to the pool, making it more suitable for large-scale systems.
 - A load balancer is used to handle the request routing between the servers.
 ---
 ## Load Balancer
 <Frame caption="Load balancer">
 ![Load balancer](./images/scaling/load-balancer.png)
 </Frame>
 A **load balancer** distributes traffic among multiple servers. Benefits include:
 1. Redundancy: If a server goes offline, traffic is rerouted.
   -  If server 1 goes offline, all the traffic will be routed to server 2.
 2. Scalability: Easily add servers to handle traffic spikes.
   -  If the website traffic grows rapidly, subsequent servers can be added to handle the additional traffic.
 ---
 ## Database Replication
 <Frame caption="Database replication">
 ![Database replication](./images/scaling/database-replication.png)
 </Frame>
 ### Master-Slave Model
 - **Master Database:** Handles write operations.
   - All the data-modifying commands like insert, delete, or update must be sent to the master database.
 - **Slave Databases:** Handle read operations, improving performance and reliability.
   - Since the ratio of reads to writes is higher in most applications; thus, the number of slave
 databases in a system is usually larger than the number of master databases.
 ### Benefits
 1. Improved performance through parallel read operations.
 2. High availability and data reliability through redundancy.
 ### Failure Handling
 - If only one slave database is available and it goes offline, read operations will be directed
 to the master database temporarily.
 - In case multiple slave databases are available, read operations are
 redirected to other healthy slave databases and a new server will replace the old one.
 -  If the master database goes offline, a slave database will be promoted to be the new
 master.
 - In production system the chosen slave database might not be up to date, hence data needs to be updated by running data
 recovery scripts (methods like multi-masters and circular replication could help).
 ---
 ## Caching
 A **cache** stores frequently accessed data in memory to reduce database load. The cache tier is a temporary data store layer, much faster than the database.
 <Frame caption="Cache">
 ![Cache](./images/scaling/cache.png)
 </Frame>
 ### Caching considerations
 1. **Use case**: Consider using cache when data is read frequently but modified infrequently.
 2. **Expiration Policies:** Once cached data is expired, it is removed from the cache. When there is no expiration policy, cached
 data will be stored in the memory permanently.
 3. **Consistency:** This means keeping the data store and the cache in sync. Inconsistency
 can happen because data-modifying operations on the data store and cache are not in a single transaction.
 4. **Mitigating failures**: A single cache server represents a potential single point of failure, multiple
 cache servers across different data centers are recommended to avoid SPOF.
 5. **Eviction Policies:**: Once the cache is full, items need to be evicted to free up memory. LRU is the most popular cache eviction policy.
 ---
 ## Content Delivery Network (CDN)
 A **CDN** improves load times by caching static content (images, CSS, JavaScript) on geographically distributed servers.
 <Frame caption="Cdn">
 ![Cdn](./images/scaling/cdn.png)
 </Frame>
 ### Workflow
 1. User requests content from the nearest CDN server.
 2. If unavailable, content is fetched from the origin server and cached.
 ### CDN considerations
 1. **Cost:** CDNs are run by third-party providers which charge for data transfers in and out of the CDN.
 2. **Cache Expiry:** The cache expiry time should neither be too long nor too short.
 3. **CDN fallback:** If there is a temporary CDN outage, clients should be able to detect the problem
 and request resources from the origin.
 4. **Invalidating files:** If files are updated the cache should be invalidated to point to the updated files.
 ---
 ## Stateless Web Tier
 By moving session data to a shared datastore, web servers become stateless. This allows:
 1. Easier horizontal scaling.
 2. Auto-scaling based on traffic.
 <Frame caption="Stateless">
 ![Stateless](./images/scaling/stateless.png)
 </Frame>
 ---
 ## Multi-Data Center Setup
 Deploying across multiple data centers improves availability and reduces latency. Strategies include:
 <Frame caption="Data center">
 ![Data center](./images/scaling/data-center.png)
 </Frame>
 1. **GeoDNS Routing:** Direct users to the nearest data center.
 2. **Data Replication:** Synchronize data across centers to prevent inconsistencies.
 ### Key considerations
 - **Traffic redirection:** Effective tools are needed to direct traffic to the correct data center.
 - **Data synchronization:** A common strategy is to replicate data across multiple data centers.
 - **Test and deployment:**  Automated deployment tools are vital to keep services consistent through all the data centers.
 ---
 ## Message Queue
 A **message queue** is a durable component, stored in memory, that supports asynchronous
 communication. It serves as a buffer and distributes asynchronous requests.
 <Frame caption="Message queue">
 ![Message queue](./images/scaling/message-queue.png)
 </Frame>
 - Input services, called producers/publishers, create messages, and publish them to a message queue.
 - Other services called consumers/subscribers, connect to the queue, and perform actions defined by the messages.
 ---
 ## Logging, Metrics, and Automation
 <Frame caption="Logging">
 ![Logging](./images/scaling/logging.png)
 </Frame>
 ### Importance
 1. **Logging:** Tracks errors and system health.
 2. **Metrics:** Provides insights into performance and user activity.
 3. **Automation:** Streamlines testing, deployment, and scaling.
 ---
 ## Database Scaling
 ### Vertical Scaling
 - Adds hardware resources but has physical and cost limitations.
 - Has multiple drawbacks:
   -  Greater risk of single point of failures.
   -  Overall cost of vertical scaling is high
 ### Horizontal Scaling (Sharding)
 <Frame caption="Horizontal scaling">
 ![Horizontal scaling](./images/scaling/horizontal-scaling.png)
 </Frame>
 - Divides data across multiple shards using keys (e.g., `user_id`).
   - Sharding separates large databases into smaller, more easily managed parts called shards.
   - Each shard shares the same schema, though the actual data on each shard is unique to the shard.
 -  Sharding key is critical when implementing a sharding strategy. When choosing a sharding key it is important to choose a key that can evenly distribute data.
 #### Challenges
 1. **Resharding data:** Resharding data is needed when:
   - Single shard could no longer hold more data due to rapid growth.
   - Certain shards might experience shard exhaustion faster than others due to uneven data distribution.
   - Consistent Hashing is used to overcome these problems
 2. **Celebrity problem:**  Excessive access to a specific shard could cause server overload.
   - To solve this problem, we may need to allocate a shard for each celebrity.
 3. **Join and de-normalization:** Once a database has been sharded across multiple servers, it is hard to perform join operations across database shards.
   -  A common workaround is to de-normalize the database so that queries can be performed in a single table.
 ---
 ## Conclusion
 ### Key Takeaways
 1. Keep the web tier stateless.
 2. Build redundancy at every tier.
 3. Use caching and CDNs to optimize performance.
 4. Scale the data tier with sharding.
 5. Decouple components for flexibility.
 This chapter provides a solid foundation for building scalable systems that can handle millions of users.
@@ -1,109 +0,0 @@
 ---
 title: "Back-of-the-Envelope Estimation"
 description: "Powers of two, latency numbers and availability figures, and how to estimate QPS and storage quickly in an interview."
 icon: calculator
 sidebar:
  label: "Estimation"
 ---
 ## Introduction
 Back-of-the-envelope estimation is a crucial skill in system design interviews. It involves making quick, rough calculations to assess system capacity or performance. According to Jeff Dean, Google Senior Fellow, these estimates help evaluate whether designs meet requirements through thought experiments and common performance benchmarks.
 This chapter covers key concepts, methodologies, and examples to build proficiency in scalability and estimation.
 ---
 ## Key Concepts
 ### Power of Two
 Understanding data volume in terms of powers of two is fundamental:
 <Frame caption="Powers of two">
 ![Powers of two](./images/back-of-the-envelope-estimation/power-of-two.png)
 </Frame>
 This knowledge helps in performing accurate storage and bandwidth calculations.
 ---
 ### Latency Numbers Every Programmer Should Know
 Latency numbers represent the time taken for various operations in computing systems. These provide insights into relative performance:
 | Operation                | Latency (2020) |
 |--------------------------|----------------|
 | L1 Cache Access          | 0.5 ns         |
 | L2 Cache Access          | 7 ns           |
 | Main Memory Access       | 100 ns         |
 | SSD Random Read          | 150 µs         |
 | HDD Random Seek          | 10 ms          |
 | Round-Trip in Data Center| 500 µs         |
 | Inter-Region Data Center | 150 ms         |
 :::tip[Key insights]
 - Memory is fast, disk is slow.
 - Avoid disk seeks whenever possible.
 - Compress data before transmitting over the internet to save bandwidth.
 :::
 ---
 ### Availability Numbers
 High availability (HA) ensures minimal downtime. Availability is expressed in **nines**:
 | Availability | Downtime per year |
 | --- | --- |
 | 99% (two nines) | ~3.65 days |
 | 99.9% (three nines) | ~8.8 hours |
 | 99.99% (four nines) | ~52 minutes |
 | 99.999% (five nines) | ~5.3 minutes |
 | 99.9999% (six nines) | ~31.56 seconds |
 Cloud providers like Amazon, Google, and Microsoft aim for SLAs (Service Level Agreements) of **99.9% or higher**.
 ---
 ## Example Estimation - Twitter QPS and Storage Requirements
 ### Assumptions
 - **300 million monthly active users (MAU).**
 - **50% daily active users (DAU).**
 - **Average tweets/user/day:** 2.
 - **10% of tweets contain media.**
 - **Data retention:** 5 years.
 ### Estimations
 1. **Query Per Second (QPS):**
   - DAU: $$300\text{M} \times 50\% = 150\text{M}$$
   - Tweets QPS: $$150\text{M} \times 2 \text{ tweets} / 24 \text{ h} / 3600 \text{ s} \approx 3500$$
   - Peak QPS: $$2 \times 3500 \approx 7000$$
 2. **Media Storage:**
   - **Tweet Size Components:**
     - `tweet_id`: 64 bytes
     - `text`: 140 bytes
     - `media`: 1 MB
   - **Daily Media Storage:** $$150\text{M} \times 2 \times 10\% \times 1\text{ MB} = 30\text{ TB/day}$$
   - **5-Year Storage:** $$30\text{ TB} \times 365 \times 5 \approx 55\text{ PB}$$
 ---
 ## Tips for Effective Estimation
 ### 1. Rounding and Approximation
 Precision is not critical; focus on the process. Simplify complex calculations using round numbers. For example:
 - $$99987 / 9.1$$ can be approximated as $$100{,}000 / 10 = 10{,}000$$.
 ### 2. Write Down Assumptions
 Document assumptions clearly for future reference.
 ### 3. Label Units
 Avoid ambiguity by labeling units (e.g., `5 MB` instead of `5`).
 ### 4. Common Estimation Scenarios
 - **QPS (Queries Per Second):** Measure traffic intensity.
 - **Peak QPS:** Account for traffic spikes.
 - **Storage Requirements:** Estimate total data needs.
 - **Cache Requirements:** Evaluate memory requirements for caching.
 - **Number of Servers:** Calculate hardware needs based on workload.
@@ -1,125 +0,0 @@
 ---
 title: "A Framework for System Design Interviews"
 description: "A four-step process for the interview: scope the problem, propose a high-level design, dive deep, and wrap up."
 icon: list-checks
 sidebar:
  label: "Interview Framework"
 ---
 ## Introduction
 System design interviews are a key part of the hiring process, simulating real-life problem-solving scenarios. These interviews evaluate not just technical skills but also collaboration, communication, and the ability to handle ambiguous requirements.
 This chapter introduces a **4-step framework** for navigating system design interviews effectively.
 ---
 ## Step 1: Understand the Problem and Establish Design Scope
 ### Key Objectives
 - Clarify requirements and assumptions.
 - Avoid jumping into solutions prematurely.
 - Showcase critical thinking by asking good questions.
 ### Approach
 - **Ask Clarifying Questions:**
  - What are the most important features?
  - What scale does the system need to handle?
  - Are we building for web, mobile, or both?
  - Are there existing technologies or constraints?
 - **Document Assumptions:** Write assumptions on a whiteboard or paper for reference.
 ### Example
 **Problem:** Design a news feed system.
 **Questions:**
 - Is it a mobile app, web app, or both?
 - How many friends can a user have?
 - Should the feed include images and videos?
 - Is the feed sorted by reverse chronological order?
 ---
 ## Step 2: Propose High-Level Design and Get Buy-In
 ### Key Objectives
 - Develop a high-level architecture.
 - Collaborate with the interviewer to refine the design.
 ### Approach
 - **Draft a Blueprint:**
  - Use box diagrams for key components (e.g., clients, APIs, databases, caches, CDNs).
  - Treat the interviewer as a teammate to refine the design.
 - **Perform Back-of-the-Envelope Calculations:**
  - Ensure the design can handle the scale constraints.
 - **Walk Through Use Cases:** Identify edge cases and validate design assumptions.
 ### Example
 For a news feed system, divide the design into:
 1. **Feed Publishing Flow:** Writing posts into databases and populating friends' feeds.
 2. **Feed Retrieval Flow:** Aggregating and displaying friends' posts in reverse chronological order.
 ---
 ## Step 3: Design Deep Dive
 ### Key Objectives
 - Dive into critical components.
 - Showcase depth of understanding and adaptability.
 ### Approach
 - **Prioritize Key Components:** Focus on areas most relevant to the problem.
 - **Discuss Bottlenecks:** Identify potential performance issues and propose solutions.
 - **Balance Detail:** Avoid over-engineering or unnecessary deep dives.
 ### Example Topics
 - **URL Shortener:** Focus on hash function design.
 - **Chat System:** Explore latency reduction and online/offline status handling.
 - **News Feed System:** Examine feed publishing and retrieval processes.
 ---
 ## Step 4: Wrap-Up
 ### Key Objectives
 - Highlight areas for improvement.
 - Recap the design and discuss follow-ups.
 ### Approach
 - **Identify Bottlenecks:** Discuss potential limitations and scaling strategies.
 - **Summarize Design:** Recap major design decisions and trade-offs.
 - **Propose Enhancements:**
  - How to scale from 1 million to 10 million users.
  - Error handling for server failures or network issues.
 ---
 ## Best Practices
 :::tip[Do]
 - **Ask Questions:** Clarify ambiguities before diving into solutions.
 - **Communicate:** Share your thought process with the interviewer.
 - **Iterate with the Interviewer:** Treat them as a collaborator.
 - **Show Flexibility:** Suggest alternative approaches and refine your design.
 - **Focus on Critical Components:** Prioritize key parts of the system.
 :::
 :::warning[Don't]
 - **Avoid Premature Solutions:** Don’t design before understanding the requirements.
 - **Don’t Go Silent:** Communicate regularly during the process.
 - **Avoid Over-Engineering:** Focus on practical, scalable solutions.
 :::
 ---
 ## Time Management
 Suggested split of a 45-minute interview:
 | Step | Time |
 | --- | --- |
 | 1. Understand the problem and scope | 3–10 minutes |
 | 2. High-level design and buy-in | 10–15 minutes |
 | 3. Deep dive | 10–25 minutes |
 | 4. Wrap-up | 3–5 minutes |
@@ -1,3 +0,0 @@
 import { defineMeta } from "blume";
 export default defineMeta({ title: "Foundations", order: 1 });
@@ -1,154 +0,0 @@
 ---
 title: "Design a Rate Limiter"
 description: "Throttle client traffic with token bucket, leaking bucket, fixed window and sliding window algorithms, and run them across many servers."
 icon: gauge
 sidebar:
  label: "Rate Limiter"
 ---
 ## Introduction
 This chapter explores the design and implementation of a rate limiter—a system component used to control traffic rates sent by clients or services. Rate limiters are crucial for preventing abuse, reducing costs, and ensuring the stability of server resources. Examples of their use include limiting posts, account creations, and reward claims.
 ## Benefits of Rate Limiting
 - **Preventing DoS Attacks:** Blocking excess calls to avoid resource starvation.
 - **Cost Reduction:** Limiting unnecessary requests to reduce server expenses.
 - **Preventing Overloads:** Filtering out excessive requests to stabilize server performance.
 ## Step 1: Understanding the Problem
 ### Key Features
 - Server-side API rate limiter.
 - Support for multiple throttle rules.
 - Handle large-scale systems in distributed environments.
 - Option for a standalone service or application-level code.
 - Inform users when throttled.
 ### Requirements
 - Accurate request throttling.
 - Minimal latency.
 - Low memory usage.
 - Distributed capability.
 - Clear exception handling.
 - High fault tolerance.
 ## Step 2: High-Level Design
 ### Placement Options
 <Frame caption="Rate Limiting Middleware Architecture">
 ![Rate Limiting Middleware Architecture](./images/rate-limiter/rate_limiter_architecture.png)
 </Frame>
 1. **Client-Side Implementation:** Unreliable due to potential misuse.
 2. **Server-Side Implementation:** Preferred for control and reliability.
 3. **Middleware (API Gateway):** A flexible option for integrated rate limiting.
 ### Guidelines for Placement
 - Evaluate current tech stack and choose efficient options.
 - Select appropriate algorithms based on business needs.
 - Use an API gateway if microservices are employed.
 - Opt for commercial solutions if resources are limited.
 ## Step 3: Rate Limiting Algorithms
 ### 1. Token Bucket
 <Frame caption="Token Bucket Algorithm">
 ![Token Bucket Algorithm](./images/rate-limiter/token-bucket.png)
 </Frame>
 - **Description:** Tokens are added to a bucket at a fixed rate; each request consumes a token.
 - **Parameters:** Bucket size and refill rate.
 - **Pros:** Easy to implement, memory-efficient, supports traffic bursts.
 - **Cons:** Requires careful parameter tuning.
 ### 2. Leaking Bucket
 <Frame caption="Leaking Bucket Algorithm">
 ![Leaking Bucket Algorithm](./images/rate-limiter/leaking-bucket.png)
 </Frame>
 - **Description:** Processes requests at a fixed rate using a FIFO queue.
 - **Pros:** Memory-efficient, stable outflow rate.
 - **Cons:** Traffic bursts may delay recent requests.
  Example: https://github.com/uber-go/ratelimit
 ### 3. Fixed Window Counter
 <Frame caption="Fixed Window Counter">
 ![Fixed Window Counter](./images/rate-limiter/fixed-window-counter.png)
 </Frame>
 - **Description:** Divides time into fixed intervals and uses counters to limit requests.
 - **Pros:** Simple, efficient for specific use cases.
 - **Cons:** Traffic spikes at window edges can exceed limits.
 - Sudden burst of traffic at the edges of time windows
 could cause more requests than allowed quota to go through.
 <Frame caption="Fixed Window Issue">
 ![Fixed Window Issue](./images/rate-limiter/fixed-window-issue.png)
 </Frame>
 ### 4. Sliding Window Log
 <Frame caption="Sliding Window Log">
 ![Sliding Window Log](./images/rate-limiter/sliding-window-log.png)
 </Frame>
 - **Description:** Tracks timestamps to allow a rolling time window.
 - **Pros:** Accurate rate limiting.
 - **Cons:** High memory consumption.
 ### 5. Sliding Window Counter
 <Frame caption="Fixed Window Counter">
 ![Fixed Window Counter](./images/rate-limiter/sliding-window-counter.png)
 </Frame>
 - **Description:** Combines fixed window and sliding log methods for smoothing spikes.
 - **Pros:** Memory-efficient, handles traffic bursts.
 - **Cons:** Approximation may not be perfectly strict.
 ## High-Level Architecture
 <Frame caption="Architecture">
 ![Architecture](./images/rate-limiter/architecture.png)
 </Frame>
 - **Data Storage:** Use in-memory caching (e.g., Redis) for fast counter operations.
 - **Steps:**
  1. Client sends request to middleware.
  2. Middleware checks counters in Redis.
  3. Request is processed or rejected based on limits.
 ## Advanced Considerations
 ### Distributed Environments
 - **Challenges:** Race conditions, synchronization issues.
 - **Solutions:** Use locks, Lua scripts, or sorted sets in Redis. Employ centralized data stores for synchronization.
 ### Performance Optimizations
 - Multi-data center setups for reduced latency.
 - Eventual consistency models for synchronization.
 ### Monitoring
 - Regular analytics to ensure algorithm effectiveness and adjust rules as needed.
 ## Further reading
 - [Circuit Breaker Algorithm](https://martinfowler.com/bliki/CircuitBreaker.html)
 - [Uber Rate Limiter](https://github.com/uber-go/ratelimit/blob/master/ratelimit.go)
@@ -1,137 +0,0 @@
 ---
 title: "Design Consistent Hashing"
 description: "Spread keys across servers on a hash ring so that adding or removing a server moves only a small share of the keys."
 icon: orbit
 sidebar:
  label: "Consistent Hashing"
 ---
 ## Introduction
 This chapter explores consistent hashing, a technique essential for achieving horizontal scaling by efficiently distributing requests and data across servers. It minimizes data redistribution when servers are added or removed and ensures an even distribution of data to mitigate issues like server hotspots.
 ## The Rehashing Problem
 ### Explanation
 In traditional hashing methods, such as `serverIndex = hash(key) % N`, data redistribution becomes problematic when the number of servers changes. For example:
 - Removing a server causes most keys to be reassigned, leading to cache misses.
 - Adding a server results in unnecessary key redistributions.
  <Frame caption="Server hashing">
  ![Server hashing](./images/consistent-hashing/server-hashing.png)
  </Frame>
 - This approach works well when the size of the server pool is fixed. However, problems arise when new servers are added, or existing servers are removed.
  <Frame caption="Server hashing Miss">
  ![Server hashing Miss](./images/consistent-hashing/server-hashing-miss.png)
  </Frame>
 ### Key Issue
 Redistribution of most keys when server count changes causes inefficiency and overload.
 ## Consistent Hashing
 ### Definition
 Consistent hashing ensures that only a fraction of keys are remapped when servers are added or removed. This minimizes disruptions and enhances scalability.
 ### Key Concepts
 1. **Hash Space and Ring:** The hash space forms a continuous ring, with hash values distributed from `0` to `2^160-1` (e.g., using hash function like SHA-1). By connecting both ends we get a ring.
   <Frame caption="Hash Ring">
   ![Hash Ring](./images/consistent-hashing/hash-ring.png)
   </Frame>
 - Using the same hash function f, we map servers based on server IP or name onto the ring.
  <Frame caption="Server Ring">
  ![Server Ring](./images/consistent-hashing/server-ring.png)
  </Frame>
 1. **Server Lookup**
 - A key's server is determined by traversing clockwise on the ring until a server is found.
  <Frame caption="Server Lookup">
  ![Server Lookup](./images/consistent-hashing/server-lookup.png)
  </Frame>
 2. **Adding and Removing Servers**
 - Adding a server redistributes only nearby keys. Only a fraction of keys are redistributed to the new server.
  <Frame caption="Adding Server">
  ![Adding Server](./images/consistent-hashing/adding-server.png)
  </Frame>
 - Removing a server affects only the keys in its range. Only keys from the removed server are reassigned to the next server clockwise.
  <Frame caption="Removing Server">
  ![Removing Server](./images/consistent-hashing/removing-server.png)
  </Frame>
 ## Challenges and Solutions
 ### Two Issues in Basic Approach
 1. **Uneven Partition Sizes:** Servers may have unequal data partitions.
 2. **Non-uniform Key Distribution:** Some servers may receive significantly more keys than others.
 ### Solution: Virtual Nodes
 - Each server is represented by multiple virtual nodes on the ring uniformly distrubuted on the ring.
 - Virtual nodes improve key distribution and balance load. As the number of virtual nodes increases, the distribution of keys       becomes more balanced. This is because the standard deviation gets smaller with more virtual nodes, leading to balanced data distribution.
  <Frame caption="Virtual Nodes">
  ![Virtual Nodes](./images/consistent-hashing/virtual-nodes.png)
  </Frame>
 ## Affected Keys
 When servers are added or removed:
 - **Added Server:** Affected keys are those between the new server and its predecessor.
  In the following example server 4 is added onto the ring. The affected range starts from s4 (newly
  added node) and moves anticlockwise around the ring until a server is found (s3). Thus, keys
  located between s3 and s4 need to be redistributed to s4.
  <Frame caption="Server Addition">
  ![Server Addition](./images/consistent-hashing/server-addition.png)
  </Frame>
 - **Removed Server:** Affected keys are those between the removed server and its predecessor. In the following example when a server (s1) is removed, the affected range starts from s1
 (removed node) and moves anticlockwise around the ring until a server is found (s0). Thus, keys located between s0 and s1 must be redistributed to s2.
 <Frame caption="Server Removed">
 ![Server Removed](./images/consistent-hashing/server-removed.png)
 </Frame>
 ## Benefits of Consistent Hashing
 - **Minimized Redistribution:** Only a fraction of keys are reassigned.
 - **Scalability:** Enables horizontal scaling.
 - **Mitigates Hotspots:** Balances data distribution to avoid server overload.
 ## Real-World Applications
 - Amazon Dynamo DB
 - Apache Cassandra
 - Discord
 - Akamai CDN
 - Maglev Load Balancer
 ## Further reading
 - [Consistent Hashing](https://tom-e-white.com/2007/11/consistent-hashing.html)
 - [CS168: Introduction and Consistent Hashing](http://theory.stanford.edu/~tim/s16/l/l1.pdf)
 - [Apache Cassandra](http://www.cs.cornell.edu/Projects/ladis2009/papers/Lakshman-ladis2009.PDF)
 - [Scaling Discord](https://blog.discord.com/scaling-elixir-f9b8e1e7c29b)
 - [Google Maglev](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf)
@@ -1,314 +0,0 @@
 ---
 title: "Design a Key-Value Store"
 description: "A distributed key-value store built on partitioning, replication, quorum consensus, vector clocks and failure handling."
 icon: database
 sidebar:
  label: "Key-Value Store"
 ---
 ## Introduction
 A **key-value store** is a type of non-relational database where data is stored as key-value pairs. Each key is unique, and values are accessed using these keys. This chapter details how to design a scalable, high-availability distributed key-value store that supports operations like:
 - `put(key, value)` for inserting data.
 - `get(key)` for retrieving data.
 ### Characteristics of the Design
 - Small key-value pairs (&lt;10 KB).
 - Supports big data with high availability and scalability.
 - Automatic scaling and tunable consistency.
 - Low latency.
 ---
 ## Single Server Key-Value Store
 ### Implementation
 - Use a **hash table** to store key-value pairs in memory.
 - Optimizations:
  - Data compression.
  - Storing less frequently accessed data on disk.
 ### Limitation
 A single server's memory is limited, requiring a **distributed approach** for scalability.
 ---
 ## Distributed Key-Value Store
 A **distributed key-value store** partitions data across multiple servers and must address trade-offs outlined by the **CAP theorem**.
 ### CAP Theorem
 1. **Consistency:** All clients see the same data simultaneously.
 2. **Availability:** The system responds to every request, even if some nodes are down.
 3. **Partition Tolerance:** The system continues to operate despite network partitions.
 **Trade-off:** According to CAP theorem only two of the three guarantees can be achieved.
 <Frame caption="CAP">
 ![CAP](./images/key-value-store/cap.png)
 </Frame>
 #### System Types:
 - **CP Systems:** Consistency and partition tolerance while sacrificing availability (e.g., banking systems).
 - **AP Systems:** Availability and partition tolerance while sacrificing consistency (e.g., eventual consistency).
 - **CA Systems:** Consistency and Availability while sacrificing partition tolerance.
    **Since network failure is unavoidable, a distributed system must tolerate network partition. Thus, a CA system cannot exist in real-world applications.**
    In a distributed system, partitions are inevitable. When a partition occurs, we must choose between consistency and availability. For example, if node n3 goes down,
    any data written to nodes n1 or n2 cannot be propagated to n3. Conversely, if data is written to n3 but not yet propagated to n1 and n2, nodes n1 and n2 will have stale data.
  <Frame caption="Server down">
  ![Server down](./images/key-value-store/server-down.png)
  </Frame>
 - If we choose CP system, we must block all write operations to n1 and n2 to avoid data inconsistency.
 - If we choose AP system, the system keeps accepting reads, even though it might return stale data.
 For writes, n1 and n2 keep accepting writes,
 and data will be synced to n3 when the network partition is resolved.
 ---
 ## System Components
 ### 1. Data Partitioning
 - **Technique:** Consistent Hashing is used to distribute data across multiple servers evenly.
 - **Advantages:**
  - Automatic scaling with server addition/removal.
  - Heterogeneity through virtual nodes. The number of virtual nodes for a server is proportional to the server capacity.
 ### 2. Data Replication
 - Replicate data across `N` servers for high availability.
 - The N servers are chosen by walking clockwise from the server position and choose the first N servers on the ring to store data copies.Place replicas in distinct data centers to improve reliability in case of virtual nodes.
  <Frame caption="Data replication">
  ![Data replication](./images/key-value-store/data-replication.png)
  </Frame>
 ### 3. Consistency
 Since data is replicated at multiple nodes, it must be synchronized across replicas.
 - **Quorum Consensus:**
  - `N`: Total replicas.
  - `W`: Write quorum size. For a write to be considered successful, write must be acknowledged from W replicas.
  - `R`: Read quorum size. For a read to be considered as successful, read must wait for responses from at least R replicas.
  - **Rule:** `W + R > N` ensures strong consistency.
  - The configuration of W, R and N is a typical tradeoff between latency and consistency.
    <Frame caption="Quorum consensus">
    ![Quorum consensus](./images/key-value-store/quorum-consensus.png)
    </Frame>
    - If R = 1 and W = N, the system is optimized for a fast read.
    - If W = 1 and R = N, the system is optimized for fast write.
    - If W + R > N, strong consistency is guaranteed (Usually N = 3, W = R = 2).
    - If W + R &lt;= N, strong consistency is not guaranteed.
 - **Models**:
  - **Strong Consistency:** A read operation returns a value corresponding to the result of the most updated write data item.
  - **Weak Consistency:** Subsequent read operations may not see the most updated value.
  - **Eventual Consistency:** Given enough time, all updates are propagated, and all replicas are consisten
 ### 4. Inconsistency Resolution
 Replication gives high availability but causes inconsistencies among replicas. Versioning and
 vector locks are used to solve inconsistency problems.
 - **Versioning:**
    - Use **vector clocks** to track data versions and resolve conflicts.
    - Versioning means treating each data modification as a new immutable version of data.
      <Frame caption="Consisten hashing">
      ![Consisten hashing](./images/key-value-store/consistent-server.png)
      </Frame>
      <Frame caption="Inconsistent server">
      ![Inconsistent server](./images/key-value-store/inconsistent-server.png)
      </Frame>
    - Server 1 changes the name , and server 2 also changes the name. These two changes are performed simultaneously. Now, we have conflicting values, called versions v1 and v2.
 - **Vector Clock**
    1. **Setup**: A vector clock is a [server, version] pair associated with a data item. It can be used to check
        if one version precedes, succeeds, or in conflict with others.
        - Assume a vector clock represented by D([S1, v1], [S2, v2], …, [Sn, vn]), If data item D is written to server
        Si, the system must perform one of the following tasks.
        - Where: `D` is the data item.`Si` is the server identifier.`vi` is the version counter for the data at server `Si`.
    2. **Updating the Vector Clock:**  When a data item is modified at a server:
        - If the server exists in the vector clock, its version counter is incremented.
        - Otherwise, a new entry is added to the vector clock.
    3. **Conflict Detection:**
        - **No Conflict:** A version X is an ancestor of version Y if all counters in X are less than or equal to those in Y.
        - **Conflict Exists:** Two versions are siblings if there is at least one counter in Y that is less than its counterpart in X.
    4. **Conflict Resolution:** When conflicts are detected (sibling versions), the system relies on application-specific logic or client intervention to   reconcile the data.
       <Frame caption="Server hashing">
       ![Server hashing](./images/key-value-store/vector-clock.png)
       </Frame>
 - **Challenges:**
  - Increased complexity for clients.
  - Vector clock size may grow with many updates, requiring trimming strategies to limit its size.
 ### 5. Handling Failures
 #### a. Failure Detection
 It is insufficient to believe that a server is down because another server says so.Usually, it requires at least two independent sources of information to mark a server down.
 - **Gossip Protocol:**
  <Frame caption="Gossip protocol">
  ![Gossip protocol](./images/key-value-store/gossip-protocol.png)
  </Frame>
    - Each node maintains member IDs and heartbeat counters.
    - Each node periodically increments its heartbeat counter.
    - Each node periodically sends heartbeats to a set of random nodes.
    - If the heartbeat has not increased for more than predefined periods, the member is
    considered as offline
 #### b. Temporary Failures
 - **Sloppy Quorum:** Use healthy nodes to maintain operations temporarily.
  <Frame caption="Sloppy Quorum">
  ![Sloppy Quorum](./images/key-value-store/sloppy-quorum.png)
  </Frame>
    - After detecting failures, the system needs to deploy certain mechanisms to ensure availability
    - Instead of enforcing the quorum requirement, the system chooses the first W healthy servers for writes and first R
    healthy servers for reads on the hash ring.
    - Offline servers are ignored. If a server is unavailable, another server will process requests temporarily
 - **Hinted Handoff:** Offline servers catch up with changes upon recovery.
    - When the down server is up, changes will be pushed back to achieve data consistency
 #### c. Permanent Failures
 - Use **Merkle Trees** for efficient synchronization between replicas.
    A **Merkle Tree** (or hash tree) is a data structure to efficiently detect and resolve inconsistencies between replicas during permanent failures.
 - Working
    1. **Structure:**
        - **Leaf Nodes** store the hash of individual data blocks.
        - **Non-Leaf Nodes** store the hash of their child nodes.
        - The **root hash** represents the combined state of all data in the tree.
    2. **Building a Merkle Tree:**
        - **Step 1:** Divide the key space into buckets.
          <Frame caption="Key Bucket">
          ![Key Bucket](./images/key-value-store/key-bucket.png)
          </Frame>
        - **Step 2:** Hash each key in a bucket using uniform hashing.
          <Frame caption="Hash Key Bucket">
          ![Hash Key Bucket](./images/key-value-store/hash-key-bucket.png)
          </Frame>
        - **Step 3:** Create a single hash for each bucket.
          <Frame caption="Hash Bucket">
          ![Hash Bucket](./images/key-value-store/hash-bucket.png)
          </Frame>
        - **Step 4:** Combine hashes of buckets to compute higher-level hashes, culminating in the root hash.
          <Frame caption="Merkel Tree">
          ![Merkel Tree](./images/key-value-store/merkel-tree.png)
          </Frame>
    3. **Synchronization:**
        - To synchronize two replicas:
            - Compare their root hashes.
            - If the root hashes match, the replicas are consistent.
            - If the root hashes differ, compare child hashes recursively to identify inconsistent buckets.
        - Only the inconsistent data is synchronized.
 - Advantages
    - **Efficiency:** Only inconsistent data is synchronized, reducing data transfer.
    - **Scalability:** Effective for large datasets with minimal synchronization overhead.
    - **Reliability:** Ensures data consistency across replicas.
 ### 6. Handling Data Center Outages
 - Replicate data across multiple data centers to ensure availability during outages.
 ---
 ## Write and Read Paths
 ### 1. Write Path (Based on Cassandra architecture)
 <Frame caption="Hash Bucket">
 ![Hash Bucket](./images/key-value-store/write-path.png)
 </Frame>
 - Persist the write in a **commit log**.
 - Save data to a **memory cache**.
 - Flush data to **SSTable** (Sorted String Table) on disk when cache is full.
 ### 2. Read Path
 <Frame caption="Hash Bucket">
 ![Hash Bucket](./images/key-value-store/read-path.png)
 </Frame>
 <Frame caption="Hash Bucket">
 ![Hash Bucket](./images/key-value-store/read-path-without-cache.png)
 </Frame>
 - Check **memory cache** for the data.
 - If absent, use a **Bloom Filter** to locate the data in SSTables.
 - Retrieve and return the data.
 ---
 ## Final Architecture
 <Frame caption="Hash Bucket">
 ![Hash Bucket](./images/key-value-store/final-architecture.png)
 </Frame>
 -  Clients communicate with the key-value store through simple APIs: get(key) and put(key,
 value).
 - A coordinator is a node that acts as a proxy between the client and the key-value store.
 - Nodes are distributed on a ring using consistent hashing.
 - The system is completely decentralized so adding and moving nodes can be automatic.
 - Data is replicated at multiple nodes.
 - There is no single point of failure as every node has the same set of responsibilities.
 ## Further reading
 - [Amazon Dynamo](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)
 - [Cassandra Architecture](https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/architecture/archIntro.html)
 - [Google BigTable Architecture](https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf)
 - [Amazon Dynamo DB Internals](https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html)
 - [Design Patterns in Amazon Dynamo DB](https://www.youtube.com/watch?v=HaEPXoXVf2k)
 - [Internals of Amazon Dynamo DB](https://www.youtube.com/watch?v=yvBR71D0nAQ)
@@ -1,117 +0,0 @@
 ---
 title: "Design a Unique ID Generator in Distributed Systems"
 description: "Generate unique, sortable 64-bit IDs across servers: multi-master replication, UUIDs, a ticket server and Twitter Snowflake."
 icon: fingerprint
 sidebar:
  label: "Unique ID Generator"
 ---
 ## Introduction
 This chapter addresses the challenge of designing a **unique ID generator** for distributed systems. Traditional auto-increment keys are unsuitable in distributed environments due to scalability and synchronization challenges. The focus is on creating unique, sortable, 64-bit numerical IDs that meet the following requirements:
 - IDs must be **unique** and **ordered by date**.
 - IDs must fit within **64 bits**.
 - The system should generate **over 10,000 IDs per second**.
 ---
 ## Step 1: Understanding the Problem
 ### Basic Requirements
 - IDs must be unique and numerical and should fit in 64 bits.
 - IDs increment with time but not strictly by `+1`.
 - IDs should be sortable by date.
 - System must handle high throughput (10,000 IDs/sec).
 ---
 ## Step 2: High-Level Design Options
 ### 1. Multi-Master Replication
 - **Approach:** Use database `auto_increment` with step increments (e.g., `+k` for k servers).
  <Frame caption="Multi Master">
  ![Multi Master](./images/unique-id-generator/multi-master.png)
  </Frame>
 - **Drawbacks:**
  - Hard to scale across data centers.
  - IDs do not always increase with time.
  - Scaling issues when servers are added/removed.
 ### 2. UUID (Universally Unique Identifier)
 - **Approach:**
    - Generate 128-bit unique identifiers independently on each server using UUID.
    - UUIDs can be generated independently without coordination between servers
      <Frame caption="UUID generator">
      ![UUID generator](./images/unique-id-generator/uuid.png)
      </Frame>
 - **Advantages:**
  - No coordination needed between servers.
  - Scales easily with web servers.
 - **Drawbacks:**
  - Exceeds 64-bit requirement.
  - IDs are not sortable by time and may be non-numeric.
 ### 3. Ticket Server
 - **Approach:** Use a centralized database server to increment and assign IDs.
  <Frame caption="UUID generator">
  ![UUID generator](./images/unique-id-generator/ticket-server.png)
  </Frame>
 - **Advantages:**
  - Simple to implement for small-scale systems.
  - Generates numeric IDs.
 - **Drawbacks:**
  - Single point of failure.
  - Synchronization challenges in multi-server setups.
 ### 4. Twitter Snowflake Approach
 - **Approach:**
  <Frame caption="Snowflake approach">
  ![Snowflake approach](./images/unique-id-generator/twitter-snowflake.png)
  </Frame>
  <Frame caption="Snowflake ID breakdow">
  ![Snowflake ID breakdow](./images/unique-id-generator/snowflake-id-breakdown.png)
  </Frame>
    - Divide IDs into sections to ensure uniqueness and scalability.
    - **Sign Bit (1 bit):** Always `0`, potentially distinguishing signed and unsigned numbers.
    - **Timestamp (41 bits):** Milliseconds since a custom epoch (Twitter's default is `1288834974657`, equivalent to Nov 04, 2010, 01:42:54 UTC). Ensures IDs are time-ordered.
    - **Datacenter ID (5 bits):** Identifies up to `2^5 = 32` datacenters.
    - **Machine ID (5 bits):** Identifies up to `2^5 = 32` machines within each datacenter.
    - **Sequence Number (12 bits):** Tracks IDs generated on a machine within the same millisecond, supporting up to `2^12 = 4096` IDs per millisecond. The sequence resets to `0` every millisecond.
 - **Advantages:**
    - **Scalability:** Handles 10,000+ IDs per second across multiple servers.
    - **Time-Order:** Ensures IDs are sortable by time.
    - **Decentralization:** No single point of failure.
 ## Step 4: Additional Considerations
 ### 1. Clock Synchronization
 - **Challenge:** ID generation assumes synchronized clocks across servers.
 - **Solution:** Use **Network Time Protocol (NTP)** to minimize drift.
 ### 2. Section Length Tuning
 - Adjust section sizes (e.g., fewer sequence bits, more timestamp bits) based on use case.
 ### 3. High Availability
 - ID generators are mission-critical and must be fault-tolerant.
 - Consider redundancy and failover mechanisms.
 ## Further reading
 - [Ticket Servers: Distributed Unique Primary Keys on the Cheap](https://code.flickr.net/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap)
 - [Snowflake](https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake.html)
@@ -1,163 +0,0 @@
 ---
 title: "Design a URL Shortener"
 description: "Shorten and redirect URLs: the API, hash-plus-collision versus base-62 conversion, and the redirect flow."
 icon: link
 sidebar:
  label: "URL Shortener"
 ---
 ## Introduction
 This chapter discusses the design of a URL shortening service like TinyURL. The system's main goals include **URL shortening**, **redirecting**, and **high scalability** to handle large traffic volumes.
 ### Requirements
 - Shortened URLs must be **unique** and as **short as possible**.
 - Handle **100 million URL generations per day** with a 10-year support capacity.
 - Support **efficient read operations** with a 10:1 read-to-write ratio.
 - Store 365 billion records, requiring approximately **365 TB** of storage over 10 years.
 ---
 ## Step 1: High-Level Design
 ### API Endpoints
 1. **URL Shortening:**
   - Endpoint: `POST api/v1/data/shorten`
   - Parameters: `{longUrl: longURLString}`
   - Returns: `shortURL`
 2. **URL Redirecting:**
   - Endpoint: `GET api/v1/shortUrl`
   - Returns: `longURL` for redirection.
     <Frame caption="URL Redirection">
     ![URL Redirection](./images/url-shortener/url-redirection.png)
     </Frame>
 ### URL Redirection
 - **301 Redirect:**  A 301 redirect shows that the requested URL is “permanently” moved to the long URL. The browser caches the response, and
 subsequent requests for the same URL will not be sent to the URL shortening service.
 - **302 Redirect:** Temporary; useful for analytics like tracking clicks.
 ### URL Shortening
 <Frame caption="URL Shortening">
 ![URL Shortening](./images/url-shortener/url-shortening.png)
 </Frame>
 - Use a **hash function** to generate a short URL, mapping long URLs to unique shortened versions.
 - The hash function must satisfy the following requirements:
    - Each longURL must be hashed to one hashValue.
    - Each hashValue can be mapped back to the longURL.
 ---
 ## Step 2: Deep Dive into Design
 ### Data Model
 Store `<shortURL, longURL>` mappings in a relational database to optimize memory usage. The table schema includes:
 - `id` (primary key),
 - `shortURL`,
 - `longURL`.
  <Frame caption="Table Schema">
  ![Table Schema](./images/url-shortener/table-schema.png)
  </Frame>
 ### Hash Function
 #### 1. Base 62 Conversion:
 - Encodes numbers using characters `[0-9, a-z, A-Z]`, providing **62 possible characters**.
 - Base conversion is another approach commonly used for URL shorteners.
 - A unique id can be assigned to the short url and ID can be base 62 converted to get the short URL.
 - A 7-character hash supports up to **3.5 trillion unique URLs**, enough for 365 billion URLs.
 **Example:**
 Convert ID `2009215674938` to Base 62:
 - `2009215674938` → `zn9edcu`.
 #### 2. Hash + Collision Resolution:
 - Use hash functions like CRC32, MD5, or SHA-1.
  <Frame caption="Hash Function">
  ![Hash Function](./images/url-shortener/hash-function.png)
  </Frame>
 - One approach is to collect the first 7 characters of a hash value; however, this method can lead to hash collisions.
 - To resolve collisions,recursively append a new predefined string until no more collision but this can be expensive.
 - Resolve collisions with **Bloom Filters** for efficient lookup.
  <Frame caption="URL Lookup">
  ![URL Lookup](./images/url-shortener/url-lookup.png)
  </Frame>
 ### Comparison
 -  **Hash + Collision Resolution:**
    - Fixed short URL length
    - Does not need a unique ID generator
    - Collision is possbile and needs resolution
    - Not possible to find the next available short URL because it does not depend on ID
 - **Base 62 Conversion**
    - The length is not fixed and goes up with ID
    - It needs a unique ID generator
    - Collision is not possbile
    - Easy to find the next short URL if ID increments by 1 (Can be a security concern)
 ---
 ### URL Shortening Flow
 <Frame caption="URL Shortening">
 ![URL Shortening](./images/url-shortener/url-shortening-flow.png)
 </Frame>
 1. Check if `longURL` exists in the database.
 2. If found, return the existing `shortURL`.
 3. Otherwise:
   - Generate a unique ID using a **distributed ID generator**.
   - Convert the ID to `shortURL` using Base 62.
   - Store the `<id, shortURL, longURL>` mapping in the database.
 ---
 ### URL Redirecting Flow
 <Frame caption="URL Shortening">
 ![URL Shortening](./images/url-shortener/url-redirecting-flow.png)
 </Frame>
 1. User clicks a `shortURL`.
 2. Query `<shortURL, longURL>` mapping:
   - Check the **cache** first for faster access.
   - If not in the cache, query the database.
 3. Redirect the user to `longURL`.
 ---
 ## Additional Considerations
 ### Rate Limiter
 - Prevent abuse by setting limits on requests per IP.
 ### Scalability
 1. **Web Tier:** Stateless, scalable by adding/removing web servers.
 2. **Database Tier:** Use replication and sharding.
 ### Analytics
 - Collect data like click rates, source, and timestamps for business insights.
 ### High Availability and Reliability
 - Ensure consistent and reliable services using database replication and fault-tolerant design.
@@ -1,174 +0,0 @@
 ---
 title: "Design a Web Crawler"
 description: "Crawl billions of pages politely: the URL frontier, deduplication, robots.txt, and extensibility."
 icon: bug
 sidebar:
  label: "Web Crawler"
 ---
 ## Introduction
 A **web crawler**, also known as a spider or robot, is used to discover and collect web content, such as web pages, images, and videos. This chapter focuses on designing a scalable web crawler for **search engine indexing**.
 ### Applications of Web Crawlers
 1. **Search Engine Indexing:** Collect web pages to create searchable indexes (e.g., Googlebot).
 2. **Web Archiving:** Preserve web data for future use (e.g., US Library of Congress).
 3. **Web Mining:** Extract knowledge from web data (e.g., financial analysis of shareholder reports).
 4. **Web Monitoring:** Detect copyright or trademark infringements.
 ### Design Challenges
 A good web crawler must address:
 - **Scalability:** Handle billions of pages using parallelization.
 - **Robustness:** Manage bad HTML, crashes, and malicious links.
 - **Politeness:** Avoid overwhelming servers with too many requests.
 - **Extensibility:** Support new content types with minimal changes.
 ---
 ## Step 1: Understanding the Problem
 ### Requirements
 1. Crawl **1 billion web pages per month** (400 pages/second, peak 800 QPS).
 2. Collect **HTML-only content**.
 3. Track new and updated pages.
 4. Ignore duplicate content.
 5. Store crawled data for **5 years**, requiring ~30 PB of storage.
 ---
 ## Step 2: High-Level Design
 ### Components
 <Frame caption="Web Crawler Architecture">
 ![Web Crawler Architecture](./images/web-crawler/web-crawler-architecture.png)
 </Frame>
 1. **Seed URLs:** Starting points for the crawler.
    - Need to selective as a good starting point that a crawler can utilize to traverse as many links as possible.
    - Can be based on locality based on different popular website or based on topics.
    - Strategies: Categorize by locality or topic (e.g., sports, healthcare).
 2. **URL Frontier:** Stores URLs to be downloaded.
   - Implemented as a **FIFO queue**.
 3. **HTML Downloader:** Downloads web pages from URLs provided by the URL Frontier.
 4. **DNS Resolver:** Converts URLs to IP addresses.
 5. **Content Parser:** Validates and parses web pages.
   - Discards malformed pages.
 6. **Content Seen?:** Checks for duplicate content using hash comparisons (compare the hash values of the two web pages).
 7. **Content Storage:** Stores HTML pages on disk (popular content in memory to reduce latency).
 8. **URL Extractor:** Extracts new links from parsed pages.
 9. **URL Filter:** Excludes blacklisted or erroneous URLs.
 10. **URL Seen?** Tracks visited URLs to avoid duplication.
 11. **URL Storage:** Stores already visited URLs.
 ---
 ### Workflow
 1. Add **Seed URLs** to the URL Frontier.
 2. **HTML Downloader** fetches URLs and resolves their IPs via the DNS Resolver.
 3. **Content Parser** validates and passes content to the "Content Seen?" component.
 4. If the content is new, extract links via the **URL Extractor**.
 5. Filter and add unique links to the URL Frontier.
 ---
 ## Step 3: Deep Dive into Key Components
 ### DFS/BFS
 -  The web can be though of as a directed graph where web pages are nodes and hyperlinks (URLs) as edges.
 -  BFS is usually used for graph traversal as the depth can be be very deep thus DFS is not ideal.
 -  Standard BFS does not take the priority of a URL into consideration, not every page has the same level of quality and importance.
 ### URL Frontier
 - **Politeness:**
    - Ensure only one request per host at a time. Add a dealy b/w two download tasks.
    - Use a mapping from hostnames to queues and worker (download) threads.
    - Each downloader thread has a separate FIFO queue and only downloads URLs from that queue.
      <Frame caption="Politeness">
      ![Politeness](./images/web-crawler/politeness.png)
      </Frame>
    - **Queue router:** Ensures that each queue (b1, b2, … bn) only contains URLs from the same host.
    - **Mapping table:** It maps each host to a queue.
    - **Queue selector:** Each worker thread is mapped to a FIFO queue, and it only downloads URLs from that queue. The queue selection logic is done by the Queue selector.
    - **Worker thread 1 to N.** A worker thread downloads web pages sequentially from the same host. A delay can be added between two download tasks.
 - **Priority:**
    - Assign higher priority to important pages (e.g., by PageRank or update frequency).
      <Frame caption="Politeness">
      ![Politeness](./images/web-crawler/prioritizer.png)
      </Frame>
    - **Prioritizer:** It takes URLs as input and computes the priorities.
    - **Queue f1 to fn:** Each queue has an assigned priority. Queues with high priority are selected with higher probability.
    - **Queue selector:** Randomly choose a queue with a bias towards queues with higher priority.
    - **Front queues:** manage prioritization
    - **Back queues:** manage politeness
 - **Freshness:** Recrawl based on update history or importance.
 ### HTML Downloader
 - **Robots.txt Compliance:** Respect rules in robots.txt files.
 - **Performance Optimizations:**
  1. Distributed crawling using multiple servers.
  2. Use a **DNS cache** to avoid repeated lookups.
  3. Geographically distribute crawl servers for faster downloads.
  4. Use a short timeout to avoid slow or unresponsive servers.
 ### Robustness
 1. **Consistent Hashing:** Distribute load among servers effectively.
 2. **Error Handling:** Prevent system crashes from exceptions.
 3. **Data Validation:** Ensure content integrity.
 ### Extensibility
 - Add modules for new content types (e.g., PNG downloader, web monitor).
 - Example: Plug in a module to monitor web content for copyright violations.
  <Frame caption="Politeness">
  ![Politeness](./images/web-crawler/extensibility.png)
  </Frame>
 ---
 ### Avoiding Problematic Content
 1. **Duplicate Content:** Detect using hash comparisons.
 2. **Spider Traps:** Avoid infinite loops with techniques like URL length limits.
 3. **Data Noise:** Filter irrelevant content like ads or spam.
 ---
 ## Step 4: Wrap Up
 ### Key Takeaways
 1. Web crawlers must balance scalability, robustness, politeness, and extensibility.
 2. **Politeness** prevents overloading servers, while **priority** ensures important pages are crawled first.
 3. Efficient storage and error handling are crucial for handling large-scale crawling.
 ### Additional Considerations
 - **Server-Side Rendering:** Handle dynamic content generated by JavaScript or AJAX.
 - **Anti-Spam Measures:** Exclude low-quality or irrelevant pages.
 - **Database Sharding:** Scale the data layer using replication and sharding.
 - **Horizontal Scaling:** Use stateless servers to scale crawl jobs efficiently.
 - **Analytics:** Collect and analyze data for insights.
 ## Further reading
 - [Web Crawling](http://infolab.stanford.edu/~olston/publications/crawling_survey.pdf)
 - [Google Dynamic Rendering](https://developers.google.com/search/docs/guides/dynamic-rendering)
@@ -1,153 +0,0 @@
 ---
 title: "Design a Notification System"
 description: "Send push, SMS and email notifications at scale through queues and third-party providers, without losing or duplicating them."
 icon: bell
 sidebar:
  label: "Notification System"
 ---
 ## Introduction
 A **notification system** is essential for modern applications, providing timely updates like product notifications, events, offers, and alerts. Notifications can be sent through:
 1. **Push notifications** (mobile or desktop),
 2. **SMS messages**, and
 3. **Emails**.
 The chapter focuses on designing a scalable system capable of sending millions of notifications daily.
 ---
 ## Step 1: Understanding the Problem
 ### Requirements
 - **Notification Types:** Push notifications, SMS, and Emails.
 - **Delivery:** Soft real-time system with minimal delays.
 - **Platforms:** iOS, Android, and desktop.
 - **Triggers:** Notifications can be triggered by client applications or scheduled on servers.
 - **Scale:**
  - **Push Notifications:** 10 million/day,
  - **SMS:** 1 million/day,
  - **Emails:** 5 million/day.
 - **Opt-out Support:** Users can disable specific notification types.
 ---
 ## Step 2: High-Level Design
 ### Components
 1. **Notification Types:**
   - **iOS Push Notifications:** Use **Apple Push Notification Service (APNS)**.
   - **Android Push Notifications:** Use **Firebase Cloud Messaging (FCM)**.
   - **SMS Messages:** Third-party services like Twilio or Nexmo.
   - **Emails:** Commercial email services like SendGrid or Mailchimp.
 2. **Contact Info Gathering:**
   <Frame caption="Contact Info Gathering">
   ![Contact Info Gathering](./images/notification-system/contact-info-gathering.png)
   </Frame>
   - Collect device tokens, phone numbers, or email addresses during app installation or signup.
   - Store contact info in the database:
     - **Device Tokens Table:** For push notifications.
     - **User Table:** For emails and phone numbers.
 3. **Notification Sending Flow:**
   <Frame caption="High Level Design">
   ![High Level Design](./images/notification-system/high-level-design.png)
   </Frame>
   - **Trigger Services:**
      - Generate events to initiate notifications (e.g., billing reminders, shipping updates).
      - A service can be a micro-service, a cron job, or a distributed system that triggers notification sending events.
   - **Notification Server:**
      - Provide APIs for services to send notifications.
      - Carry out basic validations to verify emails, phone numbers.
      - Query the database or cache to fetch data needed to render a notification.
   - **Third-Party Services:** Deliver notifications to users.
 ### Challenges in Initial Design
 - **Single Point of Failure (SPOF):** One notification server can crash the entire system.
 - **Scalability Issues:** Hard to scale databases, caches, and processing components independently.
 - **Performance Bottlenecks:** High resource demands for sending notifications.
 ### Improved Design
 <Frame caption="Improved Design">
 ![Improved Design](./images/notification-system/improved-design.png)
 </Frame>
 - Move databases and caches out of the notification server.
 - Introduce **horizontal scaling** with multiple notification servers.
 - Use **message queues** to decouple system components.
   -  Message queues serve as buffers when high volumes of notifications are to be sent out.
 - Add workers that pull notification events from message queues and send them to corresponding third party services.
 ---
 ## Step 3: Design Deep Dive
 ### Reliability
 1. **Prevent Data Loss:**
   <Frame caption="Data Loss">
   ![Data Loss](./images/notification-system/data-loss.png)
   </Frame>
   - Persist notification data in a database and implement a retry mechanism.
   - The Notification log database is included for data persistence.
 2. **Deduplication:**
   - Check event IDs to avoid sending duplicate notifications.
   - When a notification event first arrives, check if it is seen before by checking the event ID.
 If seen before discard it, otherwise send out the notification.
 ### Additional Components
 <Frame caption="Events Tracking">
 ![Events Tracking](./images/notification-system/events-tracking.png)
 </Frame>
 1. **Notification Templates:** Preformatted templates for consistent and efficient notifications.
 2. **Notification Settings:**
   - Users can opt-in or opt-out for specific channels (push, SMS, or email).
   - Stored in a dedicated notification settings table.
 3. **Rate Limiting:** Cap the frequency of notifications sent to users.
 4. **Retry Mechanism:** Retry sending notifications if third-party services fail.
 5. **Monitoring Queues:** Track queued notifications to scale workers dynamically.
 6. **Event Tracking:** Collect metrics like open rate, click rate, and engagement.
 ### Security
 - Use **AppKey** and **AppSecret** to authenticate and secure APIs for push notifications.
 ### Notification Flow
 <Frame caption="Updated Design">
 ![Updated Design](./images/notification-system/updated-design.png)
 </Frame>
 1. Trigger services call APIs to send notifications.
 2. Notification servers validate requests and fetch metadata from caches or databases.
 3. Notification events are sent to message queues.
 4. Workers process events and interact with third-party services.
 5. Third-party services deliver notifications to users.
 ---
 ## Key Optimizations
 1. **Horizontal Scaling:** Add more notification servers for load distribution.
 2. **Message Queues:** Decouple processing to handle high volumes.
 3. **Caching:** Reduce latency by caching frequently accessed data.
 4. **Distributed Crawling:** Optimize message delivery geographically for better performance.
@@ -1,153 +0,0 @@
 ---
 title: "Design a News Feed System"
 description: "Publish posts and build each user's feed with fanout on write, fanout on read, and layered caches."
 icon: newspaper
 sidebar:
  label: "News Feed System"
 ---
 ## Introduction
 A **news feed system** displays a constantly updating list of posts (status updates, photos, videos, and links) from a user’s connections. Examples include Facebook’s news feed, Instagram’s feed, and Twitter’s timeline. This chapter explores the design of a scalable news feed system.
 ---
 ## Step 1: Understanding the Problem
 ### Requirements
 1. **Platform:** The system supports both web and mobile apps.
 2. **Features:**
   - Users can publish posts.
   - Users can view posts from friends in their news feed.
 3. **Sorting:** Feeds are sorted in **reverse chronological order** for simplicity.
 4. **Scale:**
   - Users can have up to 5,000 friends.
   - 10 million daily active users (DAU).
   - Feeds may include text, images, and videos.
 ---
 ## Step 2: High-Level Design
 ### Overview
 The design includes two main flows:
 1. **Feed Publishing:** A user publishes a post, which is written to the database and propagated to their friends’ feeds.
 2. **News Feed Building:** A user retrieves their news feed by aggregating posts from friends in reverse chronological order.
 ---
 ### News Feed APIs
 1. **Feed Publishing API:**
   - **Endpoint:** `POST /v1/me/feed`
   - **Params:** `content` (post text) and `auth_token` (authentication).
 2. **News Feed Retrieval API:**
   - **Endpoint:** `GET /v1/me/feed`
   - **Params:** `auth_token` (authentication).
 ---
 ### Feed Publishing
 <Frame caption="Feed Publishing">
 ![Feed Publishing](./images/news-feed-system/feed-publishing.png)
 </Frame>
 1. **User Interaction:** The user publishes a post via the feed publishing API.
 2. **Load Balancer:** Distributes traffic to web servers.
 3. **Web Servers:** Authenticate requests and redirect to services.
 4. **Post Service:** Stores the post in the database and cache.
 5. **Fanout Service:** Propagates the post to friends’ news feeds in the cache.
 6. **Notification Service:** Sends notifications to friends.
 ---
 ### News Feed Building
 <Frame caption="News Feed Building">
 ![News Feed Building](./images/news-feed-system/news-feed-building.png)
 </Frame>
 1. **User Interaction:** The user requests their news feed via the retrieval API.
 2. **Load Balancer:** Distributes traffic to web servers.
 3. **Web Servers:** Forward requests to the news feed service.
 4. **News Feed Service:** Fetches post IDs from the news feed cache and retrieves complete post details from the database or cache.
 ---
 ## Step 3: Design Deep Dive
 ### Feed Publishing Deep Dive
 1. **Web Servers:**
   - Authenticate users using `auth_token`.
   - Enforce rate limits to prevent spam.
 2. **Fanout Service:**
   - **Fanout on Write:** Push posts to friends’ feeds at write time.
     - **Pros:** Real-time updates, fast feed retrieval.
     - **Cons:** Resource-intensive for users with many friends.
   - **Fanout on Read:** Pull posts at read time.
     - **Pros:** Efficient for inactive users.
     - **Cons:** Slower feed retrieval.
   - **Hybrid Approach:** Use a push model for most users and a pull model for high-connection users (e.g., celebrities).
     <Frame caption="Feed Publishing Deep Dive">
     ![Feed Publishing Deep Dive](./images/news-feed-system/feed-publishing-deep-dive.png)
     </Frame>
    The **fanout service** works as following:
    1. **Fetch Friend IDs:** Retrieve the friend list from a graph database.
    2. **Filter Friends from Cache:** Access user settings in the cache to exclude certain friends (e.g., muted friends or selective sharing preferences).
    3. **Send to Message Queue:** Send the filtered friend list along with the new post ID to a message queue for processing.
    4. **Fanout Workers:** Workers retrieve data from the message queue and update the news feed cache. The cache stores `<post_id, user_id>` mappings instead of full user and post objects to save memory.
    5. **Store in News Feed Cache:** Append new post IDs to the friends’ news feed cache. A configurable limit ensures that only recent posts are stored, as most users focus on the latest content, keeping cache memory consumption manageable.
       <Frame caption="Fanout Service">
       ![Fanout Service](./images/news-feed-system/fanout-service.png)
       </Frame>
 ## News Feed Retrieval Deep Dive
 ### Cache Architecture
 The cache is divided into five layers:
 1. **News Feed Cache:** Stores post IDs for quick retrieval.
 2. **Content Cache:** Stores post details (popular posts in hot cache).
 3. **Social Graph Cache:** Stores user relationship data.
 4. **Action Cache:** Tracks user actions (likes, replies, shares).
 5. **Counter Cache:** Maintains counts for likes, replies, followers, etc.
   <Frame caption="Cache Architecture">
   ![Cache Architecture](./images/news-feed-system/cache-architecture.png)
   </Frame>
 ---
 ## Key Optimizations
 ### Scaling
 1. **Database Scaling:**
   - Horizontal scaling and sharding.
   - Use of read replicas for high-traffic queries.
 2. **Stateless Web Tier:** Keep web servers stateless to enable horizontal scaling.
 ### Caching
 1. Store frequently accessed data in memory.
 2. Use cache layers to reduce latency and database load.
 ### Reliability
 1. **Consistent Hashing:** Distribute requests evenly across servers.
 2. **Message Queues:** Decouple system components and buffer traffic.
 ### Monitoring
 1. Track key metrics like QPS (queries per second) and latency.
 2. Monitor cache hit rates and adjust configurations accordingly.
@@ -1,248 +0,0 @@
 ---
 title: "Design a Chat System"
 description: "One-to-one and group chat over WebSockets: message storage, ID ordering, sync across devices, and online presence."
 icon: message-circle
 sidebar:
  label: "Chat System"
 ---
 ## Introduction
 A **chat system** supports real-time messaging between users. This chapter focuses on designing a chat app that includes:
 - **One-on-One Chat**
 - **Group Chat (max 100 users)**
 - **Online Presence Indicators**
 - **Multiple Device Support**
 - **Push Notifications**
 The system targets **50 million daily active users (DAU)** and stores chat history permanently.
 ---
 ## Step 1: Understanding the Problem
 ### Requirements
 1. **Features:**
   - One-on-one and group chat (max 100 members).
   - Text-based messages (up to 100,000 characters).
   - Online/offline indicators.
   - Support for multiple devices.
   - Push notifications.
 2. **Scale:** Design for 50 million DAU.
 3. **Storage:** Permanent chat history.
 ---
 ## Step 2: High-Level Design
 ### Communication Protocols
 1. **Sender Side:** HTTP for sending messages, leveraging persistent connections for efficiency.
   <Frame caption="Basic Design">
   ![Basic Design](./images/chat-system/basic-design.png)
   </Frame>
 2. **Receiver Side:**
   - **Polling:**
      - Client periodically asks the server if there are messages available.
      - Inefficient due to frequent, redundant requests.
        <Frame caption="Polling">
        ![Polling](./images/chat-system/polling.png)
        </Frame>
   - **Long Polling:**
      - Keeps the connection open until messages arrive.
      - Inefficient for inactive users.
        <Frame caption="Long Polling">
        ![Long Polling](./images/chat-system/long-polling.png)
        </Frame>
   - **WebSocket:**
      - A bi-directional, persistent connection for real-time communication, chosen for both sending and receiving messages.
      - Uses WebSockets (ws) protocol for sending and recieving messages.
        <Frame caption="Websocket">
        ![Websocket](./images/chat-system/websocket.png)
        </Frame>
 ---
 ### Components
 <Frame caption="High Level Architecture">
 ![High Level Architecture](./images/chat-system/high-level-stateless-arch.png)
 </Frame>
 <Frame caption="High Level Architecture">
 ![High Level Architecture](./images/chat-system/high-level-statefull-arch.png)
 </Frame>
 1. **Stateless Services:**
   - Handle signup, login, and user profile management.
   - Integrated with service discovery to recommend the best chat server.
 2. **Stateful Services:**
   - Chat servers maintain persistent WebSocket connections.
   - Responsible for message delivery and synchronization.
 3. **Third-Party Integration:**
   - Push notification services notify users about new messages.
   - Refer Notification System chapter for notifications implementation.
 ---
 ### Design
 The client maintains a persistent WebSocket connection to a chat server for real-time messaging.
 <Frame caption="High Level Design">
 ![High Level Design](./images/chat-system/high-level-design.png)
 </Frame>
 - Chat servers facilitate message sending/receiving.
 - Presence servers manage online/offline status.
 - API servers handle everything including user login, signup, change profile, etc.
 - Notification servers send push notifications.
 - Finally, the key-value store is used to store chat history.Key-value stores for the database of the chat history data for following reasons:
   - It allows easy horizontal scaling.
   - KV stores provide very low latency to access data.
   - Relational databases do not handle long tail of data well. When the indexes grow
   large, random access is expensive.
   - KV stores are adopted by other proven reliable chat applications. For example,
   both Facebook messenger and Discord.
 Following are the data models for one-to-one chat and group chat.
   - The primary key is message id, which helps to decide message sequence.
   - For the group chat the composite primary key is (channel_id, message_id).
      - IDs can be generated using a global 64-bit sequence number generator like Snowflake.
      - A better approach is to use local sequence number generator. Local means IDs are only unique within a group.
      - The reason why local IDs work is that maintaining message sequence within one-on-one channel or a group channel is sufficient.
     <Frame caption="One to one chat design">
     ![One to one chat design](./images/chat-system/one-to-one-chat.png)
     </Frame>
     <Frame caption="Group chat design">
     ![Group chat design](./images/chat-system/group-chat.png)
     </Frame>
 ## Step 3: Design Deep Dive
 ### Service Discovery
 <Frame caption="Zookeeper">
 ![Zookeeper](./images/chat-system/zookeeper.png)
 </Frame>
 - The primary role of service discovery is to recommend the best chat server for a client based
 on the criteria like geographical location, server capacity.
 - Uses **Apache Zookeeper** to allocate chat servers based on criteria like geographic location and server capacity.
 - Ensures efficient load distribution and minimizes latency.
 ### Messaging Flows
 #### One-on-One Chat
 1. User A sends a message to Chat Server 1.
 2. Chat Server 1 assigns a unique message ID and stores the message in a key-value store.
 3. If User B is online, the message is forwarded to Chat Server 2, maintaining a persistent WebSocket connection.
 4. If User B is offline, a push notification is sent.
 #### Group Chat
 <Frame caption="Group Chat Flow">
 ![Group Chat Flow](./images/chat-system/group-chat-flow.png)
 </Frame>
 - Messages are copied to individual inboxes for each recipient in the group.
 - Simplifies synchronization but becomes expensive for larger groups.
 - On the recipient side, a recipient can receive messages from multiple users. Each recipient
 has an inbox (message sync queue) which contains messages from different senders.
 ---
 #### Message Synchronization
 Many users have multiple devices. We need to synchronize the message across the devices.
 Each device maintains a variable called cur_max_message_id, which keeps track of the latest
 message ID on the device. Messages that satisfy the following two conditions are considered
 as news messages:
 <Frame caption="Message Synchronization">
 ![Message Synchronization](./images/chat-system/message-synchronization.png)
 </Frame>
 - The recipient ID is equal to the currently logged-in user ID.
 - Message ID in the key-value store is larger than cur_max_message_id
 ---
 ### Online Presence
 1. **Heartbeat Mechanism:**
   <Frame caption="Heartbeat Mechanism">
   ![Heartbeat Mechanism](./images/chat-system/heartbeat-mechanism.png)
   </Frame>
   - Clients send periodic heartbeats to presence servers to indicate they are online.
   - If no heartbeat is received within a threshold (for eg x = 30), the user is marked offline.
 2. **Fanout Model:**
   <Frame caption="Fanout Presence">
   ![Fanout Presence](./images/chat-system/fanout-presence.png)
   </Frame>
   - Presence updates are pushed to friends using a publish-subscribe model in which each friend pair maintains a channel.
   - When User A’s online status changes, it publishes the event to three channels, channel A-B, A-C, and A-D.
   - Those three channels are subscribed by User B, C, and D, respectively which get the online status updates.
   - The above design is effective for a small user groups.
 ---
 ## Additional Considerations
 ### Scalability
 - **Horizontal Scaling:** Add servers as user count increases.
 - **Load Balancing:** Distribute traffic evenly across servers.
 - **Caching:** Reduce database load and improve latency.
 ### Error Handling
 - **Retry Mechanisms:** Handle message delivery failures with retries and queuing.
 - **Server Failures:** Use service discovery to allocate new servers in case of failures.
 ### Future Extensions
 1. **Media Support:** Add handling for photos and videos, including compression and cloud storage.
 2. **End-to-End Encryption:** Ensure message privacy.
 3. **Client-Side Caching:** Reduce data transfer for better performance.
 4. **Improved Load Times:** Use geographically distributed caching networks.
 ## Further reading
 - [How Discord stores billions of messages](https://discord.com/blog/how-discord-stores-billions-of-messages)
 - [Flannel: An Application-Level Edge Cache to Make Slack Scale](https://slack.engineering/flannel-an-application-level-edge-cache-to-make-slack-scale/)
@@ -1,214 +0,0 @@
 ---
 title: "Design a Search Autocomplete System"
 description: "Return the top suggestions for a prefix in milliseconds with a trie, a data-gathering service, and sharded storage."
 icon: text-cursor-input
 sidebar:
  label: "Search Autocomplete"
 ---
 ## Introduction
 Autocomplete, also known as typeahead or incremental search, provides real-time suggestions to users as they type in search boxes. The system must efficiently deliver top-k relevant and popular suggestions based on historical query data.
 ### Key Features
 - Suggest up to **5 autocomplete results**.
 - Based on **query popularity** (frequency).
 - Support only **lowercase English characters**.
 - Fast response time (&lt;100 ms) and scalable.
 ---
 ## Step 1: Understanding the Problem
 ### Requirements
 1. **Real-Time Suggestions:** Display relevant matches as the user types.
 2. **Top-k Results:** Return up to 5 results sorted by popularity.
 3. **Scalability:** Handle **10 million DAU** with a peak QPS of **48,000**.
 4. **High Availability:** Handle failures without system downtime.
 5. **Data Growth:** Support daily storage growth of **0.4 GB** for new query data.
 ---
 ## Step 2: High-Level Design
 At the high-level, the system is broken down into two services:
 1. **Data Gathering Service:**
    - Collects user queries and aggregates them for frequency analysis in real-time.
    - Real-time processing is not practical for large data sets; however, it is a good starting point
 2. **Query Service:** Provides the top-k suggestions based on the user’s input.
 ---
 ### Data Gathering Service
 <Frame caption="Data Gathering">
 ![Data Gathering](./images/search-autocomplete/data-gathering.png)
 </Frame>
 - Aggregates query data from analytics logs and updates the frequency table.
 - Processes historical data weekly to build a **trie** (prefix tree).
 ### Query Service
 <Frame caption="Frequency Table">
 ![Frequency Table](./images/search-autocomplete/frequency-table.png)
 </Frame>
 <Frame caption="Search Suggestions">
 ![Search Suggestions](./images/search-autocomplete/basic-search-suggestions.png)
 </Frame>
 - Uses the frequency table from data gathering service.
 - Processes user input and retrieves top-k suggestions from the frequency table using a Trie.
 - Optimized for fast lookups using caching and efficient data structures.
 - For example when a user types “tw” in the search box, the following top 5 searched queries are displayed.
 ---
 ## Step 3: Design Deep Dive
 ### Trie Data Structure
 The **trie** is a tree-like data structure used to store and retrieve query strings efficiently.
 #### Key Features
 1. **Compact Storage:** Represents prefixes hierarchically to minimize redundancy.
 2. **Frequency Information:** Stores the popularity of queries at each node.
 4. **Steps to get top k most searched queries**
   <Frame caption="Trie Structure">
   ![Trie Structure](./images/search-autocomplete/trie-structure.png)
   </Frame>
    - Find the prefix
    - Traverse the subtree from prefix node to get all valid children
    - Sort the children and get top k
 3. **Optimizations:**
   - Cache top-k queries at each node to speed up retrieval and avoid traversing the whole trie.
     <Frame caption="Cached Trie">
     ![Cached Trie](./images/search-autocomplete/cached-trie.png)
     </Frame>
   - Limit prefix length to reduce search space as users rarely type a loong search query (say 50).
 #### Trie Operations
 1. **Create:**
    - Built weekly using aggregated query data.
    - The source of data is from Analytics Log/DB.
 2. **Update:** Rarely updated in real-time; weekly updates replace old data.
 3. **Delete:**
   <Frame caption="Delete KV">
   ![Delete KV](./images/search-autocomplete/delete-kv.png)
   </Frame>
    - Filters remove unwanted or harmful suggestions (e.g., hate speech).
    - Having a filter layer gives us the flexibility of removing results based on different filter rules.
    - Unwanted suggestions are removed physically from the database asynchronically.
 ---
 ### Query Processing Flow
 1. **Prefix Search:**
   - Identify the prefix node corresponding to the user’s input.
   - Traverse the subtree to collect valid suggestions.
 2. **Top-k Sorting:**
   - Cache top-k suggestions at each node to minimize sorting overhead.
 3. **Response Construction:**
   - Construct results using cached data for fast response times.
 ---
 ### Optimizations
 1. **Cache at Each Node:**
   - Store the top-k queries to avoid redundant traversals.
 2. **Limit Prefix Length:**
   - Cap prefix length to a small value (e.g., 50 characters) for faster lookups.
 3. **AJAX Requests:**
   - Use lightweight asynchronous requests for real-time responses.
 4. **Browser Caching:**
   - Save autocomplete results in the browser cache for frequently searched terms.
 ---
 ### Data Gathering Pipeline
 In the high-level design, whenever a user types a search query, data is updated in real-time. This appraoch is not practical.
 - Users may enter billions of queries per day. Updating the trie on every query is not feasible.
 - Top suggestions may not change much one the trie is built.
 #### Updated Design
 <Frame caption="Updated Data Gathering Flow">
 ![Updated Data Gathering Flow](./images/search-autocomplete/data-gathering-flow.png)
 </Frame>
 1. **Analytics Logs:**
   - Stores raw query data as logs for weekly aggregation.
   - Logs are append-only and are not indexed
 2. **Aggregators:**
   - Process logs into frequency tables, suitable for trie construction.
   - For real-time applications such as Twitter, aggregate data in a shorter time interval.
   - For other cases, aggregating data less frequently, say once per week is good enough.
 3. **Workers:**
   - Asynchronous servers rebuild the trie and store it in persistent storage.
 4. **Storage Options:**
    - **Trie Cache**: Trie Cache is a distributed cache system that keeps trie in memory for fast read.
    - **Trie DB**
        1. **Document Store (e.g., MongoDB)**: Since a new trie is built weekly, we can periodically take a snapshot of it, serialize it, and store the serialized data in the database like MongoDB
        2. **Key-Value Store:**
            - Maps prefixes to node data for fast access.
            - Every prefix in the trie is mapped to a key in a hash table.
            - Data on each trie node is mapped to a value in a hash table.
              <Frame caption="Trie DB">
              ![Trie DB](./images/search-autocomplete/trie-db.png)
              </Frame>
 ---
 ### Scalability
 1. **Sharding:**
   - Distribute trie nodes across servers based on prefix ranges (e.g., `a-m`, `n-z`).
   - Further shard within prefixes to balance uneven distributions (e.g., `aa-ag`, `ah-an`).
 2. **Load Balancing:**
   <Frame caption="Sharding">
   ![Sharding](./images/search-autocomplete/sharding.png)
   </Frame>
   - Use a shard map manager to route requests to the appropriate server.
 ---
 ## Step 4: Advanced Features
 ### Multi-Language Support
 1. **Unicode Characters:** Use Unicode to support non-English languages.
 2. **Country-Specific Tries:** Build separate tries for different countries or regions.
 ### Trending Queries
 - Handle real-time events by dynamically updating trie nodes or weighting recent queries more heavily.
 ## Further reading
 - [How We Built Prefixy](https://medium.com/@prefixyteam/how-we-built-prefixy-a-scalable-prefix-search-service-for-powering-autocomplete-c20f98e2eff1)
 - [Prefix Hash Tree](https://people.eecs.berkeley.edu/~sylvia/papers/pht.pdf)
@@ -1,267 +0,0 @@
 ---
 title: "Design YouTube"
 description: "Upload, transcode and stream video at scale with blob storage, a DAG transcoding pipeline, and CDNs."
 icon: video
 sidebar:
  label: "YouTube"
 ---
 ## Introduction
 YouTube is a massive video streaming platform supporting video uploads, playback, and various interactions. This chapter focuses on designing a scalable video streaming system with the following core features:
 - **Fast video uploads**
 - **Smooth video streaming**
 - **Ability to change video quality**
 - **Low infrastructure cost**
 - **High availability and reliability**
 ### Key Statistics (2020)
 - **2 billion monthly active users**
 - **5 billion videos watched per day**
 - **37% of mobile internet traffic comes from YouTube**
 - Available in **80 languages**
 - **$15.1 billion ad revenue** in 2019
 ---
 ## Step 1: Understand the Problem and Scope
 ### Core Functionalities
 1. Upload videos
 2. Watch videos
 ### Supported Platforms
 - Mobile apps, web browsers, and smart TVs
 ### Assumptions
 - **Daily Active Users (DAU):** 5 million
 - **Average Video Size:** 300 MB
 - **Upload Limits:** Max 1 GB per video
 - **Daily Storage Need:** 150 TB
 - **CDN Costs:** 5 million * 5 videos * 0.3GB * $0.02 =  $150,000/day (using Amazon CloudFront)
 ---
 ## Step 2: High-Level Design
 ### Components
 <Frame caption="High Level Design">
 ![High Level Design](./images/youtube/high-level-design.png)
 </Frame>
 1. **Client:** Devices like smartphones, computers, and TVs.
 2. **CDN (Content Delivery Network):** Stores and streams videos.
 3. **API Servers:** Handles all user interactions except video streaming (e.g., uploads, metadata updates).
 4. **Metadata Database:** Stores video metadata (e.g., title, description, size).
 5. **Original Storage:** Blob storage for uploaded videos.
 6. **Transcoding Servers:** Convert videos into multiple resolutions and formats.
 7. **Transcoded Storage:** Blob storage for transcoded videos.
 ---
 ### Core Workflows
 #### 1. Video Uploading Flow
 - **Parallel Processes:**
  1. Upload video to original storage.
  2. Update video metadata in the database.
 - **Video Upload (Steps):**
  <Frame caption="Video Upload Flow">
  ![Video Upload Flow](./images/youtube/video-uploading-flow.png)
  </Frame>
    - [1] Videos are uploaded to blob storage.
    - [2] Transcoding servers convert videos to multiple formats.
    - [3] One trasncoding is complete, following two steps are exectued in parallel.
        - [3a] Transcoded videos are sent to transcoded storage.
        - [3b] Transcoding completion events are queued in the completion queue.
    - [3a.1] Videos are distributed to the CDN.
    - [3b.1] Completion handlers update metadata and inform users.
 - **Metadata Upload (Steps):**
  <Frame caption="Metadata Upload">
  ![Metadata Upload](./images/youtube/metadata-upload.png)
  </Frame>
    - The client in parallel sends a request to update the video metadata
    - The request contains video metadata, including file name, size, format, etc.
 #### 2. Video Streaming Flow
 <Frame caption="Video Streaming Flow">
 ![Video Streaming Flow](./images/youtube/video-streaming-flow.png)
 </Frame>
 - Videos are streamed directly from the CDN using edge servers to minimize latency.
 - Some of te popular streaming protocols are MPEG_DASH, Apple HLS, Adobe HDS.
 -  *Different streaming protocols support different video encodings and playback players.*
 ---
 ## Step 3: Design Deep Dive
 ### Video Transcoding
 #### Importance
 1. Raw video consumes large amounts of storage space. It Reduces storage space.
 2. Ensures compatibility across devices and browsers.
 3. Adapts video quality to network conditions.
 #### Components
 - **Container:** Encapsulates video, audio, and metadata (e.g., MP4, AVI).
 - **Codecs:** Compression and Decompression algorithms (e.g., H.264, VP9).
 #### Directed Acyclic Graph (DAG) Model
 <Frame caption="DAG Video Transcoding">
 ![DAG Video Transcoding](./images/youtube/dag-video-transcoding.png)
 </Frame>
 - Transcoding a video is computationally expensive and time-consuming.
 - DAG Model defines tasks like encoding, thumbnail generation, and watermarking.
 - Allows high parallelism in video processing.
 - The original video is split into video, audio, and metadata.
    - Video encodings: Videos are converted to support different resolutions, codec, bitrates.
    - Thumbnail: It can either be uploaded by a user or automatically generated bythe system.
    - Watermark: Image overlay on top of your video contains identifying information about the video.
 ---
 ### Video Transcoding Architecture
 <Frame caption="Video Transcoding">
 ![Video Transcoding](./images/youtube/video-transcoding-architecture.png)
 </Frame>
 1. **Preprocessor:** Splits videos into smaller chunks (GOP alignment). It has 4 responsibilities.
   <Frame caption="DAG Config">
   ![DAG Config](./images/youtube/dag-config.png)
   </Frame>
    - Video splitting: Video stream is split or further split into smaller Group of Pictures (GOP) alignment.
    - It split videos by GOP alignment for old clients.
    - It generates DAG based on configuration files client programmers write.
    - It stores GOPs and metadata in temporary storage in case the encoding fails, the system could use persisted data for retry operations.
 2. **DAG Scheduler:** Organizes tasks into sequential or parallel stages.
   <Frame caption="DAG Scheduler">
   ![DAG Scheduler](./images/youtube/dag-scheduler.png)
   </Frame>
    - It splits a DAG graph into stages of tasks and puts them in the task queue in the resource manager.
    - Stage 1: video, audio, and metadata.
    - The video file is further split into two tasks in stage 2: video encoding and thumbnail.
 3. **Resource Manager:** Responsible for managing the efficiency of resource allocation.It
 contains 3 queues and a task scheduler.
 <Frame caption="Resource Manager">
 ![Resource Manager](./images/youtube/resource-manager.png)
 </Frame>
    - Task queue: priority queue that contains tasks to be executed.
    - Worker queue: priority queue that contains worker utilization info.
    - Running queue: contains  currently running tasks and workers running the tasks.
    - Task scheduler: picks the optimal task/worker, and instructs the chosen task worker to execute the job.
 4. **Task Workers:** Perform transcoding and other operations.
   <Frame caption="Task Worker">
   ![Task Worker](./images/youtube/task-worker.png)
   </Frame>
    - Different task workers may run different tasks
 5. **Temporary Storage:** Stores intermediate data for retries.
    - The choice of storage system depends on factors like data type, data size, access frequency, data life span, etc.
 6. **Output:** Transcoded videos ready for distribution.
 ---
 ## System Optimizations
 ### Speed Optimizations
 1. **Parallel Video Uploads:** Split videos into smaller chunks for faster, resumable uploads.
   <Frame caption="Video Split">
   ![Video Split](./images/youtube/video-split.png)
   </Frame>
 2. **Distributed Upload Centers:** Use CDNs as upload hubs close to users.
 3. **Parallel Processing:** Decouple modules using message queues for high parallelism.
   <Frame caption="Message Queue">
   ![Message Queue](./images/youtube/message-queue1.png)
   </Frame>
   <Frame caption="Message Queue">
   ![Message Queue](./images/youtube/message-queue2.png)
   </Frame>
 ### Safety Optimizations
 1. **Pre-Signed URLs:** Restrict video uploads to authorized users.
   <Frame caption="Pre Signed">
   ![Pre Signed](./images/youtube/pres-signed-urls.png)
   </Frame>
 2. **Protect Videos:**
   - **DRM Systems** (e.g., Apple FairPlay, Google Widevine).
   - **AES Encryption.**
   - **Watermarking.**
 ### Cost-Saving Optimizations
 1. Serve only popular videos via CDN; less popular ones from high-capacity servers.
 2. Encode on-demand for rarely accessed videos.
 3. Regionalize video distribution based on popularity.
 4. Build custom CDNs and partner with ISPs to reduce bandwidth costs.
 ---
 ## Error Handling
 ### Recoverable Errors
 - Retry failed uploads, transcoding, or resource allocation tasks.
 ### Non-Recoverable Errors
 - Stop malformed video processing and return error codes.
 ## Further reading
 - [YouTube Architecture](http://highscalability.com/youtube-architecture)
 - [YouTube scalability 2012](https://www.youtube.com/watch?v=w5WVu624fY8)
 - [Transcoding Videos at Scale](https://www.egnyte.com/blog/2018/12/transcoding-how-we-serve-videos-at-scale/)
 - [Facebook Video Broadcasting](https://engineering.fb.com/ios/under-the-hood-broadcasting-live-video-to-millions/)
 - [Netflix Video Encoding at Scale](https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746)
 - [Netflix Shot based encoding](https://netflixtechblog.com/optimized-shot-based-encodes-now-streaming-4b9464204830)
@@ -1,254 +0,0 @@
 ---
 title: "Design Google Drive"
 description: "Upload, sync and share files across devices with block storage, delta sync, a metadata database, and notifications."
 icon: hard-drive
 sidebar:
  label: "Google Drive"
 ---
 ## Introduction
 Google Drive is a cloud-based file storage and synchronization service that allows users to store, access, and share files from various devices. This chapter discusses designing a scalable system with the following features:
 - **File Upload and Download**
 - **File Sync Across Devices**
 - **File Sharing**
 - **File Revision History**
 - **Notifications for Edits, Deletes, and Shares**
 ---
 ## Step 1: Understanding the Problem
 ### Key Requirements
 #### Functional Requirements:
 - Upload and download files.
 - Sync files across multiple devices.
 - Maintain file revisions.
 - Enable file sharing with permissions.
 - Send notifications on file edits, deletions, and shares.
 #### Non-Functional Requirements:
 - **Reliability:** Data loss is unacceptable.
 - **Fast Sync Speed:** Avoid user impatience with delayed syncing.
 - **Bandwidth Efficiency:** Minimize unnecessary data usage.
 - **Scalability:** Handle 10 million daily active users (DAU).
 - **High Availability:** Operate seamlessly during server failures or network issues.
 ### Constraints and Assumptions
 - Users get **10 GB free space**.
 - Maximum file size: **10 GB**.
 - Average file upload size: **500 KB**.
 - Upload frequency: **2 files per day per user**.
 - Total storage required: **500 PB**.
 ---
 ## Step 2: High-Level Design
 ### Single-Server Setup
 A basic setup includes:
 1. **Web Server:** Handles uploads and downloads.
 2. **Metadata Database:**  to keep track of metadata like user data, login info, files info/
 3. **Storage Directory:** Holds files organized by namespaces.
 <Frame caption="Namespaces">
 ![Namespaces](./images/google-drive/namespaces.png)
 </Frame>
 - A web server and a directory called drive/ is set up as the root directory to store uploaded files.
 - Under drive/ directory, there is a list of directories called namespaces.
 - Each namespace contains all the uploaded files for that user.
 - Each file or folder can be uniquely identified by joining the namespace and the relative path.
 This design serves as a starting point but is inadequate for scaling.
 #### APIs
 1. **Upload a file to Google Drive:** Two types of uploads are supported
    - Simple upload: Used when file size is small.
    - Resumable upload:
        - Endpoint: https://api.example.com/files/upload?uploadType=resumable
        - Send the initial request to retrieve the resumable URL.
        - Upload the data and monitor upload state
        - If upload is disturbed, resume the upload.
 2. **Download a file from Google Drive:** To download a file
    -  Endpoint: https://api.example.com/files/download
 3. **Get file revisions:**
    - Endpoint: https://api.example.com/files/list_revisions
 ### Moving to Distributed Systems
 #### Improvements:
 1. **Sharding:** Split storage across servers based on `user_id`.
 2. **Amazon S3:** Use S3 for scalable and redundant file storage with cross-region replication.
   <Frame caption="Replication">
   ![Replication](./images/google-drive/replication.png)
   </Frame>
 3. **Load Balancer:** Distribute traffic across multiple web servers.
 4. **Metadata Database Replication:** Ensure availability through database sharding and replication.
 #### Sync Conflicts:
 For a large storage system like Google Drive, sync conflicts happen from time to time.
 When two users modify the same file or folder at the same time, a conflict happens.
 <Frame caption="Sync Conflicts">
 ![Sync Conflicts](./images/google-drive/sync-conflicts.png)
 </Frame>
 - In the example user 1 and user 2 tries to update the same file at the same time, but user 1’s file is processed by our system first.
 - User 1’s update operation goes through, but, user 2 gets a sync conflict.
 - The system presents both copies of the same file: user 2’s local copy and the latest version from the server.
 - User 2 has the option to merge both files or override one version with the other.
 ### Improved design
 <Frame caption="High Level Design">
 ![High Level Design](./images/google-drive/high-level-design.png)
 </Frame>
 1. **User Interaction:**: Users access the application via browser or mobile app.
 2. **Block Servers:**
   - Files are split into **4 MB blocks** (maximum size) and assigned unique hash values.
   - Blocks are stored independently in cloud storage (e.g., Amazon S3).
   - File reconstruction involves joining blocks in a specific order.
 3. **Cloud Storage:** Blocks are stored in cloud storage for scalability and redundancy.
 4. **Cold Storage:** Inactive files are moved to cold storage to reduce costs.
 5. **Load Balancer:** Distributes requests evenly among API servers to ensure efficient operation.
 6. **API Servers:**
   - Handle user authentication, profile management, and file metadata updates.
   - Manage all non-uploading workflows.
 7. **Metadata Database and Cache:**
   - Stores metadata for users, files, blocks, and versions.
   - Frequently accessed metadata is cached for faster retrieval.
 8. **Notification Service:**
   - A **publisher/subscriber system** that notifies clients about file changes (add, edit, delete).
   - Ensures clients can pull the latest updates.
 9. **Offline Backup Queue:** Temporarily stores file change information for offline clients to sync when back online.
 ---
 ## Step 3: Design Deep Dive
 ### Metadata Database
 A highly simplified is shown below version as it only includes the most important tables and fields.
 #### Schema Design:
 - **User Table:** Stores user profiles and preferences.
 - **File Table:** Maintains file metadata (e.g., size, name, path).
 - **Block Table:** Tracks file blocks for reconstructing files.
 - **File Version Table:** Stores file revision history.
 <Frame caption="Metadata Database">
 ![Metadata Database](./images/google-drive/metadata-database.png)
 </Frame>
 ---
 ### File Upload Flow
 1. **File Upload:**
   - File is split into blocks, compressed, and encrypted by the block server.
   - Blocks are uploaded to block servers and stored in S3.
 2. **Metadata Upload:**
   - Client sends metadata to the API server.
   - Metadata is stored in the database with status `pending`.
 3. **Completion:**
   - S3 triggers a callback to update the file status to `uploaded`.
   - Notification service informs relevant users.
 <Frame caption="Upload Flow">
 ![Upload Flow](./images/google-drive/upload-flow.png)
 </Frame>
 ---
 ### File Sync
 1. **Delta Sync:** Transfer only modified blocks instead of the entire file.
   <Frame caption="Delta Sync">
   ![Delta Sync](./images/google-drive/delta-sync.png)
   </Frame>
 2. **Compression:** Blocks are compressed using compression algorithms depending on file types.
 3. **Conflict Resolution:**
   - First processed version wins.
   - Conflicting versions are saved separately for user resolution.
 <Frame caption="File Synce">
 ![File Synce](./images/google-drive/file-sync.png)
 </Frame>
 ---
 ### File Download Flow
 Download flow is triggered when a file is added or edited elsewhere. There are two ways a client can know:
 - If client A is online while a file is changed by another client, notification service will inform client A.
 - If client A is offline while a file is changed by another client, data will be saved to the cache. When the offline client is online again, it pulls the latest changes.
 Once a client knows a file is changed, it first requests metadata via API servers, then
 downloads blocks to construct the file.
 1. **Trigger:** Notification service informs the client of file updates.
 2. **Metadata Fetch:** Client retrieves updated metadata via API.
 3. **Block Download:** Client downloads updated blocks from block servers and reconstructs the file.
 <Frame caption="Upload Flow">
 ![Upload Flow](./images/google-drive/download-flow.png)
 </Frame>
 ---
 ### Notification Service
 1. **Purpose:** Keeps clients updated about file changes.
 2. **Mechanism:** Implements **long polling** for asynchronous notifications.
 3. **Example:** When a file is added, edited, or deleted, notifications are pushed to all relevant clients.
 ---
 ### Storage Optimization
 1. **De-duplication:** Remove duplicate blocks at the account level using hash-based comparisons.
 2. **Versioning Strategy:**
   - Limit the number of saved revisions.
   - Prioritize recent versions for frequently edited files.
 3. **Cold Storage:** Move rarely accessed files to cheaper storage solutions (e.g., Amazon S3 Glacier).
 ---
 ### Failure Handling
 1. **Load Balancer Failure:** Secondary load balancer becomes active.
 2. **Block Server Failure:** Pending tasks are reassigned to other servers.
 3. **Metadata Database Failure:**
   - Promote a slave node to master.
   - Redirect traffic to remaining replicas.
 4. **Cloud Storage Failure:** Use cross-region replication to fetch unavailable files.
 5. **Notification Service Failure:** Clients reconnect to alternative servers.
 ## Further reading
 - [Differential Synchronization](https://neil.fraser.name/writing/sync/)
 - [Differential Synchronization Video](https://www.youtube.com/watch?v=S2Hp_1jqpY8)
 - [How We've Scaled Dropbox](https://www.youtube.com/watch?v=PE4gwstWhmc&feature=youtu.be)
		`@@ -1,3 +0,0 @@`
			`import { defineMeta } from "blume";`

			`export default defineMeta({ title: "Foundations", order: 1 });`