{
  "type": "excalidraw",
  "version": 2,
  "source": "https://excalidraw.com",
  "elements": [
    {
      "id": "bg-top",
      "type": "rectangle",
      "x": 30,
      "y": 20,
      "width": 1740,
      "height": 86,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "#f8fafc",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8000,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "title",
      "type": "text",
      "x": 60,
      "y": 39,
      "width": 841.5000000000007,
      "height": 37.5,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "FlashAttention v1  IO-aware tiling + online softmax",
      "fontSize": 30,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "FlashAttention v1  IO-aware tiling + online softmax",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "subtitle",
      "type": "text",
      "x": 60,
      "y": 75,
      "width": 700.4000000000003,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8002,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "目标：不物化 N x N 的 S/P 矩阵，用 tile 在 SRAM 内完成局部注意力并在线更新输出。",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "目标：不物化 N x N 的 S/P 矩阵，用 tile 在 SRAM 内完成局部注意力并在线更新输出。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "core-panel",
      "type": "rectangle",
      "x": 60,
      "y": 135,
      "width": 500,
      "height": 330,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "#dbeafe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8003,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "core-title",
      "type": "text",
      "x": 85,
      "y": 160,
      "width": 154,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#1e40af",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8004,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "核心瓶颈与解法",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "核心瓶颈与解法",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "core-body",
      "type": "text",
      "x": 85,
      "y": 205,
      "width": 334.90000000000015,
      "height": 255,
      "angle": 0,
      "strokeColor": "#1e3a8a",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8005,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "标准 attention：\n1. S = Q K^T 写 HBM\n2. softmax(S) 写 HBM\n3. P V 再读 HBM\n\nv1 改写为：\n按 Q tile 和 K/V tile 流式扫描\n只在 SRAM/register 中保留局部 S、P\n每行维护 m、l、acc，最后 acc/l 得 O\n\n代价：反向传播重算局部 S/P\n收益：中间矩阵显存从 O(N^2) 降到 O(Nd)",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "标准 attention：\n1. S = Q K^T 写 HBM\n2. softmax(S) 写 HBM\n3. P V 再读 HBM\n\nv1 改写为：\n按 Q tile 和 K/V tile 流式扫描\n只在 SRAM/register 中保留局部 S、P\n每行维护 m、l、acc，最后 acc/l 得 O\n\n代价：反向传播重算局部 S/P\n收益：中间矩阵显存从 O(N^2) 降到 O(Nd)",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "pseudo-panel",
      "type": "rectangle",
      "x": 610,
      "y": 135,
      "width": 540,
      "height": 420,
      "angle": 0,
      "strokeColor": "#16a34a",
      "backgroundColor": "#dcfce7",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8006,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "pseudo-title",
      "type": "text",
      "x": 635,
      "y": 160,
      "width": 421.3000000000002,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#15803d",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8007,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "概念伪代码：一个 CTA 处理一个 Q block",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "概念伪代码：一个 CTA 处理一个 Q block",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "pseudo-code",
      "type": "text",
      "x": 635,
      "y": 205,
      "width": 299.20000000000016,
      "height": 380,
      "angle": 0,
      "strokeColor": "#14532d",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8008,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "parallel for q_block i:\n  Qi = load Q[i] to SMEM/register\n  m = -inf; l = 0; acc = 0\n\n  for k_block j in 0..T:\n    Kj,Vj = load K[j],V[j] to SMEM\n    S = Qi @ Kj.T / sqrt(d)\n    if causal: S += mask(i,j)\n\n    m_new = max(m, rowmax(S))\n    alpha = exp(m - m_new)\n    P = exp(S - m_new)\n\n    acc = acc * alpha + P @ Vj\n    l = l * alpha + rowsum(P)\n    m = m_new\n\n  O[i] = acc / l\n  store O[i], m, l",
      "fontSize": 16,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "parallel for q_block i:\n  Qi = load Q[i] to SMEM/register\n  m = -inf; l = 0; acc = 0\n\n  for k_block j in 0..T:\n    Kj,Vj = load K[j],V[j] to SMEM\n    S = Qi @ Kj.T / sqrt(d)\n    if causal: S += mask(i,j)\n\n    m_new = max(m, rowmax(S))\n    alpha = exp(m - m_new)\n    P = exp(S - m_new)\n\n    acc = acc * alpha + P @ Vj\n    l = l * alpha + rowsum(P)\n    m = m_new\n\n  O[i] = acc / l\n  store O[i], m, l",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "move-panel",
      "type": "rectangle",
      "x": 60,
      "y": 505,
      "width": 1090,
      "height": 260,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "#f8fafc",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8009,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "move-title",
      "type": "text",
      "x": 85,
      "y": 530,
      "width": 198,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#334155",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8010,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "线程与数据搬运过程",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "线程与数据搬运过程",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v1-lane-lane-0",
      "type": "rectangle",
      "x": 95,
      "y": 590,
      "width": 245,
      "height": 72,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "#dbeafe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8011,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "v1-lane-lane-text-0",
      "type": "text",
      "x": 113,
      "y": 607,
      "width": 93.49999999999999,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8012,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "HBM\nQ, K, V, O",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "HBM\nQ, K, V, O",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v1-lane-lane-1",
      "type": "rectangle",
      "x": 372,
      "y": 590,
      "width": 245,
      "height": 72,
      "angle": 0,
      "strokeColor": "#16a34a",
      "backgroundColor": "#dcfce7",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8013,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "v1-lane-lane-text-1",
      "type": "text",
      "x": 390,
      "y": 607,
      "width": 140.24999999999997,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#16a34a",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8014,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "SMEM/SRAM\nQi, Kj, Vj tile",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "SMEM/SRAM\nQi, Kj, Vj tile",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v1-lane-lane-2",
      "type": "rectangle",
      "x": 649,
      "y": 590,
      "width": 245,
      "height": 72,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#f3e8ff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8015,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "v1-lane-lane-text-2",
      "type": "text",
      "x": 667,
      "y": 607,
      "width": 90,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8016,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Registers\nm, l, acc",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Registers\nm, l, acc",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v1-lane-lane-3",
      "type": "rectangle",
      "x": 926,
      "y": 590,
      "width": 245,
      "height": 72,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "#ffedd5",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8017,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "v1-lane-lane-text-3",
      "type": "text",
      "x": 944,
      "y": 607,
      "width": 110.49999999999999,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8018,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Tensor Core\nQK 和 PV MMA",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Tensor Core\nQK 和 PV MMA",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v1-lane-arrow-0",
      "type": "arrow",
      "x": 340,
      "y": 626,
      "width": 32,
      "height": 0,
      "angle": 0,
      "strokeColor": "#3b82f6",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8019,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          32,
          0
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "v1-lane-arrow-1",
      "type": "arrow",
      "x": 617,
      "y": 626,
      "width": 32,
      "height": 0,
      "angle": 0,
      "strokeColor": "#3b82f6",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8020,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          32,
          0
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "v1-lane-arrow-2",
      "type": "arrow",
      "x": 894,
      "y": 626,
      "width": 32,
      "height": 0,
      "angle": 0,
      "strokeColor": "#3b82f6",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8021,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          32,
          0
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "move-notes",
      "type": "text",
      "x": 95,
      "y": 700,
      "width": 1017.4499999999994,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#334155",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8022,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "线程视角：CTA 负责一个 query tile；warp 协作加载 K/V tile；Tensor Core 做 QK 和 PV；CUDA core 做 max、exp、sum。\nv1 典型 sliced-K：不同 warp 分 K/V 切片，Q 对所有 warp 可见；每个 warp 产生局部 acc 后，需要写 SMEM、同步、再归并。",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "线程视角：CTA 负责一个 query tile；warp 协作加载 K/V tile；Tensor Core 做 QK 和 PV；CUDA core 做 max、exp、sum。\nv1 典型 sliced-K：不同 warp 分 K/V 切片，Q 对所有 warp 可见；每个 warp 产生局部 acc 后，需要写 SMEM、同步、再归并。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "thread-panel",
      "type": "rectangle",
      "x": 1190,
      "y": 135,
      "width": 520,
      "height": 630,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#f3e8ff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8023,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "thread-title",
      "type": "text",
      "x": 1215,
      "y": 160,
      "width": 256.3,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#6d28d9",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8024,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "v1 的讲解重点：为什么快",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "v1 的讲解重点：为什么快",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t1-box",
      "type": "rectangle",
      "x": 1225,
      "y": 215,
      "width": 450,
      "height": 70,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#ffffff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8025,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "t1-head",
      "type": "text",
      "x": 1245,
      "y": 227,
      "width": 90,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#6d28d9",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8026,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "少写 HBM",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "少写 HBM",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t1-body",
      "type": "text",
      "x": 1340,
      "y": 227,
      "width": 321.00000000000006,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8027,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "S/P 不落全局内存；HBM 流量是主要收益来源。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "S/P 不落全局内存；HBM 流量是主要收益来源。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t2-box",
      "type": "rectangle",
      "x": 1225,
      "y": 310,
      "width": 450,
      "height": 70,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#ffffff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8028,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "t2-head",
      "type": "text",
      "x": 1245,
      "y": 322,
      "width": 130.89999999999998,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#6d28d9",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8029,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "online softmax",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "online softmax",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t2-body",
      "type": "text",
      "x": 1340,
      "y": 322,
      "width": 302.2500000000001,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8030,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "只要 m 和 l，就能分块得到精确 softmax。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "只要 m 和 l，就能分块得到精确 softmax。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t3-box",
      "type": "rectangle",
      "x": 1225,
      "y": 405,
      "width": 450,
      "height": 70,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#ffffff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8031,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "t3-head",
      "type": "text",
      "x": 1245,
      "y": 417,
      "width": 121.54999999999998,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#6d28d9",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8032,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "kernel fusion",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "kernel fusion",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t3-body",
      "type": "text",
      "x": 1340,
      "y": 417,
      "width": 330.00000000000006,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8033,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "QK、softmax、PV 融合，减少 kernel 间读写。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "QK、softmax、PV 融合，减少 kernel 间读写。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t4-box",
      "type": "rectangle",
      "x": 1225,
      "y": 500,
      "width": 450,
      "height": 70,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#ffffff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8034,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "t4-head",
      "type": "text",
      "x": 1245,
      "y": 512,
      "width": 90,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#6d28d9",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8035,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "重算换内存",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "重算换内存",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t4-body",
      "type": "text",
      "x": 1340,
      "y": 512,
      "width": 271.5,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8036,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "反向不存 P，重算局部分数，节省显存。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "反向不存 P，重算局部分数，节省显存。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t5-box",
      "type": "rectangle",
      "x": 1225,
      "y": 595,
      "width": 450,
      "height": 70,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#ffffff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8037,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "t5-head",
      "type": "text",
      "x": 1245,
      "y": 607,
      "width": 90,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#6d28d9",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8038,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "不足",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "不足",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "t5-body",
      "type": "text",
      "x": 1340,
      "y": 607,
      "width": 466.5000000000002,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8039,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "并行划分还粗：小 batch/少 head 时 SM 吃不满；warp 间通信多。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "并行划分还粗：小 batch/少 head 时 SM 吃不满；warp 间通信多。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "script-panel",
      "type": "rectangle",
      "x": 60,
      "y": 815,
      "width": 1650,
      "height": 240,
      "angle": 0,
      "strokeColor": "#f59e0b",
      "backgroundColor": "#fffbeb",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8040,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "script-title",
      "type": "text",
      "x": 85,
      "y": 840,
      "width": 132,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#b45309",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8041,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "视频讲解顺序",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "视频讲解顺序",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "script-body",
      "type": "text",
      "x": 85,
      "y": 885,
      "width": 743.4000000000002,
      "height": 90,
      "angle": 0,
      "strokeColor": "#78350f",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8042,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "1. 先画出标准 attention 的 S/P 两个 N x N 中间矩阵，强调不是算不动，而是搬不动。\n2. 画 HBM 到 SRAM 的 tile 搬运：K/V 一块一块进来，Q block 固定，输出累加。\n3. 推 online softmax：m_new、alpha、l、acc 四个状态变量解释清楚。\n4. 最后指出 v1 的短板：CTA/warp 划分还不够好，这就自然过渡到 v2。",
      "fontSize": 18,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "1. 先画出标准 attention 的 S/P 两个 N x N 中间矩阵，强调不是算不动，而是搬不动。\n2. 画 HBM 到 SRAM 的 tile 搬运：K/V 一块一块进来，Q block 固定，输出累加。\n3. 推 online softmax：m_new、alpha、l、acc 四个状态变量解释清楚。\n4. 最后指出 v1 的短板：CTA/warp 划分还不够好，这就自然过渡到 v2。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "legend",
      "type": "rectangle",
      "x": 60,
      "y": 1110,
      "width": 1030,
      "height": 70,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "#f8fafc",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "dashed",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8043,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "legend-text",
      "type": "text",
      "x": 80,
      "y": 1128,
      "width": 1109.999999999999,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#334155",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8044,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "读图方法：蓝色表示 HBM/全局内存，绿色表示 SMEM/SRAM，紫色表示寄存器/累加器，橙色表示 Tensor Core 或 CUDA Core 计算。箭头表示数据搬运或流水依赖。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "读图方法：蓝色表示 HBM/全局内存，绿色表示 SMEM/SRAM，紫色表示寄存器/累加器，橙色表示 Tensor Core 或 CUDA Core 计算。箭头表示数据搬运或流水依赖。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "source",
      "type": "rectangle",
      "x": 1220,
      "y": 1110,
      "width": 520,
      "height": 70,
      "angle": 0,
      "strokeColor": "#94a3b8",
      "backgroundColor": "#f8fafc",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "dashed",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8045,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "source-text",
      "type": "text",
      "x": 1240,
      "y": 1128,
      "width": 580.2999999999998,
      "height": 17.5,
      "angle": 0,
      "strokeColor": "#475569",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8046,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "来源：FlashAttention 论文 arXiv:2205.14135；NVIDIA online softmax/tile 说明。",
      "fontSize": 14,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "来源：FlashAttention 论文 arXiv:2205.14135；NVIDIA online softmax/tile 说明。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "practice-interview-panel-flashattention-v1-io-aware-tiling-原理详解-bg",
      "type": "rectangle",
      "x": 80,
      "y": 1260.0,
      "width": 1580,
      "height": 300,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "#e2e8f0",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 70,
      "groupIds": [],
      "seed": 93001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "roundness": {
        "type": 3
      }
    },
    {
      "id": "practice-interview-panel-flashattention-v1-io-aware-tiling-原理详解-title",
      "type": "text",
      "x": 108,
      "y": 1282.0,
      "width": 1524,
      "height": 38,
      "angle": 0,
      "strokeColor": "#1e40af",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 1,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 91001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "练习题与面试题补充：FlashAttention IO-aware attention",
      "fontSize": 24,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "练习题与面试题补充：FlashAttention IO-aware attention",
      "autoResize": false,
      "lineHeight": 1.25
    },
    {
      "id": "practice-interview-panel-flashattention-v1-io-aware-tiling-原理详解-practice-box",
      "type": "rectangle",
      "x": 108,
      "y": 1338.0,
      "width": 748.0,
      "height": 190,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "#bfdbfe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 93002,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "roundness": {
        "type": 3
      }
    },
    {
      "id": "practice-interview-panel-flashattention-v1-io-aware-tiling-原理详解-practice-text",
      "type": "text",
      "x": 130,
      "y": 1358.0,
      "width": 725.0,
      "height": 150,
      "angle": 0,
      "strokeColor": "#243042",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 1,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 91001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "练习题\n1. 手推一个 Q block 扫两个 K/V tile 的 online softmax 更新\n2. 画出 S tile、P tile、O tile 的生命周期\n3. 比较 v1/v2/v3/v4 在 work partition 和 pipeline 上的变化",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "练习题\n1. 手推一个 Q block 扫两个 K/V tile 的 online softmax 更新\n2. 画出 S tile、P tile、O tile 的生命周期\n3. 比较 v1/v2/v3/v4 在 work partition 和 pipeline 上的变化",
      "autoResize": false,
      "lineHeight": 1.25
    },
    {
      "id": "practice-interview-panel-flashattention-v1-io-aware-tiling-原理详解-interview-box",
      "type": "rectangle",
      "x": 884.0,
      "y": 1338.0,
      "width": 748.0,
      "height": 190,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#ddd6fe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 93003,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "roundness": {
        "type": 3
      }
    },
    {
      "id": "practice-interview-panel-flashattention-v1-io-aware-tiling-原理详解-interview-text",
      "type": "text",
      "x": 906.0,
      "y": 1358.0,
      "width": 725.0,
      "height": 150,
      "angle": 0,
      "strokeColor": "#243042",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 1,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 91001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "面试题\n1. FlashAttention 为什么降低 HBM traffic\n2. online softmax 的 m/l/O 三个状态如何保证数值正确\n3. v2 的 work partitioning 相比 v1 改善了什么",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "面试题\n1. FlashAttention 为什么降低 HBM traffic\n2. online softmax 的 m/l/O 三个状态如何保证数值正确\n3. v2 的 work partitioning 相比 v1 改善了什么",
      "autoResize": false,
      "lineHeight": 1.25
    }
  ],
  "appState": {
    "gridSize": null,
    "viewBackgroundColor": "#ffffff"
  },
  "files": {}
}