Compare commits

..
11 Commits
23 changed files with 1977 additions and 161 deletions
+1 -1
View File
@@ -29,7 +29,7 @@
<!-- gitnexus:start --> <!-- gitnexus:start -->
# GitNexus — Code Intelligence # GitNexus — Code Intelligence
This project is indexed by GitNexus as **a-share-analysis** (2186 symbols, 2864 relationships, 50 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely. This project is indexed by GitNexus as **a-share-analysis** (2429 symbols, 3280 relationships, 63 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first. > If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
+1 -1
View File
@@ -37,7 +37,7 @@ npm run schema:verify
npm run governance:start -- FEAT-MO-001 market-overview npm run governance:start -- FEAT-MO-001 market-overview
``` ```
Schema 命令只允许处理固定开发库;当前 MySQL 未启用时会为上一发布版本生成可验证的不适用清单,不会连接数据库或创建空 SQL。完整规则见 [开发流程规范](docs/governance/development-workflow.md)。 Schema 命令只允许处理固定开发库;导出器使用 Node.js `mysql2`,本机无需安装 MySQL Server 或 CLI。当前 MySQL 未启用时会为上一发布版本生成可验证的不适用清单,不会连接数据库或创建空 SQL;启用但无对象时生成 format v3 `captured_empty` 清单。完整规则见 [开发流程规范](docs/governance/development-workflow.md)。
## 项目仓库 ## 项目仓库
@@ -0,0 +1,33 @@
{
"format_version": 3,
"status": "captured_empty",
"application_version": "v0.0.9",
"git_commit": "88e2fdffc8c71dd682405c148a81953a8dcee9d6",
"source_environment": "development",
"database_name": "a_share_analysis_dev",
"charset": "utf8mb4",
"collation": "utf8mb4_general_ci",
"generated_at": "2026-08-17T18:56:33.473Z",
"generator": "schema-baseline/3.0.0",
"reason": "none",
"schema_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"objects": {
"tables": false,
"views": false,
"triggers": false,
"routines": false,
"events": false
},
"object_counts": {
"tables": 0,
"views": 0,
"triggers": 0,
"routines": 0,
"events": 0
},
"exporter_name": "mysql2-show-create",
"exporter_version": "1.0.0",
"database_driver": "mysql2",
"database_driver_version": "3.23.3",
"mysql_server_version": "5.7.42-0ubuntu0.18.04.1"
}
@@ -34,7 +34,7 @@ format v3 使用独立字段记录 Node 导出器名称、导出器版本、数
例程、触发器和事件使用稳定 `DELIMITER`。标识符只能来自元数据,并统一转义反引号。导出结果删除 `DEFINER`、当前 `AUTO_INCREMENT` 值、时间、主机、用户名和客户端噪声;同一结构必须产生逐字节一致结果。 例程、触发器和事件使用稳定 `DELIMITER`。标识符只能来自元数据,并统一转义反引号。导出结果删除 `DEFINER`、当前 `AUTO_INCREMENT` 值、时间、主机、用户名和客户端噪声;同一结构必须产生逐字节一致结果。
## 4. 完整性与安全 ## 4. 安全策略与评估
导出前后分别计算对象摘要,包含对象类型、名称、依赖和定义可观察版本;摘要不一致表示并发 DDL,命令失败。任何对象无法 `SHOW CREATE`、结果为空或类型未知时禁止跳过。 导出前后分别计算对象摘要,包含对象类型、名称、依赖和定义可观察版本;摘要不一致表示并发 DDL,命令失败。任何对象无法 `SHOW CREATE`、结果为空或类型未知时禁止跳过。
@@ -42,15 +42,15 @@ format v3 使用独立字段记录 Node 导出器名称、导出器版本、数
现有 SQL 安全扫描继续拒绝顶层数据操作、连接串、疑似凭据和残留 `DEFINER`;对象体内部合法 DML 通过分隔符解析保留。 现有 SQL 安全扫描继续拒绝顶层数据操作、连接串、疑似凭据和残留 `DEFINER`;对象体内部合法 DML 通过分隔符解析保留。
## 5. 性能与异常恢复 ## 5. 性能目标与评估
Schema 备份不在在线请求路径。使用单连接和有限并发,目标中小型开发库 30 秒内完成;连接、查询、总时长、对象数量或响应大小超过保护值时失败。 Schema 备份不在在线请求路径。使用单连接和有限并发,目标中小型开发库 30 秒内完成;连接、查询、总时长、对象数量或响应大小超过保护值时失败。
产物先写入版本父目录中的临时目录,经过 SQL 扫描、对象计数、manifest 和 SHA-256 一致性校验后原子重命名。失败时删除临时目录,既有版本目录禁止覆盖。 产物先写入版本父目录中的临时目录,经过 SQL 扫描、对象计数、manifest 和 SHA-256 一致性校验后原子重命名。失败时删除临时目录,既有版本目录禁止覆盖。
## 6. 实施迁移 ## 6. 异常恢复与实施迁移
GOV-012 因缺少 `v0.0.9` 基线使用用户授权的一次性人工启动记录。实现通过后立即运行新导出器生成并推送 `v0.0.9` 基线,再重新执行自动 `governance:start`,证明后续开发不再依赖本机 MySQL CLI。 GOV-012 因缺少 `v0.0.9` 基线使用用户授权的一次性人工启动记录。实现通过后已使用 development 配置连接 MySQL 5.7.x,生成并推送 `v0.0.9` format v3 `captured_empty` 基线;随后重新执行自动 `governance:start`,证明后续开发不再依赖本机 MySQL CLI。MySQL 5.7.x 缺少 `VIEW_TABLE_USAGE` 时,导出器回退读取 `information_schema.VIEWS`,其余查询错误不降级。
## 7. 依赖评估 ## 7. 依赖评估
@@ -0,0 +1,44 @@
# GOV-012 开发总结
## 原始开发需求
将 Schema 版本基线从本机 `mysql`/`mysqldump` 子进程迁移为 Node.js 22.x 与 mysql2 直接读取 MySQL 5.7.x,保留历史格式兼容、恢复开发门禁,并满足安全、性能、可追溯和 Gitea 推送规范。
## 需求与范围变化
- 原计划使用 MySQL 5.7 `VIEW_TABLE_USAGE` 读取视图依赖;真实 5.7.42 不提供该表,增加仅针对 ER_UNKNOWN_TABLE 的 `information_schema.VIEWS` 定义解析回退。
- 原计划空库 exporter 返回会话 SET;实际 format v3 空库必须返回空 SQL,修正为 `captured_empty`。
- GOV-012 不接入 HTTP/OpenAPI、Web、Redis、RocketMQ、API、Scheduler 或 Worker 运行时。
## 实际完成需求
- 新增 mysql2 单连接导出器:元数据校验、目录读取、视图排序、六类 `SHOW CREATE`、确定性渲染、并发漂移检测、超时/大小/对象上限和错误脱敏。
- `schema-baseline.mjs` 支持 format v3、异步 exporter 注入、原子文件操作故障注入、CLI 异步入口,并永久只读兼容 v1/v2。
- 更新当前开发规范、开发流程、README、设计和测试计划;真实开发库生成并推送 `v0.0.9` `captured_empty` 清单。
## 修复缺陷
- 拒绝 MariaDB 伪装版本。
- 严格校验触发器时机、事件和执行顺序。
- 防止目录、元数据和 SHOW CREATE 直接 API 泄露数据库错误。
- 保留 `EXPORT_TIMEOUT` 语义,避免包装器降级为普通查询错误。
- 防止全局正则改写例程字符串中的 `AUTO_INCREMENT`,并拒绝导出定义中的主机信息。
- 兼容真实 MySQL 5.7.42 缺少 `VIEW_TABLE_USAGE` 的目录差异。
## 未完成与延期项
- `glob` high 依赖风险按用户批准的范围例外保留为独立治理欠项,未在 GOV-012 中升级前端测试依赖。
- GOV-012 尚未合并到 `main`、发布 `v0.0.10` 或封存 `v0.0.10` 基线;这些是后续 Task 8/9 的发布步骤。
- MCP GitNexus 传输故障仍待工具环境恢复,不影响已保存的 CLI/静态替代证据。
## 关键提交
- `06466f3`、`8e954a3`、`54fd928`、`7e9d906`、`dc8ad7d`
- `c11de09`、`3d12c83`、`874796a`、`afd186c`、`f1c412b`
## 验证证据
- `npm run test:schema-baseline`:52/52;`npm run test:governance`:10/10;`npm run test:mysql-baseline`:2/2。
- `npm run test:environment-baseline`:10/10;contracts:3/3;boundaries:16/16;deployment:6/6。
- `pnpm lint`、`pnpm typecheck`、Prettier、`npm run schema:verify` 均通过。
- 真实 development MySQL 5.7.42 连接成功,`a_share_analysis_dev` 五类对象均为 0,v0.0.9 manifest 已提交并推送。
@@ -0,0 +1,23 @@
# GitNexus 开发后基线
- 功能编号:GOV-012
- 记录时间:2026年08月18日 CST
- 分支:`codex/gov-012-node-schema-exporter`
- 当前提交:`f1c412b`
## GitNexus 状态
```text
Indexed commit: f1c412b
Nodes: 2,383
Edges: 3,234
Clusters: 54
Flows: 63
MCP: Transport closed(按 CLI 与静态证据降级)
```
## 实际影响
- 运行时只影响 Schema 治理工具、测试、治理文档和 `v0.0.9` 开发基线。
- HTTP/OpenAPI、Web API、消息契约、Redis、API、Scheduler、Worker 和部署入口没有代码变更。
- MySQL 连接仅命中用户确认的 development 配置和 `a_share_analysis_dev`;没有生产数据库访问。
@@ -0,0 +1,9 @@
# GitNexus 前后对比
| 项目 | 开发前 | 开发后 | 判断 |
| --- | --- | --- | --- |
| Git 提交 | `3555b7c` | `f1c412b` | GOV-012 独立分支完成实现和文档提交 |
| 索引规模 | 2,112 nodes / 2,790 edges / 50 flows | 2,383 nodes / 3,234 edges / 63 flows | 仅新增导出器、测试和文档相关符号 |
| MCP | `Transport closed` | `Transport closed` | 使用 CLI、静态引用和回归测试降级核对 |
| 数据库范围 | 缺失 `v0.0.9` 基线 | `a_share_analysis_dev`,`captured_empty` | 开发门禁恢复 |
| 跨模块变更 | 待分析 | 无 HTTP/OpenAPI、消息、Redis、运行时服务变更 | 符合影响计划 |
@@ -0,0 +1,17 @@
# GitNexus 变更检测
## 工具证据
- GitNexus MCP `detect_changes` 在开发前后均返回 `Transport closed`。
- `npx gitnexus analyze` 已在关键提交后执行,最终索引为 2,383 nodes、3,234 edges、63 flows。
- `npx gitnexus detect-changes --repo a-share-analysis` 返回 `No changes detected`;CLI 对当前未提交差异的识别不完整,不能单独作为无影响结论。
## 静态与 Git 交叉核对
- `git diff --stat` 和提交清单确认变更集中在 `tools/database/schema-baseline.mjs`、`tools/database/mysql2-schema-exporter.mjs`、对应测试、GOV-012 文档及 `database/schema-baselines/v0.0.9/manifest.yaml`。
- `rg` 静态引用确认 `runCommandLine -> runBaselineGeneration -> exportMysql57Schema` 及历史 `verifyBaselineDirectory` 调用链仍在预期治理路径内。
- 没有 HTTP/OpenAPI、Web API、RocketMQ、Redis、API、Scheduler、Worker 或生产部署文件变更。
## 结论
影响范围符合开发前计划;MCP 传输故障作为残余工具风险保留,不解释为无影响。
@@ -34,3 +34,35 @@
## 高风险处置 ## 高风险处置
这是全部版本开发的关键路径,风险至少为 HIGH。修改任何相关函数前必须重新执行 GitNexus upstream impact;若 MCP 仍不可用,使用静态调用关系、现有 50 条流程和全量回归交叉验证。历史基线失败、数据语句进入 SQL、生产边界被放宽或自动门禁未恢复时立即停止发布并回退。 这是全部版本开发的关键路径,风险至少为 HIGH。修改任何相关函数前必须重新执行 GitNexus upstream impact;若 MCP 仍不可用,使用静态调用关系、现有 50 条流程和全量回归交叉验证。历史基线失败、数据语句进入 SQL、生产边界被放宽或自动门禁未恢复时立即停止发布并回退。
## GOV-012 Task 1 重新核对(2026-08-18)
### 工具状态与静态替代证据
- 已执行 `npx gitnexus analyze`,结果为 `Already up to date`;已执行 `npx gitnexus status`,索引和当前提交均为 `235163c`,状态为 up-to-date。
- GitNexus MCP 在本次开发环境没有可调用传输入口;结合任务启动时持续的 `Transport closed`,不能把 MCP 不可用解释为无影响。本轮使用 `rg -n` 对五个符号检索全部仓库静态引用(排除 `node_modules`、`.git` 与 `.gitnexus`),并以已刷新的 CLI 索引和后续 `git diff` 交叉验证。
- 静态引用集中在 `tools/database/schema-baseline.mjs`、`tools/database/schema-baseline.test.mjs`、GOV-007/009/011/012 的治理档案及 `tools/runtime/verify-runtime.mjs`;Task 1 不修改这些既有执行符号或 format v1/v2 目录。
- 以本 worktree 路径指定 CLI 后,`assertMysql57Metadata` 的 upstream 结果为 CRITICAL(4 个直接调用方、9 个受影响符号、5 组流程),`verifyBaselineDirectory` 为 HIGH(3 个直接调用方、6 个受影响符号、4 组流程)。前者较历史 5/10 计数少一,是当前索引未计入测试引用,不能降低历史关键路径风险判断。
- CLI 将 `runBaselineGeneration` 和 `buildCapturedArtifacts` 显示为 LOW/0;与源码中 `runCommandLine -> runBaselineGeneration -> buildCapturedArtifacts` 的静态调用不一致,因此以静态调用补足,不记录为无影响。`runCommandLine` 与 `tools/runtime/verify-runtime.mjs` 中同名符号歧义,CLI 不支持按 UID 或文件路径消歧,使用 `schema-baseline.mjs` 主入口守卫的静态引用作为替代证据。
### 关键符号影响结论
| 符号 | 直接调用方/静态入口 | 受影响流程 | 实际风险与本任务控制 |
| --- | --- | --- | --- |
| `runBaselineGeneration` | `runCommandLine` 的 `generate` 分支;Schema 单元测试直接调用 | `schema:baseline` 生成、随后 `schema:verify`、当前发布基线校验和 `governance:start` | HIGH;后续异步替换会改变错误传播与注入边界。Task 1 只新建模块,不改此函数。 |
| `assertMysql57Metadata` | `buildCapturedArtifacts`、`buildCapturedEmptyArtifacts`、`verifyBaselineDirectory` 的 captured/captured_empty 分支、`runBaselineGeneration` 与 Schema 单元测试 | captured、captured_empty、历史目录验证、当前发布基线、基线生成和 `governance:start` | CRITICAL;历史 GitNexus 证据为 5 个直接调用方、10 个受影响符号和 5 组流程。Task 1 建立独立 `assertMysql57ServerMetadata`,不改变 v1/v2 语义。 |
| `buildCapturedArtifacts` | `runBaselineGeneration`;Schema 单元测试 | captured 产物、清单/checksum、`schema:verify` | MEDIUM;v3 产物契约将在后续 Task 修改。本任务不改该函数。 |
| `verifyBaselineDirectory` | `runBaselineGeneration` 的已存在目录、MySQL 关闭和 captured 分支;`assertCurrentReleasedBaseline`;`assertCurrentReleasedBaselinePublished`;Schema 单元测试 | `schema:verify`、已发布基线校验、`governance:start` | HIGH;format v1/v2 只读兼容是发布门禁。本任务不改该函数或历史目录。 |
| `runCommandLine` | 模块主入口守卫;内部 `generate` 分支调用 `runBaselineGeneration` | `schema:baseline`、`schema:verify` 的 CLI 非零退出边界 | HIGH;后续改异步入口会影响命令完成时序。本任务不改此函数。 |
### 跨模块同步判断
- HTTP/OpenAPI、Web API、消息契约、Redis、API、Scheduler、Worker、部署:本 Task 仅新增 Node 治理工具安全基础,不接入运行时服务,均无同步代码修改;边界是后续 Task 将替换 Schema CLI 内部实现。
- 数据库:不建立连接、不查询业务数据、不执行 DDL/DML/多语句/锁表;本 Task 仅固定未来连接选项和服务端元数据契约。
- 测试、依赖和文档:新增 mysql2 根开发依赖、Node 单元测试和本影响记录;`pnpm audit --audit-level high` 是依赖安全门禁。
### Task 1 依赖审计范围限定例外
- 官方 registry 审计确认既有 Web 测试依赖链 `@vue/test-utils -> js-beautify -> glob` 存在一项 high;新增的 `mysql2@3.23.3` 不在该依赖路径中。
- 用户于 2026-08-18 明确同意 GOV-012 使用范围限定例外继续:该既有风险作为单独治理欠项处理,本功能不得擅自升级前端测试依赖,也不得把该例外解释为 `mysql2` 安全审计通过之外的通用放行。
- GOV-012 发布前的代码 Review 和开发规范 Review 必须继续披露该残余风险;若后续审计发现 `mysql2`、Schema 导出链路或生产依赖新增 high/critical,则本例外不适用并立即阻断。
@@ -0,0 +1,24 @@
# QA 测试结果
## 通过项
- `npm run test:schema-baseline`:52/52 通过,包含 Node exporter、format v1/v2 只读兼容、format v3、错误脱敏、MySQL 5.7 视图目录回退和真实 `v0.0.9` 基线门禁。
- `npm run test:governance`:10/10 通过。
- `npm run test:mysql-baseline`:2/2 通过。
- `npm run test:environment-baseline`:10/10 通过。
- `npm run test:contracts`:3/3 通过。
- `npm run test:boundaries`:16/16 通过。
- `npm run test:deployment`:6/6 通过。
- `npm run check:runtime`:Node.js 22.11.0、pnpm 10.25.0 通过。
- `pnpm lint`:Web 和 Server ESLint 通过。
- `pnpm typecheck`:Web 与 Server TypeScript 检查通过。
- `npm run schema:verify`:`v0.0.9 (captured_empty)` 通过。
## 真实开发库
- 使用 Git 忽略的 development 本地配置,只读取 `a_share_analysis_dev` 的元数据和 `SHOW CREATE`;服务端为 MySQL 5.7.42,五类对象计数均为 0。
- 生成结果只有 `manifest.yaml`,没有 `schema.sql`、checksum 中没有密码或连接信息。
## 未执行项
- `npm run governance:start -- GOV-012 node-schema-exporter` 未重复覆盖已有开发前报告,工具按历史保护规则拒绝;已用 `schema:verify` 和独立远程基线检查替代验证。
@@ -0,0 +1,9 @@
# 视觉 QA
## 适用性
GOV-012 只修改数据库治理工具、测试、文档和 Schema 清单,不修改 Web 页面、样式、图表或交互流程,因此不适用浏览器截图和视觉回归。
## 边界
下一项市场概览业务功能涉及 Web 页面时,必须按现行规范补充 Playwright 流程和同视口视觉证据;本结论不能外推到后续前端功能。
@@ -0,0 +1,17 @@
# GOV-012 发布说明(待发布)
## 计划版本
`v0.0.10`,发布主题为纯 Node.js 开发库 Schema 导出器。
## 变更
- 移除 Schema 生成对本机 MySQL CLI 的运行时依赖,改用 mysql2 和 MySQL 5.7.x `SHOW CREATE`。
- 新增 format v3,记录 exporter/driver 元数据;format v1/v2 保持只读兼容。
- 恢复下一版本开发前的 Schema 基线门禁,已封存 `v0.0.9` `captured_empty` 基线。
## 发布前置
- 先在 `main` 完成代码 Review、规范 Review、`governance:finish` 和全量验证。
- 创建并推送 `v0.0.10` 标签,再独立生成并推送 `database/schema-baselines/v0.0.10/manifest.yaml`。
- 发布失败时按 `rollback.md` 回退代码提交,禁止删除或覆盖 v0.0.9 基线。
@@ -0,0 +1,24 @@
# 代码 Review
## 范围
Review 覆盖 `mysql2-schema-exporter.mjs`、`schema-baseline.mjs`、对应测试、format v3 清单和文档变更。Task 2/3 Review 发现的问题已通过修复提交关闭:触发器元数据严格校验、目录/SHOW CREATE/元数据错误脱敏、超时错误语义保留、例程字面量不误改、主机信息拒绝,以及 MySQL 5.7 视图目录回退。
## 安全与残余风险
- 只读 `information_schema` 和 `SHOW CREATE`,禁用多语句,不读取业务数据,不执行 DDL/DML,不锁表。
- 配置边界固定为 development、`a_share_analysis_dev` 和开发主机白名单;密码不进入日志、异常、SQL、清单或 checksum。
- 连接、查询、总时长、对象数和定义大小均有上限,所有路径最终关闭连接。
- 既有 Web 测试依赖链 `glob` 的 high 漏洞是用户批准的范围限定治理欠项,不在本功能新增路径;若 mysql2 或导出链出现 high/critical,必须阻断发布。
## 性能与资源影响
- 单连接、按目录顺序读取,查询 5 秒超时、总时长 30 秒、对象 5,000、单定义 1 MiB;视图排序为确定性 Kahn 算法。
- 治理命令不在在线请求路径,不新增 API、Redis、RocketMQ 或 Worker 负载;空库实际生成 `captured_empty`。
## GitNexus 与同步修改
- MCP `Transport closed`,已用 CLI reindex/status、静态引用和回归测试交叉核对;影响集中在 Schema 治理链路。
- HTTP/OpenAPI、Web API、消息、Redis、API、Scheduler、Worker、部署均无需同步修改;Schema 文档、测试脚本和 v0.0.9 基线已同步。
最终结论:passed
@@ -0,0 +1,15 @@
# 开发规范符合性 Review
## 安全、性能与架构检查
- 代码使用中文注释说明开发库边界、错误脱敏、并发变化和 MySQL 5.7 回退;文件职责没有继续过度拆分。
- Node.js 22.x、mysql2、MySQL 5.7.x、`utf8mb4_general_ci` 和 format v3 约束已同步实现、规范和测试。
- 安全策略覆盖凭据、SQL 注入、生产边界、错误、依赖和残余风险;性能目标、保护阈值和不适用压测边界均有证据。
## GitNexus 与开发总结检查
- 开发前影响计划、开发后索引、变更检测和前后对比已归档;MCP 传输限制已明确记录。
- 开发总结记录原始需求、范围变化、实际完成、修复缺陷、延期项、提交和验证证据。
- 提交均为中文 Conventional Commit,当前分支和每次提交均已推送项目 Gitea;未修改 Git 全局配置。
最终结论:passed
@@ -25,8 +25,13 @@
- 当前为空时生成 `captured_empty`;发现实际对象时生成完整 `captured`,不得按预期强制为空。 - 当前为空时生成 `captured_empty`;发现实际对象时生成完整 `captured`,不得按预期强制为空。
- 运行 `schema:verify`、治理测试、格式、lint、类型检查和完整仓库验证。 - 运行 `schema:verify`、治理测试、格式、lint、类型检查和完整仓库验证。
## 安全与性能 ## 安全验证
- 证明没有子进程数据库客户端调用、业务表数据查询、生产配置读取或多语句执行。 - 证明没有子进程数据库客户端调用、业务表数据查询、生产配置读取或多语句执行。
- 固定夹具验证对象数量、响应大小、查询和总时长保护。 - 固定夹具验证对象数量、响应大小、查询和总时长保护。
- 残余风险包括真实库权限差异和并发 DDL;通过失败关闭、摘要复核和人工重跑控制。 - 残余风险包括真实库权限差异和并发 DDL;通过失败关闭、摘要复核和人工重跑控制。
## 性能验证
- 假连接验证单连接、5 秒查询超时、30 秒总时长、5,000 对象和 1 MiB 单定义限制;真实开发库记录导出状态和耗时,不记录 SQL 或凭据。
- 本功能是低频治理命令,不在 API 请求路径,也不承载在线行情吞吐;因此不执行业务负载压测,边界是版本基线生成和验证命令本身。
@@ -144,11 +144,13 @@ count += 1;
### Schema 基线规范 ### Schema 基线规范
- 每个新应用版本开发前必须封存上一已发布版本的开发库 Schema;只能读取 `APP_ENV=development`、`a_share_analysis_dev` 和 MySQL 5.7.x,主机仅允许回环地址或固定开发白名单 `112.192.20.11`,禁止连接、探测或读取生产数据库。配置优先读取 Git 忽略的 `deploy/env/development/api.env.local`。 - 每个新应用版本开发前必须封存上一已发布版本的开发库 Schema;只能读取 `APP_ENV=development`、`a_share_analysis_dev` 和 MySQL 5.7.x,主机仅允许回环地址或固定开发白名单 `112.192.20.11`,禁止连接、探测或读取生产数据库。配置优先读取 Git 忽略的 `deploy/env/development/api.env.local`。
- 导出使用 MySQL 5.7.x `mysqldump` 的 `--no-data --routines --triggers --events`,完整保留表、索引、外键、视图、触发器、存储过程、函数和事件,不得包含任何表数据。 - 导出使用 Node.js 22.x 与 `mysql2` 连接 MySQL 5.7.x,读取固定 `information_schema` 元数据并执行 `SHOW CREATE`,完整保留表、索引、外键、视图、触发器、存储过程、函数和事件,不得包含任何表数据;本机不要求安装 MySQL Server、`mysql` 或 `mysqldump`。
- 规范化必须确定性删除导出时间、环境 `DEFINER` 和表选项中的当前 `AUTO_INCREMENT=<值>`,同时保留列定义的 `AUTO_INCREMENT` 属性和业务 SQL。 - 规范化必须确定性删除导出时间、环境 `DEFINER` 和表选项中的当前 `AUTO_INCREMENT=<值>`,同时保留列定义的 `AUTO_INCREMENT` 属性和业务 SQL。
- 账号密码只能通过受控子进程环境传递,不得进入参数、日志、SQL、清单或 checksum 文件。 - 账号密码只存在于内存连接配置,不得进入命令参数、日志、异常、SQL、清单或 checksum 文件。
- `manifest.yaml`、`checksum.sha256` 和 `schema.sql` 的 SHA-256 必须一致;既有版本目录只允许内容完全一致的幂等验证,不允许覆盖。 - `manifest.yaml`、`checksum.sha256` 和 `schema.sql` 的 SHA-256 必须一致;既有版本目录只允许内容完全一致的幂等验证,不允许覆盖。
- MySQL 未启用时只创建状态为 `not_applicable` 的清单,不得创建空 `schema.sql`;启用但无对象时创建 format v2 的 `captured_empty` 清单和零对象计数,不伪造空 SQL;启用后的缺失基线必须阻断下一版本开发。既有 format v1 基线必须继续只读验证,不得静默升级或重写。 - 导出前后必须复核对象目录和定义摘要,发现并发 DDL 或定义变化立即失败;MySQL 5.7.x 不提供 `VIEW_TABLE_USAGE` 时,只能回退读取 `information_schema.VIEWS.VIEW_DEFINITION` 解析视图依赖。
- MySQL 未启用时只创建状态为 `not_applicable` 的清单,不得创建空 `schema.sql`;启用但无对象时创建 format v3 的 `captured_empty` 清单和零对象计数,不伪造空 SQL;启用后的缺失基线必须阻断下一版本开发。既有 format v1/v2 基线必须继续只读验证,不得静默升级或重写。
- format v3 清单记录 Node exporter 和数据库驱动版本,不复用 v1/v2 的 `mysql_client_version` 与 `mysqldump_client_version` 字段。
## 12. 测试要求 ## 12. 测试要求
+3 -3
View File
@@ -17,7 +17,7 @@ released → superseded
## 3. 开发前流程 ## 3. 开发前流程
1. 确认上一已发布语义化版本及标签提交,执行 `npm run schema:baseline` 和 `npm run schema:verify`。 1. 确认上一已发布语义化版本及标签提交,执行 `npm run schema:baseline` 和 `npm run schema:verify`。
2. MySQL 已启用时只能从 `APP_ENV=development`、`a_share_analysis_dev`、MySQL 5.7、`utf8mb4_general_ci` 和固定开发主机白名单(回环地址或 `112.192.20.11`)导出 Schema;配置优先读取被 Git 忽略的 `deploy/env/development/api.env.local`,不存在时才读取无秘密模板。空开发库保存 format v2 的 `captured_empty` 清单,不创建空 SQL;MySQL 未启用时只保存 `not_applicable` 清单。 2. MySQL 已启用时只能从 `APP_ENV=development`、`a_share_analysis_dev`、MySQL 5.7.x、`utf8mb4_general_ci` 和固定开发主机白名单(回环地址或 `112.192.20.11`)导出 Schema;配置优先读取被 Git 忽略的 `deploy/env/development/api.env.local`,不存在时才读取无秘密模板。空开发库保存 format v3 的 `captured_empty` 清单,不创建空 SQL;MySQL 未启用时只保存 `not_applicable` 清单。
3. 将 `database/schema-baselines/<上一发布版本>/` 作为独立中文提交推送 Gitea,核对本地和远程 SHA;既有版本目录禁止覆盖。只有 Git 已跟踪、工作区无改动、当前分支已推送且远程发布标签提交一致时,`governance:start` 才会继续。 3. 将 `database/schema-baselines/<上一发布版本>/` 作为独立中文提交推送 Gitea,核对本地和远程 SHA;既有版本目录禁止覆盖。只有 Git 已跟踪、工作区无改动、当前分支已推送且远程发布标签提交一致时,`governance:start` 才会继续。
4. 分配功能编号和英文短名称。 4. 分配功能编号和英文短名称。
5. 建立 `docs/features/<编号>-<短名称>/`。 5. 建立 `docs/features/<编号>-<短名称>/`。
@@ -101,10 +101,10 @@ npm run governance:finish -- FEAT-MO-001 market-overview
- 基线目录固定为 `database/schema-baselines/vX.Y.Z/`,版本是被备份的已发布版本。 - 基线目录固定为 `database/schema-baselines/vX.Y.Z/`,版本是被备份的已发布版本。
- 真实基线由 `schema.sql`、`manifest.yaml`、`checksum.sha256` 组成;范围包括表、索引、外键、视图、触发器、存储过程、函数和 Event Scheduler 事件,不含数据。 - 真实基线由 `schema.sql`、`manifest.yaml`、`checksum.sha256` 组成;范围包括表、索引、外键、视图、触发器、存储过程、函数和 Event Scheduler 事件,不含数据。
- MySQL 未启用时目录只能包含 `manifest.yaml`,状态为 `not_applicable`、原因为 `mysql_disabled`;启用但无对象时目录也只包含 `manifest.yaml`,状态为 format v2 的 `captured_empty` 并记录零对象计数。 - MySQL 未启用时目录只能包含 `manifest.yaml`,状态为 `not_applicable`、原因为 `mysql_disabled`;启用但无对象时目录也只包含 `manifest.yaml`,状态为 format v3 的 `captured_empty` 并记录零对象计数。
- `schema:baseline` 不接受 env 路径、环境类型或数据库名参数;配置路径固定在开发环境,生产配置和生产数据库不属于该命令的权限范围。 - `schema:baseline` 不接受 env 路径、环境类型或数据库名参数;配置路径固定在开发环境,生产配置和生产数据库不属于该命令的权限范围。
- 修正已发布基线必须新增治理记录说明原因,禁止静默覆盖或重写历史目录。 - 修正已发布基线必须新增治理记录说明原因,禁止静默覆盖或重写历史目录。
- 首次启用 MySQL 的版本必须在 MySQL 5.7 开发环境生成第一份真实或 `captured_empty` Schema 基线;format v1 历史清单保持只读兼容,禁止因工具升级重写。 - 首次启用 MySQL 的版本必须在 MySQL 5.7 开发环境生成第一份真实或 `captured_empty` Schema 基线;format v1/v2 历史清单保持只读兼容,禁止因工具升级重写。
## 8. 提交流程 ## 8. 提交流程
+2 -1
View File
@@ -19,7 +19,7 @@
"test:governance": "node --test tools/governance/feature-record.test.mjs", "test:governance": "node --test tools/governance/feature-record.test.mjs",
"test:mysql-baseline": "node --test tools/governance/mysql-baseline.test.mjs", "test:mysql-baseline": "node --test tools/governance/mysql-baseline.test.mjs",
"test:environment-baseline": "node --test tools/governance/environment-baseline.test.mjs", "test:environment-baseline": "node --test tools/governance/environment-baseline.test.mjs",
"test:schema-baseline": "node --test tools/database/schema-baseline.test.mjs", "test:schema-baseline": "node --test tools/database/schema-baseline.test.mjs tools/database/mysql2-schema-exporter.test.mjs",
"schema:baseline": "node tools/database/schema-baseline.mjs generate", "schema:baseline": "node tools/database/schema-baseline.mjs generate",
"schema:verify": "node tools/database/schema-baseline.mjs verify", "schema:verify": "node tools/database/schema-baseline.mjs verify",
"governance:start": "node tools/governance/start.mjs", "governance:start": "node tools/governance/start.mjs",
@@ -44,6 +44,7 @@
"eslint": "9.39.5", "eslint": "9.39.5",
"eslint-plugin-vue": "9.33.0", "eslint-plugin-vue": "9.33.0",
"globals": "17.11.0", "globals": "17.11.0",
"mysql2": "3.23.3",
"prettier": "3.9.6", "prettier": "3.9.6",
"typescript": "5.9.3", "typescript": "5.9.3",
"typescript-eslint": "8.46.2" "typescript-eslint": "8.46.2"
+3
View File
@@ -26,6 +26,9 @@ importers:
globals: globals:
specifier: 17.11.0 specifier: 17.11.0
version: 17.11.0 version: 17.11.0
mysql2:
specifier: 3.23.3
version: 3.23.3(@types/node@22.20.0)
prettier: prettier:
specifier: 3.9.6 specifier: 3.9.6
version: 3.9.6 version: 3.9.6
+606
View File
@@ -0,0 +1,606 @@
import { createHash } from 'node:crypto';
import { createRequire } from 'node:module';
import mysql from 'mysql2/promise';
const require = createRequire(import.meta.url);
const mysql2Version = require('mysql2/package.json').version;
const DEVELOPMENT_DATABASE = 'a_share_analysis_dev';
const VALID_TRIGGER_TIMINGS = new Set(['BEFORE', 'AFTER']);
const VALID_TRIGGER_EVENTS = new Set(['INSERT', 'UPDATE', 'DELETE']);
const CATALOG_QUERIES = Object.freeze({
tables: `SELECT TABLE_NAME, TABLE_TYPE
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = ?
ORDER BY TABLE_NAME`,
viewDependencies: `SELECT VIEW_NAME, TABLE_SCHEMA AS REFERENCED_SCHEMA_NAME,
TABLE_NAME AS REFERENCED_TABLE_NAME
FROM information_schema.VIEW_TABLE_USAGE
WHERE VIEW_SCHEMA = ?
ORDER BY VIEW_NAME, REFERENCED_SCHEMA_NAME, REFERENCED_TABLE_NAME`,
viewDefinitions: `SELECT TABLE_NAME AS VIEW_NAME, VIEW_DEFINITION
FROM information_schema.VIEWS
WHERE TABLE_SCHEMA = ?
ORDER BY TABLE_NAME`,
routines: `SELECT ROUTINE_NAME, ROUTINE_TYPE
FROM information_schema.ROUTINES
WHERE ROUTINE_SCHEMA = ?
ORDER BY ROUTINE_TYPE, ROUTINE_NAME`,
triggers: `SELECT TRIGGER_NAME, EVENT_OBJECT_TABLE, ACTION_TIMING,
EVENT_MANIPULATION, ACTION_ORDER
FROM information_schema.TRIGGERS
WHERE TRIGGER_SCHEMA = ?
ORDER BY EVENT_OBJECT_TABLE, ACTION_TIMING, EVENT_MANIPULATION,
ACTION_ORDER, TRIGGER_NAME`,
events: `SELECT EVENT_NAME
FROM information_schema.EVENTS
WHERE EVENT_SCHEMA = ?
ORDER BY EVENT_NAME`,
});
const SERVER_METADATA_QUERY = `SELECT VERSION() AS SERVER_VERSION,
DEFAULT_CHARACTER_SET_NAME AS DATABASE_CHARSET,
DEFAULT_COLLATION_NAME AS DATABASE_COLLATION
FROM information_schema.SCHEMATA
WHERE SCHEMA_NAME = ?`;
const CREATE_RESULT_COLUMNS = Object.freeze({
TABLE: 'Create Table',
VIEW: 'Create View',
PROCEDURE: 'Create Procedure',
FUNCTION: 'Create Function',
TRIGGER: 'SQL Original Statement',
EVENT: 'Create Event',
});
export const MYSQL2_EXPORTER_METADATA = Object.freeze({
name: 'mysql2-show-create',
version: '1.0.0',
driverName: 'mysql2',
driverVersion: mysql2Version,
});
export const DEFAULT_EXPORT_LIMITS = Object.freeze({
connectTimeoutMs: 5_000,
queryTimeoutMs: 5_000,
totalTimeoutMs: 30_000,
maxObjects: 5_000,
maxDefinitionBytes: 1_048_576,
});
export class SchemaExportError extends Error {
constructor(code, message) {
super(message);
this.name = 'SchemaExportError';
this.code = code;
}
}
/**
* 配置在 Schema 治理入口已完成 development、主机白名单和固定库名校验;这里仅映射
* 连接所需字段,固定禁用多语句以防止未来元数据查询被拼接为任意 SQL。
*/
export function buildConnectionOptions(config) {
return {
host: config.MYSQL_HOST,
port: Number(config.MYSQL_PORT),
user: config.MYSQL_USERNAME,
password: config.MYSQL_PASSWORD,
database: config.MYSQL_DATABASE,
charset: 'utf8mb4',
timezone: '+08:00',
connectTimeout: DEFAULT_EXPORT_LIMITS.connectTimeoutMs,
multipleStatements: false,
enableKeepAlive: true,
};
}
/**
* 元数据对象名仍需防御性转义,避免控制字符破坏 `SHOW CREATE` 查询边界;名称不合法时
* 不回显输入,确保异常路径不会把潜在敏感值写入日志。
*/
export function quoteIdentifier(identifier) {
if (typeof identifier !== 'string' || identifier.length === 0 || /[\u0000-\u001F\u007F]/.test(identifier)) {
throw new SchemaExportError('INVALID_IDENTIFIER', 'Schema 标识符不能为空且不得包含控制字符');
}
return `\`${identifier.replaceAll('`', '``')}\``;
}
/**
* 导出器只接受项目数据库兼容基线,拒绝 MariaDB 和 MySQL 5.7 之外的服务器,避免相同
* `SHOW CREATE` 文本在不兼容语义下被误认为可恢复的 Schema 快照。
*/
export function assertMysql57ServerMetadata(metadata) {
const serverVersion = metadata?.serverVersion ?? '';
if (!/^5\.7\.\d+(?:[-+].*)?$/.test(serverVersion) || /mariadb/i.test(serverVersion)) {
throw new SchemaExportError('UNSUPPORTED_SERVER_VERSION', 'MySQL 服务端必须为 5.7.x');
}
if (metadata.databaseCharset !== 'utf8mb4') {
throw new SchemaExportError('INVALID_DATABASE_CHARSET', '数据库字符集必须为 utf8mb4');
}
if (metadata.databaseCollation !== 'utf8mb4_general_ci') {
throw new SchemaExportError('INVALID_DATABASE_COLLATION', '数据库排序规则必须为 utf8mb4_general_ci');
}
}
function requireCatalogText(value) {
if (typeof value !== 'string' || value.length === 0) {
throw new SchemaExportError('CATALOG_SHAPE_INVALID', 'Schema 对象目录字段无效');
}
return value;
}
async function queryCatalogRows(connection, sql, database, limits, options = {}) {
let result;
try {
result = await connection.query({
sql,
values: [database],
timeout: limits.queryTimeoutMs,
});
} catch (error) {
if (options.allowMissingTable && error?.code === 'ER_UNKNOWN_TABLE') {
throw new SchemaExportError('OPTIONAL_CATALOG_UNAVAILABLE', '可选 Schema 目录来源不可用');
}
// mysql2 错误可能包含连接 URL、用户名或密码,目录读取的公共接口一律不透传原始文本。
sanitizeDatabaseQueryError(error);
}
if (!Array.isArray(result) || !Array.isArray(result[0])) {
throw new SchemaExportError('CATALOG_SHAPE_INVALID', 'Schema 对象目录查询结果无效');
}
return result[0];
}
function unquoteViewIdentifier(identifier) {
const value = identifier.startsWith('`') && identifier.endsWith('`')
? identifier.slice(1, -1).replaceAll('``', '`')
: identifier;
return value;
}
function parseViewDependencies(viewRows, database) {
const dependencies = [];
for (const row of viewRows) {
const view = requireCatalogText(row.VIEW_NAME);
const definition = requireCatalogText(row.VIEW_DEFINITION);
const inspectable = maskStringLiterals(definition);
const referencePattern = /\b(?:FROM|JOIN)\s+((?:`[^`]+`|[A-Za-z0-9_$]+)(?:\s*\.\s*(?:`[^`]+`|[A-Za-z0-9_$]+))?)/gi;
for (const match of inspectable.matchAll(referencePattern)) {
const parts = match[1].split(/\s*\.\s*/).map(unquoteViewIdentifier);
dependencies.push({
view,
referencedSchema: parts.length === 2 ? parts[0] : database,
referencedObject: parts.length === 2 ? parts[1] : parts[0],
});
}
}
dependencies.sort(
(left, right) =>
left.view.localeCompare(right.view) ||
left.referencedSchema.localeCompare(right.referencedSchema) ||
left.referencedObject.localeCompare(right.referencedObject),
);
return dependencies;
}
function assertObjectLimit(count, limits) {
if (!Number.isInteger(limits.maxObjects) || limits.maxObjects < 0 || count > limits.maxObjects) {
throw new SchemaExportError('OBJECT_LIMIT_EXCEEDED', 'Schema 对象数量超过安全上限');
}
}
/**
* 目录查询只从 information_schema 读取固定列,库名始终作为参数传入。每类对象
* 在进入后续 SHOW CREATE 前先转换成稳定的小对象,并在累计数量越界时立即停止。
*/
export async function readSchemaCatalog(connection, database, limits = DEFAULT_EXPORT_LIMITS) {
const tableRows = await queryCatalogRows(connection, CATALOG_QUERIES.tables, database, limits);
const tables = [];
const views = [];
for (const row of tableRows) {
const name = requireCatalogText(row.TABLE_NAME);
if (row.TABLE_TYPE === 'BASE TABLE') tables.push(name);
else if (row.TABLE_TYPE === 'VIEW') views.push(name);
else throw new SchemaExportError('UNKNOWN_TABLE_TYPE', 'Schema 目录包含未知表类型');
}
tables.sort();
views.sort();
let objectCount = tables.length + views.length;
assertObjectLimit(objectCount, limits);
let viewDependencies;
try {
const dependencyRows = await queryCatalogRows(
connection,
CATALOG_QUERIES.viewDependencies,
database,
limits,
{ allowMissingTable: true },
);
viewDependencies = dependencyRows.map((row) => ({
view: requireCatalogText(row.VIEW_NAME),
referencedSchema: requireCatalogText(row.REFERENCED_SCHEMA_NAME),
referencedObject: requireCatalogText(row.REFERENCED_TABLE_NAME),
}));
} catch (error) {
if (error.code !== 'OPTIONAL_CATALOG_UNAVAILABLE') throw error;
const viewRows = await queryCatalogRows(
connection,
CATALOG_QUERIES.viewDefinitions,
database,
limits,
);
viewDependencies = parseViewDependencies(viewRows, database);
}
const routineRows = await queryCatalogRows(connection, CATALOG_QUERIES.routines, database, limits);
const routines = routineRows.map((row) => {
const type = requireCatalogText(row.ROUTINE_TYPE);
if (!['FUNCTION', 'PROCEDURE'].includes(type)) {
throw new SchemaExportError('CATALOG_SHAPE_INVALID', 'Schema 目录包含未知例程类型');
}
return { name: requireCatalogText(row.ROUTINE_NAME), type };
});
routines.sort((left, right) => left.type.localeCompare(right.type) || left.name.localeCompare(right.name));
objectCount += routines.length;
assertObjectLimit(objectCount, limits);
const triggerRows = await queryCatalogRows(connection, CATALOG_QUERIES.triggers, database, limits);
const triggers = triggerRows.map((row) => {
const timing = requireCatalogText(row.ACTION_TIMING);
const event = requireCatalogText(row.EVENT_MANIPULATION);
const rawOrder = row.ACTION_ORDER;
const order = Number(rawOrder);
if (
!VALID_TRIGGER_TIMINGS.has(timing) ||
!VALID_TRIGGER_EVENTS.has(event) ||
(typeof rawOrder !== 'number' && typeof rawOrder !== 'string') ||
(typeof rawOrder === 'string' && rawOrder.trim().length === 0) ||
!Number.isInteger(order) ||
order < 1
) {
throw new SchemaExportError('CATALOG_SHAPE_INVALID', 'Schema 触发器执行顺序无效');
}
return {
name: requireCatalogText(row.TRIGGER_NAME),
table: requireCatalogText(row.EVENT_OBJECT_TABLE),
timing,
event,
order,
};
});
triggers.sort(
(left, right) =>
left.table.localeCompare(right.table) ||
left.timing.localeCompare(right.timing) ||
left.event.localeCompare(right.event) ||
left.order - right.order ||
left.name.localeCompare(right.name),
);
objectCount += triggers.length;
assertObjectLimit(objectCount, limits);
const eventRows = await queryCatalogRows(connection, CATALOG_QUERIES.events, database, limits);
const events = eventRows.map((row) => requireCatalogText(row.EVENT_NAME)).sort();
objectCount += events.length;
assertObjectLimit(objectCount, limits);
return Object.freeze({
tables: Object.freeze(tables),
views: Object.freeze(views),
viewDependencies: Object.freeze(viewDependencies.map((dependency) => Object.freeze(dependency))),
routines: Object.freeze(routines.map((routine) => Object.freeze(routine))),
triggers: Object.freeze(triggers.map((trigger) => Object.freeze(trigger))),
events: Object.freeze(events),
});
}
/**
* 视图恢复顺序使用稳定 Kahn 拓扑排序。跨库或未知依赖不能被默默忽略,否则恢复
* 出来的 Schema 可能引用未捕获对象;同层节点始终按名称排序以保证逐字节确定性。
*/
export function orderViewNames(catalog) {
const tables = new Set(catalog.tables ?? []);
const views = new Set(catalog.views ?? []);
const incoming = new Map([...views].map((view) => [view, 0]));
const dependents = new Map([...views].map((view) => [view, []]));
for (const dependency of catalog.viewDependencies ?? []) {
if (dependency.referencedSchema !== DEVELOPMENT_DATABASE) {
throw new SchemaExportError(
'CROSS_SCHEMA_VIEW_DEPENDENCY',
'Schema 视图包含跨库依赖',
);
}
if (!views.has(dependency.view)) {
throw new SchemaExportError('UNKNOWN_VIEW_DEPENDENCY', 'Schema 视图依赖来源未知');
}
if (tables.has(dependency.referencedObject)) continue;
if (!views.has(dependency.referencedObject)) {
throw new SchemaExportError('UNKNOWN_VIEW_DEPENDENCY', 'Schema 视图依赖对象未知');
}
incoming.set(dependency.view, incoming.get(dependency.view) + 1);
dependents.get(dependency.referencedObject).push(dependency.view);
}
for (const names of dependents.values()) names.sort();
const ready = [...views].filter((view) => incoming.get(view) === 0).sort();
const ordered = [];
while (ready.length > 0) {
const view = ready.shift();
ordered.push(view);
for (const dependent of dependents.get(view)) {
incoming.set(dependent, incoming.get(dependent) - 1);
if (incoming.get(dependent) === 0) {
ready.push(dependent);
ready.sort();
}
}
}
if (ordered.length !== views.size) {
throw new SchemaExportError('VIEW_DEPENDENCY_CYCLE', 'Schema 视图依赖存在循环');
}
return ordered;
}
function sha256(content) {
return createHash('sha256').update(content, 'utf8').digest('hex');
}
/** 只屏蔽 SQL 字符串字面量,保留标识符参与 DEFINER 和凭据扫描。 */
function maskStringLiterals(sql) {
return String(sql).replace(/'(?:''|\\.|[^'\\])*'/gs, "''");
}
function removeTableAutoIncrementOutsideLiterals(sql) {
const literals = [];
const masked = String(sql).replace(/'(?:''|\\.|[^'\\])*'/gs, (literal) => {
const index = literals.push(literal) - 1;
return `\u0001${index}\u0002`;
});
const cleaned = masked.replace(
/\sAUTO_INCREMENT=\d+(?=\s(?:DEFAULT\s+)?CHARSET=)/i,
'',
);
return cleaned.replace(/\u0001(\d+)\u0002/g, (_match, index) => literals[Number(index)]);
}
/**
* SHOW CREATE 的格式由服务端决定,这里只移除与结构无关的账号绑定、当前表自增值、
* 行尾分号和换行差异,不重排对象体,也不删除字符串字面量中的业务文本。
*/
export function normalizeCreateDefinition(definition) {
const normalized = String(definition)
.replace(/\r\n?/g, '\n')
.replace(
/^(CREATE\s+(?:ALGORITHM=\S+\s+)?)DEFINER=`(?:``|[^`])+`@`(?:`(?:``|[^`])+`|'(?:''|[^'])+'|\S+)\s+/i,
'$1',
)
.trim()
.replace(/;+\s*$/, '');
const withoutCurrentAutoIncrement = removeTableAutoIncrementOutsideLiterals(normalized);
const inspectable = maskStringLiterals(withoutCurrentAutoIncrement);
if (
/\bDEFINER\s*=/i.test(inspectable) ||
/\b(?:password|passwd|pwd|api[_-]?key|api[_-]?token|access[_-]?token|client[_-]?secret|secret)\s*[:=]/i.test(
inspectable,
) ||
/\b(?:mysql|mariadb):\/\/[^\s'"`]+/i.test(inspectable) ||
/(?:--|\/\*)\s*(?:host|hostname|server)\s*[:=]\s*[^\s*]+/i.test(inspectable)
) {
throw new SchemaExportError('UNSAFE_DEFINITION', 'Schema 对象定义包含禁止内容');
}
return withoutCurrentAutoIncrement;
}
function sanitizeDatabaseQueryError(error) {
if (error instanceof SchemaExportError) throw error;
throw new SchemaExportError('DATABASE_QUERY_FAILED', '开发库 Schema 查询失败');
}
export async function readMysql57ServerMetadata(
connection,
database,
limits = DEFAULT_EXPORT_LIMITS,
) {
let rows;
try {
[rows] = await connection.query({
sql: SERVER_METADATA_QUERY,
values: [database],
timeout: limits.queryTimeoutMs,
});
} catch (error) {
sanitizeDatabaseQueryError(error);
}
if (!Array.isArray(rows) || rows.length !== 1) {
throw new SchemaExportError('DATABASE_METADATA_INVALID', '开发库元数据查询结果无效');
}
const metadata = {
serverVersion: rows[0].SERVER_VERSION,
databaseCharset: rows[0].DATABASE_CHARSET,
databaseCollation: rows[0].DATABASE_COLLATION,
};
assertMysql57ServerMetadata(metadata);
return Object.freeze(metadata);
}
function getCreateDescriptors(catalog) {
const viewOrder = orderViewNames(catalog);
const viewDefinitions = new Map((catalog.views ?? []).map((name) => [name, { name, type: 'VIEW' }]));
return [
...(catalog.tables ?? []).map((name) => ({ name, type: 'TABLE' })),
...(catalog.routines ?? []).map(({ name, type }) => ({ name, type })),
...viewOrder.map((name) => viewDefinitions.get(name)),
...(catalog.triggers ?? []).map(({ name }) => ({ name, type: 'TRIGGER' })),
...(catalog.events ?? []).map((name) => ({ name, type: 'EVENT' })),
];
}
async function readCreateDefinition(connection, descriptor, limits) {
const sql = `SHOW CREATE ${descriptor.type} ${quoteIdentifier(DEVELOPMENT_DATABASE)}.${quoteIdentifier(descriptor.name)}`;
let rows;
try {
[rows] = await connection.query({ sql, timeout: limits.queryTimeoutMs });
} catch (error) {
sanitizeDatabaseQueryError(error);
}
if (!Array.isArray(rows) || rows.length === 0) {
throw new SchemaExportError('SHOW_CREATE_EMPTY', 'SHOW CREATE 未返回对象定义');
}
const rawDefinition = rows[0]?.[CREATE_RESULT_COLUMNS[descriptor.type]];
if (typeof rawDefinition !== 'string' || rawDefinition.length === 0) {
throw new SchemaExportError('SHOW_CREATE_SHAPE_INVALID', 'SHOW CREATE 返回列无效');
}
if (Buffer.byteLength(rawDefinition, 'utf8') > limits.maxDefinitionBytes) {
throw new SchemaExportError('DEFINITION_LIMIT_EXCEEDED', 'Schema 对象定义超过安全上限');
}
return Object.freeze({
...descriptor,
definition: normalizeCreateDefinition(rawDefinition),
sourceSha256: sha256(rawDefinition.replace(/\r\n?/g, '\n')),
});
}
export async function readDefinitions(connection, catalog, limits = DEFAULT_EXPORT_LIMITS) {
const definitions = [];
for (const descriptor of getCreateDescriptors(catalog)) {
definitions.push(await readCreateDefinition(connection, descriptor, limits));
}
return Object.freeze(definitions);
}
function appendDefinition(parts, definition) {
if (['TABLE', 'VIEW'].includes(definition.type)) {
parts.push(`${definition.definition};`);
return;
}
parts.push(`DELIMITER ;;\n${definition.definition} ;;\nDELIMITER ;`);
}
/** 固定恢复顺序和会话保护,不重新格式化对象体。 */
export function renderSchemaSql(catalog, definitions) {
const definitionsByKey = new Map(
definitions.map((definition) => [`${definition.type}:${definition.name}`, definition]),
);
const parts = [
'SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0;',
'SET NAMES utf8mb4;',
];
for (const descriptor of getCreateDescriptors(catalog)) {
const definition = definitionsByKey.get(`${descriptor.type}:${descriptor.name}`);
if (!definition) {
throw new SchemaExportError('SHOW_CREATE_SHAPE_INVALID', 'Schema 对象定义集合不完整');
}
appendDefinition(parts, definition);
}
parts.push('SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS;');
return `${parts.join('\n\n')}\n`;
}
function assertWithinDeadline(startedAt, now, limits) {
if (now() - startedAt > limits.totalTimeoutMs) {
throw new SchemaExportError('EXPORT_TIMEOUT', 'Schema 导出超过总时长限制');
}
}
function buildObjectCounts(catalog) {
return Object.freeze({
tables: catalog.tables.length,
views: catalog.views.length,
triggers: catalog.triggers.length,
routines: catalog.routines.length,
events: catalog.events.length,
});
}
/**
* 单连接执行两轮目录和定义读取;任何漂移都失败。底层数据库异常统一脱敏,连接在
* 成功、查询失败、校验失败和超时路径均只关闭一次。
*/
export async function exportMysql57Schema({
config,
createConnection = mysql.createConnection,
now = Date.now,
limits = DEFAULT_EXPORT_LIMITS,
}) {
const startedAt = now();
let connection;
let result;
let failure;
try {
connection = await createConnection(buildConnectionOptions(config));
const guardedConnection = {
async query(options) {
assertWithinDeadline(startedAt, now, limits);
const queryResult = await connection.query(options);
assertWithinDeadline(startedAt, now, limits);
return queryResult;
},
};
const serverMetadata = await readMysql57ServerMetadata(
guardedConnection,
config.MYSQL_DATABASE,
limits,
);
const beforeCatalog = await readSchemaCatalog(
guardedConnection,
config.MYSQL_DATABASE,
limits,
);
const beforeDefinitions = await readDefinitions(guardedConnection, beforeCatalog, limits);
const afterCatalog = await readSchemaCatalog(
guardedConnection,
config.MYSQL_DATABASE,
limits,
);
if (sha256(JSON.stringify(beforeCatalog)) !== sha256(JSON.stringify(afterCatalog))) {
throw new SchemaExportError(
'CATALOG_CHANGED_DURING_EXPORT',
'Schema 对象目录在导出期间发生变化',
);
}
const afterDefinitions = await readDefinitions(guardedConnection, afterCatalog, limits);
if (sha256(JSON.stringify(beforeDefinitions)) !== sha256(JSON.stringify(afterDefinitions))) {
throw new SchemaExportError(
'DEFINITION_CHANGED_DURING_EXPORT',
'Schema 对象定义在导出期间发生变化',
);
}
assertWithinDeadline(startedAt, now, limits);
const objectCounts = buildObjectCounts(beforeCatalog);
result = Object.freeze({
metadata: Object.freeze({
exporterName: MYSQL2_EXPORTER_METADATA.name,
exporterVersion: MYSQL2_EXPORTER_METADATA.version,
driverName: MYSQL2_EXPORTER_METADATA.driverName,
driverVersion: MYSQL2_EXPORTER_METADATA.driverVersion,
...serverMetadata,
}),
rawSql: Object.values(objectCounts).every((count) => count === 0)
? ''
: renderSchemaSql(beforeCatalog, beforeDefinitions),
objectCounts,
durationMs: now() - startedAt,
});
} catch (error) {
failure =
error instanceof SchemaExportError
? error
: new SchemaExportError('DATABASE_QUERY_FAILED', '开发库 Schema 查询失败');
} finally {
if (connection) {
try {
await connection.end();
} catch {
if (!failure) {
failure = new SchemaExportError('DATABASE_QUERY_FAILED', '开发库 Schema 连接关闭失败');
}
}
}
}
if (failure) throw failure;
return result;
}
@@ -0,0 +1,705 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import {
DEFAULT_EXPORT_LIMITS,
MYSQL2_EXPORTER_METADATA,
SchemaExportError,
assertMysql57ServerMetadata,
buildConnectionOptions,
exportMysql57Schema,
normalizeCreateDefinition,
orderViewNames,
quoteIdentifier,
readDefinitions,
readMysql57ServerMetadata,
readSchemaCatalog,
renderSchemaSql,
} from './mysql2-schema-exporter.mjs';
const config = Object.freeze({
MYSQL_HOST: '127.0.0.1',
MYSQL_PORT: '3306',
MYSQL_DATABASE: 'a_share_analysis_dev',
MYSQL_USERNAME: 'schema_reader',
MYSQL_PASSWORD: 'not-for-logs',
});
test('连接配置固定单连接、utf8mb4、东八区且禁用多语句', () => {
const options = buildConnectionOptions(config);
assert.deepEqual(options, {
host: '127.0.0.1',
port: 3306,
user: 'schema_reader',
password: 'not-for-logs',
database: 'a_share_analysis_dev',
charset: 'utf8mb4',
timezone: '+08:00',
connectTimeout: 5_000,
multipleStatements: false,
enableKeepAlive: true,
});
});
test('标识符转义反引号并拒绝空值和控制字符', () => {
assert.equal(quoteIdentifier('market`view'), '`market``view`');
for (const value of ['', 'line\nbreak', 'zero\0byte']) {
assert.throws(() => quoteIdentifier(value), SchemaExportError);
}
});
test('只接受 MySQL 5.7.x 与固定字符规则', () => {
const valid = {
serverVersion: '5.7.42-log',
databaseCharset: 'utf8mb4',
databaseCollation: 'utf8mb4_general_ci',
};
assert.doesNotThrow(() => assertMysql57ServerMetadata(valid));
for (const serverVersion of [
'5.6.51',
'8.0.40',
'5.7.42-MariaDB',
'10.6.18-MariaDB',
]) {
assert.throws(() => assertMysql57ServerMetadata({ ...valid, serverVersion }));
}
});
function createCatalogConnection(overrides = {}) {
const rowsBySource = {
'information_schema.TABLES': [
{ TABLE_NAME: 'market_snapshot', TABLE_TYPE: 'BASE TABLE' },
{ TABLE_NAME: 'market_dashboard', TABLE_TYPE: 'VIEW' },
{ TABLE_NAME: 'market_summary', TABLE_TYPE: 'VIEW' },
],
'information_schema.VIEW_TABLE_USAGE': [
{
VIEW_NAME: 'market_dashboard',
REFERENCED_SCHEMA_NAME: 'a_share_analysis_dev',
REFERENCED_TABLE_NAME: 'market_summary',
},
{
VIEW_NAME: 'market_summary',
REFERENCED_SCHEMA_NAME: 'a_share_analysis_dev',
REFERENCED_TABLE_NAME: 'market_snapshot',
},
],
'information_schema.ROUTINES': [
{ ROUTINE_NAME: 'calculate_breadth', ROUTINE_TYPE: 'FUNCTION' },
{ ROUTINE_NAME: 'refresh_market_snapshot', ROUTINE_TYPE: 'PROCEDURE' },
],
'information_schema.TRIGGERS': [
{
TRIGGER_NAME: 'snapshot_before_insert',
EVENT_OBJECT_TABLE: 'market_snapshot',
ACTION_TIMING: 'BEFORE',
EVENT_MANIPULATION: 'INSERT',
ACTION_ORDER: 1,
},
],
'information_schema.EVENTS': [{ EVENT_NAME: 'purge_market_snapshot' }],
...overrides,
};
const calls = [];
return {
calls,
async query(options) {
calls.push(options);
const source = Object.keys(rowsBySource).find((candidate) => options.sql.includes(candidate));
assert.ok(source, `未识别的目录查询:${options.sql}`);
return [rowsBySource[source], []];
},
};
}
test('目录查询使用固定参数和超时并返回稳定对象', async () => {
const connection = createCatalogConnection();
const catalog = await readSchemaCatalog(connection, 'a_share_analysis_dev', DEFAULT_EXPORT_LIMITS);
assert.deepEqual(catalog, {
tables: ['market_snapshot'],
views: ['market_dashboard', 'market_summary'],
viewDependencies: [
{
view: 'market_dashboard',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_summary',
},
{
view: 'market_summary',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_snapshot',
},
],
routines: [
{ name: 'calculate_breadth', type: 'FUNCTION' },
{ name: 'refresh_market_snapshot', type: 'PROCEDURE' },
],
triggers: [
{
name: 'snapshot_before_insert',
table: 'market_snapshot',
timing: 'BEFORE',
event: 'INSERT',
order: 1,
},
],
events: ['purge_market_snapshot'],
});
assert.equal(connection.calls.length, 5);
for (const call of connection.calls) {
assert.deepEqual(call.values, ['a_share_analysis_dev']);
assert.equal(call.timeout, DEFAULT_EXPORT_LIMITS.queryTimeoutMs);
}
});
test('MySQL 5.7 缺少 VIEW_TABLE_USAGE 时从 VIEWS 定义解析依赖', async () => {
const fallback = createCatalogConnection();
const connection = {
calls: fallback.calls,
async query(options) {
if (options.sql.includes('information_schema.VIEW_TABLE_USAGE')) {
throw Object.assign(new Error('unknown metadata table'), { code: 'ER_UNKNOWN_TABLE' });
}
if (options.sql.includes('information_schema.VIEWS')) {
return [
[
{
VIEW_NAME: 'market_dashboard',
VIEW_DEFINITION:
'SELECT * FROM `a_share_analysis_dev`.`market_summary`',
},
{
VIEW_NAME: 'market_summary',
VIEW_DEFINITION: 'SELECT * FROM `market_snapshot`',
},
],
[],
];
}
return fallback.query(options);
},
};
const catalog = await readSchemaCatalog(connection, 'a_share_analysis_dev', DEFAULT_EXPORT_LIMITS);
assert.deepEqual(catalog.viewDependencies, [
{
view: 'market_dashboard',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_summary',
},
{
view: 'market_summary',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_snapshot',
},
]);
});
test('视图按依赖拓扑排序且同层按名称稳定排序', () => {
const catalog = {
tables: ['market_snapshot'],
views: ['z_dashboard', 'market_summary', 'a_dashboard'],
viewDependencies: [
{
view: 'z_dashboard',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_summary',
},
{
view: 'a_dashboard',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_summary',
},
{
view: 'market_summary',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_snapshot',
},
],
};
assert.deepEqual(orderViewNames(catalog), ['market_summary', 'a_dashboard', 'z_dashboard']);
});
test('视图排序拒绝循环、跨库和未知依赖', () => {
const base = { tables: ['market_snapshot'], views: ['first_view', 'second_view'] };
const invalidCases = [
{
code: 'VIEW_DEPENDENCY_CYCLE',
dependencies: [
{ view: 'first_view', referencedSchema: 'a_share_analysis_dev', referencedObject: 'second_view' },
{ view: 'second_view', referencedSchema: 'a_share_analysis_dev', referencedObject: 'first_view' },
],
},
{
code: 'CROSS_SCHEMA_VIEW_DEPENDENCY',
dependencies: [
{ view: 'first_view', referencedSchema: 'production', referencedObject: 'market_snapshot' },
],
},
{
code: 'UNKNOWN_VIEW_DEPENDENCY',
dependencies: [
{ view: 'first_view', referencedSchema: 'a_share_analysis_dev', referencedObject: 'missing_object' },
],
},
];
for (const invalid of invalidCases) {
assert.throws(
() => orderViewNames({ ...base, viewDependencies: invalid.dependencies }),
(error) => error instanceof SchemaExportError && error.code === invalid.code,
);
}
});
test('目录拒绝未知表类型和超过对象上限', async () => {
const unknownType = createCatalogConnection({
'information_schema.TABLES': [{ TABLE_NAME: 'temporary_data', TABLE_TYPE: 'SYSTEM VIEW' }],
});
await assert.rejects(
() => readSchemaCatalog(unknownType, 'a_share_analysis_dev', DEFAULT_EXPORT_LIMITS),
(error) => error instanceof SchemaExportError && error.code === 'UNKNOWN_TABLE_TYPE',
);
const overLimit = createCatalogConnection({
'information_schema.TABLES': [
{ TABLE_NAME: 'first_table', TABLE_TYPE: 'BASE TABLE' },
{ TABLE_NAME: 'second_table', TABLE_TYPE: 'BASE TABLE' },
],
});
await assert.rejects(
() => readSchemaCatalog(overLimit, 'a_share_analysis_dev', { ...DEFAULT_EXPORT_LIMITS, maxObjects: 1 }),
(error) => error instanceof SchemaExportError && error.code === 'OBJECT_LIMIT_EXCEEDED',
);
});
test('目录拒绝非法触发器时机、事件和空执行顺序', async () => {
for (const trigger of [
{
TRIGGER_NAME: 'invalid_timing',
EVENT_OBJECT_TABLE: 'market_snapshot',
ACTION_TIMING: 'DURING',
EVENT_MANIPULATION: 'INSERT',
ACTION_ORDER: 1,
},
{
TRIGGER_NAME: 'invalid_event',
EVENT_OBJECT_TABLE: 'market_snapshot',
ACTION_TIMING: 'BEFORE',
EVENT_MANIPULATION: 'MERGE',
ACTION_ORDER: 1,
},
{
TRIGGER_NAME: 'empty_order',
EVENT_OBJECT_TABLE: 'market_snapshot',
ACTION_TIMING: 'BEFORE',
EVENT_MANIPULATION: 'INSERT',
ACTION_ORDER: '',
},
{
TRIGGER_NAME: 'missing_order',
EVENT_OBJECT_TABLE: 'market_snapshot',
ACTION_TIMING: 'BEFORE',
EVENT_MANIPULATION: 'INSERT',
ACTION_ORDER: null,
},
]) {
const connection = createCatalogConnection({
'information_schema.TRIGGERS': [trigger],
});
await assert.rejects(
() => readSchemaCatalog(connection, 'a_share_analysis_dev', DEFAULT_EXPORT_LIMITS),
(error) => error instanceof SchemaExportError && error.code === 'CATALOG_SHAPE_INVALID',
);
}
});
test('直接目录读取将数据库异常转为固定脱敏错误', async () => {
const connection = {
async query() {
throw new Error('connect mysql://schema_reader:not-for-logs@127.0.0.1:3306/a_share_analysis_dev');
},
};
await assert.rejects(
() => readSchemaCatalog(connection, 'a_share_analysis_dev', DEFAULT_EXPORT_LIMITS),
(error) =>
error instanceof SchemaExportError &&
error.code === 'DATABASE_QUERY_FAILED' &&
error.message === '开发库 Schema 查询失败',
);
});
const completeCatalog = Object.freeze({
tables: ['market_snapshot'],
views: ['market_dashboard', 'market_summary'],
viewDependencies: [
{
view: 'market_dashboard',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_summary',
},
{
view: 'market_summary',
referencedSchema: 'a_share_analysis_dev',
referencedObject: 'market_snapshot',
},
],
routines: [
{ name: 'calculate_breadth', type: 'FUNCTION' },
{ name: 'refresh_market_snapshot', type: 'PROCEDURE' },
],
triggers: [
{
name: 'snapshot_before_insert',
table: 'market_snapshot',
timing: 'BEFORE',
event: 'INSERT',
order: 1,
},
],
events: ['purge_market_snapshot'],
});
const createRows = Object.freeze({
'SHOW CREATE TABLE `a_share_analysis_dev`.`market_snapshot`': {
'Create Table':
'CREATE TABLE `market_snapshot` (`id` bigint NOT NULL AUTO_INCREMENT) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8mb4',
},
'SHOW CREATE VIEW `a_share_analysis_dev`.`market_summary`': {
'Create View':
'CREATE ALGORITHM=UNDEFINED DEFINER=`schema_user`@`192.168.1.10` SQL SECURITY DEFINER VIEW `market_summary` AS select `market_snapshot`.`id` AS `id` from `market_snapshot`',
},
'SHOW CREATE VIEW `a_share_analysis_dev`.`market_dashboard`': {
'Create View':
'CREATE ALGORITHM=UNDEFINED DEFINER=`schema_user`@`192.168.1.10` SQL SECURITY DEFINER VIEW `market_dashboard` AS select `market_summary`.`id` AS `id` from `market_summary`',
},
'SHOW CREATE FUNCTION `a_share_analysis_dev`.`calculate_breadth`': {
'Create Function':
"CREATE DEFINER=`schema_user`@`192.168.1.10` FUNCTION `calculate_breadth`() RETURNS int RETURN 1",
},
'SHOW CREATE PROCEDURE `a_share_analysis_dev`.`refresh_market_snapshot`': {
'Create Procedure':
'CREATE DEFINER=`schema_user`@`192.168.1.10` PROCEDURE `refresh_market_snapshot`()\nBEGIN\n UPDATE `market_snapshot` SET `id` = `id`;\nEND',
},
'SHOW CREATE TRIGGER `a_share_analysis_dev`.`snapshot_before_insert`': {
'SQL Original Statement':
'CREATE DEFINER=`schema_user`@`192.168.1.10` TRIGGER `snapshot_before_insert` BEFORE INSERT ON `market_snapshot` FOR EACH ROW SET NEW.id = NEW.id',
},
'SHOW CREATE EVENT `a_share_analysis_dev`.`purge_market_snapshot`': {
'Create Event':
'CREATE DEFINER=`schema_user`@`192.168.1.10` EVENT `purge_market_snapshot` ON SCHEDULE EVERY 1 DAY DO DELETE FROM `market_snapshot` WHERE `id` < 0',
},
});
function catalogRowsFor(sql) {
if (sql.includes('information_schema.TABLES')) {
return [
{ TABLE_NAME: 'market_snapshot', TABLE_TYPE: 'BASE TABLE' },
{ TABLE_NAME: 'market_dashboard', TABLE_TYPE: 'VIEW' },
{ TABLE_NAME: 'market_summary', TABLE_TYPE: 'VIEW' },
];
}
if (sql.includes('information_schema.VIEW_TABLE_USAGE')) {
return [
{
VIEW_NAME: 'market_dashboard',
REFERENCED_SCHEMA_NAME: 'a_share_analysis_dev',
REFERENCED_TABLE_NAME: 'market_summary',
},
{
VIEW_NAME: 'market_summary',
REFERENCED_SCHEMA_NAME: 'a_share_analysis_dev',
REFERENCED_TABLE_NAME: 'market_snapshot',
},
];
}
if (sql.includes('information_schema.ROUTINES')) {
return [
{ ROUTINE_NAME: 'calculate_breadth', ROUTINE_TYPE: 'FUNCTION' },
{ ROUTINE_NAME: 'refresh_market_snapshot', ROUTINE_TYPE: 'PROCEDURE' },
];
}
if (sql.includes('information_schema.TRIGGERS')) {
return [
{
TRIGGER_NAME: 'snapshot_before_insert',
EVENT_OBJECT_TABLE: 'market_snapshot',
ACTION_TIMING: 'BEFORE',
EVENT_MANIPULATION: 'INSERT',
ACTION_ORDER: 1,
},
];
}
if (sql.includes('information_schema.EVENTS')) {
return [{ EVENT_NAME: 'purge_market_snapshot' }];
}
return undefined;
}
function createExportConnection({ mutateCatalog, mutateDefinition, queryError } = {}) {
let catalogRound = 0;
let definitionRound = 0;
let tableCatalogCalls = 0;
let tableDefinitionCalls = 0;
const connection = {
endCalls: 0,
async query(options) {
if (queryError) throw queryError;
if (options.sql.includes('information_schema.SCHEMATA')) {
return [
[
{
SERVER_VERSION: '5.7.42-log',
DATABASE_CHARSET: 'utf8mb4',
DATABASE_COLLATION: 'utf8mb4_general_ci',
},
],
[],
];
}
const catalogRows = catalogRowsFor(options.sql);
if (catalogRows) {
if (options.sql.includes('information_schema.TABLES')) {
tableCatalogCalls += 1;
catalogRound = tableCatalogCalls > 1 ? 1 : 0;
}
const rows = mutateCatalog?.(options.sql, catalogRows, catalogRound) ?? catalogRows;
return [rows, []];
}
const row = createRows[options.sql];
assert.ok(row, `未识别的 SHOW CREATE:${options.sql}`);
if (options.sql.includes('SHOW CREATE TABLE')) {
tableDefinitionCalls += 1;
definitionRound = tableDefinitionCalls > 1 ? 1 : 0;
}
const changed = mutateDefinition?.(options.sql, row, definitionRound);
return [changed === undefined ? [row] : changed, []];
},
async end() {
connection.endCalls += 1;
},
};
return connection;
}
test('读取并校验固定开发库服务端元数据', async () => {
const connection = createExportConnection();
assert.deepEqual(
await readMysql57ServerMetadata(connection, 'a_share_analysis_dev', DEFAULT_EXPORT_LIMITS),
{
serverVersion: '5.7.42-log',
databaseCharset: 'utf8mb4',
databaseCollation: 'utf8mb4_general_ci',
},
);
});
test('SHOW CREATE 覆盖全部对象并按可恢复顺序渲染', async () => {
const connection = createExportConnection();
const definitions = await readDefinitions(connection, completeCatalog, DEFAULT_EXPORT_LIMITS);
const sql = renderSchemaSql(completeCatalog, definitions);
const order = [
'CREATE TABLE `market_snapshot`',
'FUNCTION `calculate_breadth`',
'PROCEDURE `refresh_market_snapshot`',
'VIEW `market_summary`',
'VIEW `market_dashboard`',
'TRIGGER `snapshot_before_insert`',
'EVENT `purge_market_snapshot`',
].map((fragment) => sql.indexOf(fragment));
assert.ok(order.every((index) => index >= 0));
assert.deepEqual(order, [...order].sort((left, right) => left - right));
assert.match(sql, /DELIMITER ;;\nCREATE PROCEDURE[\s\S]+END ;;\nDELIMITER ;/);
assert.doesNotMatch(sql, /DEFINER\s*=/i);
assert.doesNotMatch(sql, /AUTO_INCREMENT=12/);
assert.match(sql, /`id` bigint NOT NULL AUTO_INCREMENT/);
assert.match(sql, /UPDATE `market_snapshot`/);
});
test('定义规范化只移除账号绑定和当前自增值', () => {
const routine = normalizeCreateDefinition(
"CREATE DEFINER=`user`@`host` PROCEDURE `sample`() SELECT 'DEFINER=`kept`@`text` AUTO_INCREMENT=99'",
);
const table = normalizeCreateDefinition(
'CREATE TABLE `sample` (`id` bigint AUTO_INCREMENT) ENGINE=InnoDB AUTO_INCREMENT=99 DEFAULT CHARSET=utf8mb4;',
);
assert.equal(
routine,
"CREATE PROCEDURE `sample`() SELECT 'DEFINER=`kept`@`text` AUTO_INCREMENT=99'",
);
assert.equal(
table,
'CREATE TABLE `sample` (`id` bigint AUTO_INCREMENT) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4',
);
});
test('规范化保留例程中的自增文本并拒绝主机信息', () => {
const routine = normalizeCreateDefinition(
"CREATE PROCEDURE `sample`() SELECT ' AUTO_INCREMENT=123 DEFAULT CHARSET=utf8mb4'",
);
assert.equal(
routine,
"CREATE PROCEDURE `sample`() SELECT ' AUTO_INCREMENT=123 DEFAULT CHARSET=utf8mb4'",
);
assert.throws(
() => normalizeCreateDefinition('CREATE TABLE `sample` (`id` int) /* Host: prod-db.internal */'),
(error) => error.code === 'UNSAFE_DEFINITION',
);
});
test('完整导出双重读取一致并始终关闭连接', async () => {
const connection = createExportConnection();
let receivedOptions;
const result = await exportMysql57Schema({
config,
createConnection: async (options) => {
receivedOptions = options;
return connection;
},
now: () => 100,
});
assert.equal(receivedOptions.multipleStatements, false);
assert.equal(connection.endCalls, 1);
assert.deepEqual(result.objectCounts, {
tables: 1,
views: 2,
triggers: 1,
routines: 2,
events: 1,
});
assert.equal(result.durationMs, 0);
assert.equal(result.metadata.driverName, 'mysql2');
});
test('导出拒绝目录变化和定义变化', async () => {
const catalogConnection = createExportConnection({
mutateCatalog(sql, rows, round) {
if (round === 1 && sql.includes('information_schema.TABLES')) {
return [...rows, { TABLE_NAME: 'new_table', TABLE_TYPE: 'BASE TABLE' }];
}
return rows;
},
});
await assert.rejects(
() =>
exportMysql57Schema({
config,
createConnection: async () => catalogConnection,
now: () => 100,
}),
(error) => error.code === 'CATALOG_CHANGED_DURING_EXPORT',
);
assert.equal(catalogConnection.endCalls, 1);
const definitionConnection = createExportConnection({
mutateDefinition(sql, row, round) {
if (round === 1 && sql.includes('SHOW CREATE TABLE')) {
return [{ ...row, 'Create Table': `${row['Create Table']} COMMENT='changed'` }];
}
return undefined;
},
});
await assert.rejects(
() =>
exportMysql57Schema({
config,
createConnection: async () => definitionConnection,
now: () => 100,
}),
(error) => error.code === 'DEFINITION_CHANGED_DURING_EXPORT',
);
assert.equal(definitionConnection.endCalls, 1);
});
test('SHOW CREATE 拒绝空结果、错误形状和过大定义', async () => {
for (const [changed, expectedCode] of [
[[], 'SHOW_CREATE_EMPTY'],
[[{ unexpected: 'value' }], 'SHOW_CREATE_SHAPE_INVALID'],
]) {
const connection = createExportConnection({
mutateDefinition(sql) {
return sql.includes('SHOW CREATE TABLE') ? changed : undefined;
},
});
await assert.rejects(
() => readDefinitions(connection, completeCatalog, DEFAULT_EXPORT_LIMITS),
(error) => error.code === expectedCode,
);
}
const oversized = createExportConnection({
mutateDefinition(sql, row) {
return sql.includes('SHOW CREATE TABLE')
? [{ ...row, 'Create Table': 'x'.repeat(100) }]
: undefined;
},
});
await assert.rejects(
() =>
readDefinitions(oversized, completeCatalog, {
...DEFAULT_EXPORT_LIMITS,
maxDefinitionBytes: 32,
}),
(error) => error.code === 'DEFINITION_LIMIT_EXCEEDED',
);
});
test('直接元数据和 SHOW CREATE 查询错误统一脱敏且保留既有业务错误', async () => {
const rawFailure = new Error('host=prod-db.internal password=not-for-logs');
await assert.rejects(
() => readMysql57ServerMetadata({ query: async () => { throw rawFailure; } }, 'a_share_analysis_dev'),
(error) =>
error.code === 'DATABASE_QUERY_FAILED' &&
!error.message.includes('prod-db.internal') &&
!error.message.includes('not-for-logs'),
);
await assert.rejects(
() => readDefinitions({ query: async () => { throw rawFailure; } }, completeCatalog),
(error) =>
error.code === 'DATABASE_QUERY_FAILED' &&
!error.message.includes('prod-db.internal') &&
!error.message.includes('not-for-logs'),
);
const timeout = new SchemaExportError('EXPORT_TIMEOUT', 'Schema 导出超过总时长限制');
await assert.rejects(
() => readDefinitions({ query: async () => { throw timeout; } }, completeCatalog),
(error) => error === timeout,
);
});
test('查询失败和总超时只返回脱敏错误并关闭连接', async () => {
const failed = createExportConnection({
queryError: new Error('connect 127.0.0.1 password=not-for-logs'),
});
await assert.rejects(
() =>
exportMysql57Schema({
config,
createConnection: async () => failed,
now: () => 100,
}),
(error) =>
error.code === 'DATABASE_QUERY_FAILED' &&
!error.message.includes('127.0.0.1') &&
!error.message.includes('not-for-logs'),
);
assert.equal(failed.endCalls, 1);
const timedOut = createExportConnection();
let tick = 0;
await assert.rejects(
() =>
exportMysql57Schema({
config,
createConnection: async () => timedOut,
now: () => (tick++ === 0 ? 0 : 30_001),
}),
(error) => error.code === 'EXPORT_TIMEOUT',
);
assert.equal(timedOut.endCalls, 1);
});
+256 -110
View File
@@ -12,12 +12,20 @@ import { createHash } from 'node:crypto';
import path from 'node:path'; import path from 'node:path';
import { fileURLToPath } from 'node:url'; import { fileURLToPath } from 'node:url';
import {
MYSQL2_EXPORTER_METADATA,
assertMysql57ServerMetadata as assertMysql57ServerMetadataV3,
exportMysql57Schema,
} from './mysql2-schema-exporter.mjs';
const DEVELOPMENT_DATABASE = 'a_share_analysis_dev'; const DEVELOPMENT_DATABASE = 'a_share_analysis_dev';
const REQUIRED_CHARSET = 'utf8mb4'; const REQUIRED_CHARSET = 'utf8mb4';
const REQUIRED_COLLATION = 'utf8mb4_general_ci'; const REQUIRED_COLLATION = 'utf8mb4_general_ci';
const BASELINE_FORMAT_VERSION = 2; const BASELINE_FORMAT_VERSION = 3;
const PREVIOUS_BASELINE_FORMAT_VERSION = 2;
const LEGACY_BASELINE_FORMAT_VERSION = 1; const LEGACY_BASELINE_FORMAT_VERSION = 1;
const TOOL_VERSION = '2.0.0'; const TOOL_VERSION = '3.0.0';
const PREVIOUS_TOOL_VERSION = '2.0.0';
const LEGACY_TOOL_VERSION = '1.0.0'; const LEGACY_TOOL_VERSION = '1.0.0';
const VERSION_PATTERN = /^v\d+\.\d+\.\d+$/; const VERSION_PATTERN = /^v\d+\.\d+\.\d+$/;
const DEVELOPMENT_HOSTS = new Set(['127.0.0.1', 'localhost', '::1', '112.192.20.11']); const DEVELOPMENT_HOSTS = new Set(['127.0.0.1', 'localhost', '::1', '112.192.20.11']);
@@ -283,54 +291,108 @@ function countSchemaObjects(sql) {
function buildManifest(fields) { function buildManifest(fields) {
// JSON 是 YAML 1.2 的严格子集;使用标准解析器生成和读取,避免手写 YAML // JSON 是 YAML 1.2 的严格子集;使用标准解析器生成和读取,避免手写 YAML
// 对缩进、引号和同名嵌套字段产生歧义,同时仍保留约定的 manifest.yaml 文件名。 // 对缩进、引号和同名嵌套字段产生歧义,同时仍保留约定的 manifest.yaml 文件名。
return `${JSON.stringify( const common = {
{ format_version: fields.formatVersion,
format_version: BASELINE_FORMAT_VERSION, status: fields.status,
status: fields.status, application_version: fields.applicationVersion,
application_version: fields.applicationVersion, git_commit: fields.gitCommit,
git_commit: fields.gitCommit, source_environment: 'development',
source_environment: 'development', database_name: DEVELOPMENT_DATABASE,
database_name: DEVELOPMENT_DATABASE, charset: REQUIRED_CHARSET,
mysql_client_version: fields.mysqlClientVersion, collation: REQUIRED_COLLATION,
mysqldump_client_version: fields.mysqldumpClientVersion, generated_at: fields.generatedAt,
mysql_server_version: fields.serverVersion, generator: `schema-baseline/${fields.formatVersion === 3 ? TOOL_VERSION : fields.formatVersion === 2 ? PREVIOUS_TOOL_VERSION : LEGACY_TOOL_VERSION}`,
charset: REQUIRED_CHARSET, reason: fields.reason ?? 'none',
collation: REQUIRED_COLLATION, schema_sha256: fields.schemaSha256 ?? 'not_applicable',
generated_at: fields.generatedAt, objects: {
generator: `schema-baseline/${TOOL_VERSION}`, tables: fields.captured,
reason: fields.reason ?? 'none', views: fields.captured,
schema_sha256: fields.schemaSha256 ?? 'not_applicable', triggers: fields.captured,
objects: { routines: fields.captured,
tables: fields.captured, events: fields.captured,
views: fields.captured,
triggers: fields.captured,
routines: fields.captured,
events: fields.captured,
},
object_counts: fields.objectCounts,
}, },
null, object_counts: fields.objectCounts,
2, };
)}\n`; if (fields.formatVersion === 3) {
common.exporter_name = fields.exporterName;
common.exporter_version = fields.exporterVersion;
common.database_driver = fields.driverName;
common.database_driver_version = fields.driverVersion;
common.mysql_server_version = fields.serverVersion;
} else {
common.mysql_client_version = fields.mysqlClientVersion;
common.mysqldump_client_version = fields.mysqldumpClientVersion;
common.mysql_server_version = fields.serverVersion;
}
return `${JSON.stringify(common, null, 2)}\n`;
} }
export function buildCapturedArtifacts(metadata, sql) { function assertNodeMetadata(metadata) {
if (metadata.formatVersion !== 3) {
throw new Error('Node Schema 基线必须使用 format v3');
}
for (const key of ['exporterName', 'exporterVersion', 'driverName', 'driverVersion']) {
if (!String(metadata[key] ?? '').trim()) {
throw new Error(`Node Schema 元数据缺少 ${key}`);
}
}
assertMysql57ServerMetadataV3(metadata);
}
export function buildCapturedArtifacts(metadata, sql, objectCounts) {
if (metadata.formatVersion === 3) {
assertNodeMetadata(metadata);
if (!objectCounts || Object.keys(objectCounts).sort().join(',') !== [...OBJECT_KEYS].sort().join(',')) {
throw new Error('Node Schema 对象计数字段无效');
}
const normalizedSql = normalizeSchemaSql(sql);
const derivedCounts = countSchemaObjects(normalizedSql);
for (const key of OBJECT_KEYS) {
if (!Number.isInteger(objectCounts[key]) || objectCounts[key] < 0 || objectCounts[key] !== derivedCounts[key]) {
throw new Error(`Node Schema 对象计数与 SQL 不一致:${key}`);
}
}
if (String(sql).trim() && Object.values(objectCounts).every((count) => count === 0)) {
throw new Error('Node Schema SQL 非空但对象计数全为零');
}
if (Object.values(objectCounts).every((count) => count === 0)) {
return buildCapturedEmptyArtifacts(metadata);
}
const schemaSha256 = sha256(normalizedSql);
const manifest = buildManifest({
...metadata,
status: 'captured',
captured: true,
objectCounts,
schemaSha256,
});
return Object.freeze({
status: 'captured',
schemaSha256,
files: Object.freeze({
'manifest.yaml': manifest,
'schema.sql': normalizedSql,
'checksum.sha256': `${schemaSha256} schema.sql\n`,
}),
});
}
assertMysql57Metadata(metadata); assertMysql57Metadata(metadata);
if (!VERSION_PATTERN.test(metadata.applicationVersion ?? '')) { if (!VERSION_PATTERN.test(metadata.applicationVersion ?? '')) {
throw new Error(`应用版本格式无效:${metadata.applicationVersion || '(空)'}`); throw new Error(`应用版本格式无效:${metadata.applicationVersion || '(空)'}`);
} }
const normalizedSql = normalizeSchemaSql(sql); const normalizedSql = normalizeSchemaSql(sql);
const objectCounts = countSchemaObjects(normalizedSql); const derivedCounts = countSchemaObjects(normalizedSql);
// mysqldump 对空库仍会输出会话 SET 与头尾注释;结构对象计数为零才是空库判据。 // mysqldump 对空库仍会输出会话 SET 与头尾注释;结构对象计数为零才是空库判据。
if (Object.values(objectCounts).every((count) => count === 0)) { if (Object.values(derivedCounts).every((count) => count === 0)) {
return buildCapturedEmptyArtifacts(metadata); return buildCapturedEmptyArtifacts(metadata);
} }
const schemaSha256 = sha256(normalizedSql); const schemaSha256 = sha256(normalizedSql);
const manifest = buildManifest({ const manifest = buildManifest({
...metadata, ...metadata,
formatVersion: metadata.formatVersion ?? PREVIOUS_BASELINE_FORMAT_VERSION,
status: 'captured', status: 'captured',
captured: true, captured: true,
objectCounts, objectCounts: derivedCounts,
schemaSha256, schemaSha256,
}); });
return Object.freeze({ return Object.freeze({
@@ -345,6 +407,23 @@ export function buildCapturedArtifacts(metadata, sql) {
} }
export function buildCapturedEmptyArtifacts(metadata) { export function buildCapturedEmptyArtifacts(metadata) {
if (metadata.formatVersion === 3) {
assertNodeMetadata(metadata);
const schemaSha256 = sha256('');
const objectCounts = Object.freeze(Object.fromEntries(OBJECT_KEYS.map((key) => [key, 0])));
const manifest = buildManifest({
...metadata,
status: 'captured_empty',
captured: false,
objectCounts,
schemaSha256,
});
return Object.freeze({
status: 'captured_empty',
schemaSha256,
files: Object.freeze({ 'manifest.yaml': manifest }),
});
}
assertMysql57Metadata(metadata); assertMysql57Metadata(metadata);
if (!VERSION_PATTERN.test(metadata.applicationVersion ?? '')) { if (!VERSION_PATTERN.test(metadata.applicationVersion ?? '')) {
throw new Error(`应用版本格式无效:${metadata.applicationVersion || '(空)'}`); throw new Error(`应用版本格式无效:${metadata.applicationVersion || '(空)'}`);
@@ -353,6 +432,7 @@ export function buildCapturedEmptyArtifacts(metadata) {
const objectCounts = Object.freeze(Object.fromEntries(OBJECT_KEYS.map((key) => [key, 0]))); const objectCounts = Object.freeze(Object.fromEntries(OBJECT_KEYS.map((key) => [key, 0])));
const manifest = buildManifest({ const manifest = buildManifest({
...metadata, ...metadata,
formatVersion: metadata.formatVersion ?? PREVIOUS_BASELINE_FORMAT_VERSION,
status: 'captured_empty', status: 'captured_empty',
captured: false, captured: false,
objectCounts, objectCounts,
@@ -369,8 +449,14 @@ export function buildNotApplicableArtifacts(metadata) {
if (!VERSION_PATTERN.test(metadata.applicationVersion ?? '')) { if (!VERSION_PATTERN.test(metadata.applicationVersion ?? '')) {
throw new Error(`应用版本格式无效:${metadata.applicationVersion || '(空)'}`); throw new Error(`应用版本格式无效:${metadata.applicationVersion || '(空)'}`);
} }
const manifest = buildManifest({ const nodeMetadata = metadata.formatVersion === 3 ? metadata : {
...metadata, ...metadata,
formatVersion: PREVIOUS_BASELINE_FORMAT_VERSION,
};
const manifest = buildManifest({
...nodeMetadata,
...metadata,
formatVersion: nodeMetadata.formatVersion,
mysqlClientVersion: 'not_enabled', mysqlClientVersion: 'not_enabled',
mysqldumpClientVersion: 'not_enabled', mysqldumpClientVersion: 'not_enabled',
serverVersion: 'not_enabled', serverVersion: 'not_enabled',
@@ -403,28 +489,65 @@ function compareExistingArtifacts(targetDirectory, artifacts) {
* 先在目标父目录写入临时目录,再原子改名。这样异常不会留下半份基线; * 先在目标父目录写入临时目录,再原子改名。这样异常不会留下半份基线;
* 已发布版本目录只允许逐字节相同的幂等重跑,任何差异都必须人工留痕修正。 * 已发布版本目录只允许逐字节相同的幂等重跑,任何差异都必须人工留痕修正。
*/ */
export function persistBaselineArtifacts(targetDirectory, artifacts) { const DEFAULT_FILE_OPERATIONS = Object.freeze({
if (existsSync(targetDirectory)) { existsSync,
if (compareExistingArtifacts(targetDirectory, artifacts)) { mkdirSync,
readFileSync,
readdirSync,
renameSync,
rmSync,
writeFileSync,
});
export function persistBaselineArtifacts(
targetDirectory,
artifacts,
fileOperations = DEFAULT_FILE_OPERATIONS,
) {
const operations = { ...DEFAULT_FILE_OPERATIONS, ...fileOperations };
const {
existsSync: fileExists,
mkdirSync: makeDirectory,
readFileSync: readFile,
readdirSync: readDirectory,
renameSync: renameFile,
rmSync: removePath,
writeFileSync: writeFile,
} = operations;
const compare = () => {
const actualFiles = readDirectory(targetDirectory).sort();
const expectedFiles = Object.keys(artifacts.files).sort();
if (
actualFiles.length !== expectedFiles.length ||
actualFiles.some((file, index) => file !== expectedFiles[index])
) {
return false;
}
return expectedFiles.every(
(file) => readFile(path.join(targetDirectory, file), 'utf8') === artifacts.files[file],
);
};
if (fileExists(targetDirectory)) {
if (compare()) {
return { reused: true }; return { reused: true };
} }
throw new Error(`目标版本基线已存在且内容不同,禁止覆盖:${targetDirectory}`); throw new Error(`目标版本基线已存在且内容不同,禁止覆盖:${targetDirectory}`);
} }
const parentDirectory = path.dirname(targetDirectory); const parentDirectory = path.dirname(targetDirectory);
mkdirSync(parentDirectory, { recursive: true }); makeDirectory(parentDirectory, { recursive: true });
const temporaryDirectory = path.join( const temporaryDirectory = path.join(
parentDirectory, parentDirectory,
`.${path.basename(targetDirectory)}.tmp-${process.pid}-${Date.now()}`, `.${path.basename(targetDirectory)}.tmp-${process.pid}-${Date.now()}`,
); );
mkdirSync(temporaryDirectory); makeDirectory(temporaryDirectory);
try { try {
for (const [file, content] of Object.entries(artifacts.files)) { for (const [file, content] of Object.entries(artifacts.files)) {
writeFileSync(path.join(temporaryDirectory, file), content, 'utf8'); writeFile(path.join(temporaryDirectory, file), content, 'utf8');
} }
renameSync(temporaryDirectory, targetDirectory); renameFile(temporaryDirectory, targetDirectory);
} catch (error) { } catch (error) {
rmSync(temporaryDirectory, { recursive: true, force: true }); removePath(temporaryDirectory, { recursive: true, force: true });
throw error; throw error;
} }
return { reused: false }; return { reused: false };
@@ -468,6 +591,8 @@ export function verifyBaselineDirectory(targetDirectory, expectedRelease, expect
const supportedFormat = const supportedFormat =
(manifest.format_version === LEGACY_BASELINE_FORMAT_VERSION && (manifest.format_version === LEGACY_BASELINE_FORMAT_VERSION &&
manifest.generator === `schema-baseline/${LEGACY_TOOL_VERSION}`) || manifest.generator === `schema-baseline/${LEGACY_TOOL_VERSION}`) ||
(manifest.format_version === PREVIOUS_BASELINE_FORMAT_VERSION &&
manifest.generator === `schema-baseline/${PREVIOUS_TOOL_VERSION}`) ||
(manifest.format_version === BASELINE_FORMAT_VERSION && (manifest.format_version === BASELINE_FORMAT_VERSION &&
manifest.generator === `schema-baseline/${TOOL_VERSION}`); manifest.generator === `schema-baseline/${TOOL_VERSION}`);
if ( if (
@@ -506,30 +631,49 @@ export function verifyBaselineDirectory(targetDirectory, expectedRelease, expect
assertExactFileSet(targetDirectory, NOT_APPLICABLE_FILES); assertExactFileSet(targetDirectory, NOT_APPLICABLE_FILES);
const reason = manifest.reason; const reason = manifest.reason;
requireExactValue(reason, 'mysql_disabled', 'Schema 不适用原因无效'); requireExactValue(reason, 'mysql_disabled', 'Schema 不适用原因无效');
if ( const commonInvalid =
manifest.mysql_client_version !== 'not_enabled' ||
manifest.mysqldump_client_version !== 'not_enabled' ||
manifest.mysql_server_version !== 'not_enabled' || manifest.mysql_server_version !== 'not_enabled' ||
manifest.schema_sha256 !== 'not_applicable' || manifest.schema_sha256 !== 'not_applicable' ||
Object.keys(manifest.objects ?? {}).sort().join(',') !== [...OBJECT_KEYS].sort().join(',') || Object.keys(manifest.objects ?? {}).sort().join(',') !==
OBJECT_KEYS.some((key) => manifest.objects[key] !== false) [...OBJECT_KEYS].sort().join(',') ||
) { OBJECT_KEYS.some((key) => manifest.objects[key] !== false);
const formatInvalid =
manifest.format_version === BASELINE_FORMAT_VERSION
? !String(manifest.exporter_name ?? '').trim() ||
!String(manifest.exporter_version ?? '').trim() ||
!String(manifest.database_driver ?? '').trim() ||
!String(manifest.database_driver_version ?? '').trim() ||
'mysql_client_version' in manifest ||
'mysqldump_client_version' in manifest
: manifest.mysql_client_version !== 'not_enabled' ||
manifest.mysqldump_client_version !== 'not_enabled';
if (commonInvalid || formatInvalid) {
throw new Error('Schema 不适用清单必须证明 MySQL 未启用且所有对象均未捕获'); throw new Error('Schema 不适用清单必须证明 MySQL 未启用且所有对象均未捕获');
} }
return { status, applicationVersion, reason }; return { status, applicationVersion, reason };
} }
if (status === 'captured_empty') { if (status === 'captured_empty') {
if (manifest.format_version !== BASELINE_FORMAT_VERSION) {
throw new Error('空库 Schema 基线只允许 format v2');
}
assertExactFileSet(targetDirectory, CAPTURED_EMPTY_FILES); assertExactFileSet(targetDirectory, CAPTURED_EMPTY_FILES);
assertMysql57Metadata({ if (manifest.format_version === BASELINE_FORMAT_VERSION) {
mysqlClientVersion: manifest.mysql_client_version, assertNodeMetadata({
mysqldumpClientVersion: manifest.mysqldump_client_version, formatVersion: 3,
serverVersion: manifest.mysql_server_version, exporterName: manifest.exporter_name,
databaseCharset: manifest.charset, exporterVersion: manifest.exporter_version,
databaseCollation: manifest.collation, driverName: manifest.database_driver,
}); driverVersion: manifest.database_driver_version,
serverVersion: manifest.mysql_server_version,
databaseCharset: manifest.charset,
databaseCollation: manifest.collation,
});
} else {
assertMysql57Metadata({
mysqlClientVersion: manifest.mysql_client_version,
mysqldumpClientVersion: manifest.mysqldump_client_version,
serverVersion: manifest.mysql_server_version,
databaseCharset: manifest.charset,
databaseCollation: manifest.collation,
});
}
const expectedEmptyChecksum = sha256(''); const expectedEmptyChecksum = sha256('');
if ( if (
manifest.schema_sha256 !== expectedEmptyChecksum || manifest.schema_sha256 !== expectedEmptyChecksum ||
@@ -546,13 +690,26 @@ export function verifyBaselineDirectory(targetDirectory, expectedRelease, expect
} }
assertExactFileSet(targetDirectory, CAPTURED_FILES); assertExactFileSet(targetDirectory, CAPTURED_FILES);
assertMysql57Metadata({ if (manifest.format_version === BASELINE_FORMAT_VERSION) {
mysqlClientVersion: manifest.mysql_client_version, assertNodeMetadata({
mysqldumpClientVersion: manifest.mysqldump_client_version, formatVersion: 3,
serverVersion: manifest.mysql_server_version, exporterName: manifest.exporter_name,
databaseCharset: manifest.charset, exporterVersion: manifest.exporter_version,
databaseCollation: manifest.collation, driverName: manifest.database_driver,
}); driverVersion: manifest.database_driver_version,
serverVersion: manifest.mysql_server_version,
databaseCharset: manifest.charset,
databaseCollation: manifest.collation,
});
} else {
assertMysql57Metadata({
mysqlClientVersion: manifest.mysql_client_version,
mysqldumpClientVersion: manifest.mysqldump_client_version,
serverVersion: manifest.mysql_server_version,
databaseCharset: manifest.charset,
databaseCollation: manifest.collation,
});
}
if (Object.keys(manifest.objects ?? {}).sort().join(',') !== [...OBJECT_KEYS].sort().join(',')) { if (Object.keys(manifest.objects ?? {}).sort().join(',') !== [...OBJECT_KEYS].sort().join(',')) {
throw new Error('Schema 对象范围字段无效'); throw new Error('Schema 对象范围字段无效');
} }
@@ -583,6 +740,14 @@ export function verifyBaselineDirectory(targetDirectory, expectedRelease, expect
if (manifestChecksum !== schemaSha256 || checksumFile !== `${schemaSha256} schema.sql\n`) { if (manifestChecksum !== schemaSha256 || checksumFile !== `${schemaSha256} schema.sql\n`) {
throw new Error('Schema SQL、manifest 和 checksum 校验值不一致'); throw new Error('Schema SQL、manifest 和 checksum 校验值不一致');
} }
if (manifest.format_version === BASELINE_FORMAT_VERSION) {
const derivedCounts = countSchemaObjects(sql);
for (const key of OBJECT_KEYS) {
if (manifest.object_counts[key] !== derivedCounts[key]) {
throw new Error(`Schema 对象计数与 SQL 不一致:${key}`);
}
}
}
return { status, applicationVersion, schemaSha256 }; return { status, applicationVersion, schemaSha256 };
} }
@@ -614,7 +779,7 @@ function extractClientVersion(output) {
* 真实执行器可注入是为了让测试证明“关闭时零数据库调用”和命令边界;默认执行器 * 真实执行器可注入是为了让测试证明“关闭时零数据库调用”和命令边界;默认执行器
* 仍使用 execFileSync 参数数组。密码仅放入子进程环境 MYSQL_PWD,不会进入 argv。 * 仍使用 execFileSync 参数数组。密码仅放入子进程环境 MYSQL_PWD,不会进入 argv。
*/ */
export function runBaselineGeneration(options) { export async function runBaselineGeneration(options) {
if ((options.commandArguments ?? []).length > 0) { if ((options.commandArguments ?? []).length > 0) {
throw new Error('Schema 基线命令不接受命令参数,不能覆盖环境、数据库或 env 路径'); throw new Error('Schema 基线命令不接受命令参数,不能覆盖环境、数据库或 env 路径');
} }
@@ -641,52 +806,33 @@ export function runBaselineGeneration(options) {
generatedAt: options.generatedAt ?? new Date().toISOString(), generatedAt: options.generatedAt ?? new Date().toISOString(),
}; };
if (!config.MYSQL_ENABLED) { if (!config.MYSQL_ENABLED) {
const artifacts = buildNotApplicableArtifacts(commonMetadata); const artifacts = buildNotApplicableArtifacts({
persistBaselineArtifacts(targetDirectory, artifacts); ...commonMetadata,
formatVersion: BASELINE_FORMAT_VERSION,
exporterName: MYSQL2_EXPORTER_METADATA.name,
exporterVersion: MYSQL2_EXPORTER_METADATA.version,
driverName: MYSQL2_EXPORTER_METADATA.driverName,
driverVersion: MYSQL2_EXPORTER_METADATA.driverVersion,
serverVersion: 'not_enabled',
databaseCharset: REQUIRED_CHARSET,
databaseCollation: REQUIRED_COLLATION,
});
persistBaselineArtifacts(targetDirectory, artifacts, options.fileOperations);
return verifyBaselineDirectory(targetDirectory, options.applicationVersion); return verifyBaselineDirectory(targetDirectory, options.applicationVersion);
} }
const executeFile = options.executeFile ?? execFileSync; const exported = await (options.exportSchema ?? exportMysql57Schema)({ config });
const childEnvironment = { ...process.env, MYSQL_PWD: config.MYSQL_PASSWORD ?? '' }; const exportedMetadata = exported.metadata ?? {};
const executionOptions = {
cwd: options.repositoryRoot,
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe'],
env: childEnvironment,
};
const mysqlClientVersion = extractClientVersion(
executeFile('mysql', ['--version'], executionOptions),
);
const mysqldumpClientVersion = extractClientVersion(
executeFile('mysqldump', ['--version'], executionOptions),
);
assertMysql57Metadata({
mysqlClientVersion,
mysqldumpClientVersion,
serverVersion: '5.7.0-pending-server-check',
databaseCharset: REQUIRED_CHARSET,
databaseCollation: REQUIRED_COLLATION,
});
const serverMetadata = String(
executeFile('mysql', buildMysqlMetadataArguments(config), executionOptions),
)
.trim()
.split('\t');
if (serverMetadata.length !== 3) {
throw new Error('无法从固定开发库读取 MySQL 版本、字符集和排序规则');
}
const metadata = { const metadata = {
...commonMetadata, ...commonMetadata,
mysqlClientVersion, ...exportedMetadata,
mysqldumpClientVersion, applicationVersion: exportedMetadata.applicationVersion ?? commonMetadata.applicationVersion,
serverVersion: serverMetadata[0], gitCommit: exportedMetadata.gitCommit ?? commonMetadata.gitCommit,
databaseCharset: serverMetadata[1], generatedAt: exportedMetadata.generatedAt ?? commonMetadata.generatedAt,
databaseCollation: serverMetadata[2], formatVersion: BASELINE_FORMAT_VERSION,
}; };
assertMysql57Metadata(metadata); const artifacts = buildCapturedArtifacts(metadata, exported.rawSql, exported.objectCounts);
const rawSql = executeFile('mysqldump', buildMysqldumpArguments(config), executionOptions); persistBaselineArtifacts(targetDirectory, artifacts, options.fileOperations);
const artifacts = buildCapturedArtifacts(metadata, rawSql);
persistBaselineArtifacts(targetDirectory, artifacts);
return verifyBaselineDirectory(targetDirectory, options.applicationVersion); return verifyBaselineDirectory(targetDirectory, options.applicationVersion);
} }
@@ -815,7 +961,7 @@ export function resolveFixedDevelopmentEnv(repositoryRoot) {
return path.join(repositoryRoot, 'deploy', 'env', 'development', 'api.env.example'); return path.join(repositoryRoot, 'deploy', 'env', 'development', 'api.env.example');
} }
function runCommandLine() { export async function runCommandLine() {
const [command, ...commandArguments] = process.argv.slice(2); const [command, ...commandArguments] = process.argv.slice(2);
if (!['generate', 'verify'].includes(command) || commandArguments.length > 0) { if (!['generate', 'verify'].includes(command) || commandArguments.length > 0) {
throw new Error('用法:node tools/database/schema-baseline.mjs <generate|verify>(不接受额外参数)'); throw new Error('用法:node tools/database/schema-baseline.mjs <generate|verify>(不接受额外参数)');
@@ -824,7 +970,7 @@ function runCommandLine() {
const release = getCurrentReleasedVersion(repositoryRoot); const release = getCurrentReleasedVersion(repositoryRoot);
const result = const result =
command === 'generate' command === 'generate'
? runBaselineGeneration({ ? await runBaselineGeneration({
repositoryRoot, repositoryRoot,
envFile: resolveFixedDevelopmentEnv(repositoryRoot), envFile: resolveFixedDevelopmentEnv(repositoryRoot),
...release, ...release,
@@ -837,5 +983,5 @@ function runCommandLine() {
} }
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
runCommandLine(); await runCommandLine();
} }
+138 -37
View File
@@ -1,6 +1,6 @@
import assert from 'node:assert/strict'; import assert from 'node:assert/strict';
import { createHash } from 'node:crypto'; import { createHash } from 'node:crypto';
import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; import { mkdirSync, mkdtempSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os'; import { tmpdir } from 'node:os';
import path from 'node:path'; import path from 'node:path';
import test from 'node:test'; import test from 'node:test';
@@ -45,6 +45,20 @@ const validMetadata = Object.freeze({
generatedAt: '2026-08-13T03:00:00.000Z', generatedAt: '2026-08-13T03:00:00.000Z',
}); });
const validNodeMetadata = Object.freeze({
formatVersion: 3,
applicationVersion: 'v0.0.9',
gitCommit: '88e2fdffc8c71dd682405c148a81953a8dcee9d6',
exporterName: 'mysql2-show-create',
exporterVersion: '1.0.0',
driverName: 'mysql2',
driverVersion: '3.23.3',
serverVersion: '5.7.42-log',
databaseCharset: 'utf8mb4',
databaseCollation: 'utf8mb4_general_ci',
generatedAt: '2026-08-18T00:00:00.000Z',
});
function makeTemporaryDirectory() { function makeTemporaryDirectory() {
return mkdtempSync(path.join(tmpdir(), 'a-share-schema-baseline-')); return mkdtempSync(path.join(tmpdir(), 'a-share-schema-baseline-'));
} }
@@ -300,6 +314,29 @@ test('拒绝静默覆盖已存在且内容不同的版本基线', () => {
); );
}); });
test('写入第三个文件失败时清理目标目录和临时目录', () => {
const root = makeTemporaryDirectory();
const target = path.join(root, 'v0.0.5');
const artifacts = buildCapturedArtifacts(
validMetadata,
'CREATE TABLE `sample` (`id` int);\n',
);
let writeCount = 0;
assert.throws(
() =>
persistBaselineArtifacts(target, artifacts, {
writeFileSync(...args) {
writeCount += 1;
if (writeCount === 3) throw new Error('injected write failure');
return writeFileSync(...args);
},
}),
/injected write failure/,
);
assert.equal(readdirSync(root).includes('v0.0.5'), false);
assert.equal(readdirSync(root).some((entry) => entry.includes('.tmp-')), false);
});
test('MySQL 未启用时只生成可验证不适用清单', () => { test('MySQL 未启用时只生成可验证不适用清单', () => {
const root = makeTemporaryDirectory(); const root = makeTemporaryDirectory();
const target = path.join(root, 'v0.0.5'); const target = path.join(root, 'v0.0.5');
@@ -314,7 +351,7 @@ test('MySQL 未启用时只生成可验证不适用清单', () => {
assert.throws(() => readFileSync(path.join(target, 'checksum.sha256')), /ENOENT/); assert.throws(() => readFileSync(path.join(target, 'checksum.sha256')), /ENOENT/);
}); });
test('未启用入口不调用数据库且拒绝额外命令参数', () => { test('未启用入口不调用数据库且拒绝额外命令参数', async () => {
const root = makeTemporaryDirectory(); const root = makeTemporaryDirectory();
const envFile = path.join(root, 'development.env'); const envFile = path.join(root, 'development.env');
writeFileSync( writeFileSync(
@@ -323,7 +360,7 @@ test('未启用入口不调用数据库且拒绝额外命令参数', () => {
); );
let executionCount = 0; let executionCount = 0;
const result = runBaselineGeneration({ const result = await runBaselineGeneration({
repositoryRoot: root, repositoryRoot: root,
envFile, envFile,
applicationVersion: 'v0.0.5', applicationVersion: 'v0.0.5',
@@ -337,13 +374,13 @@ test('未启用入口不调用数据库且拒绝额外命令参数', () => {
assert.equal(result.status, 'not_applicable'); assert.equal(result.status, 'not_applicable');
assert.equal(executionCount, 0); assert.equal(executionCount, 0);
assert.throws( await assert.rejects(
() => runBaselineGeneration({ ...result, commandArguments: ['--env', 'production.env'] }), () => runBaselineGeneration({ ...result, commandArguments: ['--env', 'production.env'] }),
/不接受命令参数/, /不接受命令参数/,
); );
}); });
test('启用入口分别校验 mysql 与 mysqldump 客户端版本', () => { test('启用入口使用异步 Node exporter', async () => {
const root = makeTemporaryDirectory(); const root = makeTemporaryDirectory();
const envFile = path.join(root, 'development.env'); const envFile = path.join(root, 'development.env');
writeFileSync( writeFileSync(
@@ -353,31 +390,27 @@ test('启用入口分别校验 mysql 与 mysqldump 客户端版本', () => {
.join('\n'), .join('\n'),
); );
assert.throws( const result = await runBaselineGeneration({
() =>
runBaselineGeneration({
repositoryRoot: root, repositoryRoot: root,
envFile, envFile,
applicationVersion: 'v0.0.5', applicationVersion: 'v0.0.5',
gitCommit: validMetadata.gitCommit, gitCommit: validMetadata.gitCommit,
generatedAt: validMetadata.generatedAt, generatedAt: validMetadata.generatedAt,
executeFile(command, args) { exportSchema: async () => ({
if (args[0] === '--version') { metadata: {
return command === 'mysql' ...validNodeMetadata,
? 'mysql Ver 8.0.39 for macos' applicationVersion: undefined,
: 'mysqldump Ver 10.13 Distrib 5.7.44, for macos'; gitCommit: undefined,
} generatedAt: undefined,
if (command === 'mysql') { },
return '5.7.42-log\tutf8mb4\tutf8mb4_general_ci\n'; rawSql: '',
} objectCounts: { tables: 0, views: 0, triggers: 0, routines: 0, events: 0 },
return 'CREATE TABLE `sample` (`id` int);\n'; }),
}, });
}), assert.equal(result.status, 'captured_empty');
/mysql 客户端必须为 5\.7/,
);
}); });
test('幂等重跑仍核对清单提交与发布标签提交', () => { test('幂等重跑仍核对清单提交与发布标签提交', async () => {
const root = makeTemporaryDirectory(); const root = makeTemporaryDirectory();
const envFile = path.join(root, 'development.env'); const envFile = path.join(root, 'development.env');
writeFileSync( writeFileSync(
@@ -387,9 +420,8 @@ test('幂等重跑仍核对清单提交与发布标签提交', () => {
const target = path.join(root, 'database', 'schema-baselines', 'v0.0.5'); const target = path.join(root, 'database', 'schema-baselines', 'v0.0.5');
persistBaselineArtifacts(target, buildNotApplicableArtifacts(validMetadata)); persistBaselineArtifacts(target, buildNotApplicableArtifacts(validMetadata));
assert.throws( await assert.rejects(
() => () => runBaselineGeneration({
runBaselineGeneration({
repositoryRoot: root, repositoryRoot: root,
envFile, envFile,
applicationVersion: 'v0.0.5', applicationVersion: 'v0.0.5',
@@ -451,7 +483,7 @@ test('仅含 mysqldump 会话设置的真实空库仍生成 captured_empty', ()
assert.deepEqual(Object.keys(artifacts.files), ['manifest.yaml']); assert.deepEqual(Object.keys(artifacts.files), ['manifest.yaml']);
}); });
test('启用入口把 mysqldump 空结构持久化为 captured_empty', () => { test('启用入口把 Node exporter 空结构持久化为 captured_empty', async () => {
const root = makeTemporaryDirectory(); const root = makeTemporaryDirectory();
const envFile = path.join(root, 'development.env'); const envFile = path.join(root, 'development.env');
writeFileSync( writeFileSync(
@@ -461,21 +493,17 @@ test('启用入口把 mysqldump 空结构持久化为 captured_empty', () => {
.join('\n'), .join('\n'),
); );
const result = runBaselineGeneration({ const result = await runBaselineGeneration({
repositoryRoot: root, repositoryRoot: root,
envFile, envFile,
applicationVersion: 'v0.0.8', applicationVersion: 'v0.0.8',
gitCommit: validMetadata.gitCommit, gitCommit: validMetadata.gitCommit,
generatedAt: validMetadata.generatedAt, generatedAt: validMetadata.generatedAt,
executeFile(command, args) { exportSchema: async () => ({
if (args[0] === '--version') { metadata: { ...validNodeMetadata, applicationVersion: 'v0.0.8' },
return `${command} Ver 14.14 Distrib 5.7.44, for Linux`; rawSql: '',
} objectCounts: { tables: 0, views: 0, triggers: 0, routines: 0, events: 0 },
if (command === 'mysql') { }),
return '5.7.42-log\tutf8mb4\tutf8mb4_general_ci\n';
}
return '';
},
}); });
assert.equal(result.status, 'captured_empty'); assert.equal(result.status, 'captured_empty');
@@ -488,6 +516,79 @@ test('启用入口把 mysqldump 空结构持久化为 captured_empty', () => {
); );
}); });
test('format v3 清单使用 Node exporter/driver 字段且不复用 CLI 字段', () => {
const artifacts = buildCapturedArtifacts(
validNodeMetadata,
'CREATE TABLE `sample` (`id` int);\n',
{ tables: 1, views: 0, triggers: 0, routines: 0, events: 0 },
);
const manifest = JSON.parse(artifacts.files['manifest.yaml']);
assert.equal(manifest.format_version, 3);
assert.equal(manifest.generator, 'schema-baseline/3.0.0');
assert.equal(manifest.exporter_name, 'mysql2-show-create');
assert.equal(manifest.exporter_version, '1.0.0');
assert.equal(manifest.database_driver, 'mysql2');
assert.equal(manifest.database_driver_version, '3.23.3');
assert.equal('mysql_client_version' in manifest, false);
assert.equal('mysqldump_client_version' in manifest, false);
});
test('format v3 空库和不适用清单保留可验证状态', () => {
const emptyManifest = JSON.parse(
buildCapturedEmptyArtifacts(validNodeMetadata).files['manifest.yaml'],
);
assert.equal(emptyManifest.format_version, 3);
assert.equal(emptyManifest.status, 'captured_empty');
assert.equal(emptyManifest.object_counts.tables, 0);
const disabledManifest = JSON.parse(
buildNotApplicableArtifacts(validNodeMetadata).files['manifest.yaml'],
);
assert.equal(disabledManifest.format_version, 3);
assert.equal(disabledManifest.status, 'not_applicable');
assert.equal(disabledManifest.database_driver, 'mysql2');
assert.equal(disabledManifest.mysql_server_version, 'not_enabled');
});
test('异步 Node exporter 注入路径生成 format v3 captured_empty', async () => {
const root = makeTemporaryDirectory();
const envFile = path.join(root, 'development.env');
writeFileSync(
envFile,
Object.entries(validConfig)
.map(([key, value]) => `${key}=${value}`)
.join('\n'),
);
let exportCalls = 0;
const result = await runBaselineGeneration({
repositoryRoot: root,
envFile,
applicationVersion: 'v0.0.9',
gitCommit: validNodeMetadata.gitCommit,
generatedAt: validNodeMetadata.generatedAt,
exportSchema: async () => {
exportCalls += 1;
return {
metadata: validNodeMetadata,
rawSql: '',
objectCounts: { tables: 0, views: 0, triggers: 0, routines: 0, events: 0 },
};
},
});
assert.equal(result.status, 'captured_empty');
assert.equal(exportCalls, 1);
assert.equal(
JSON.parse(
readFileSync(
path.join(root, 'database', 'schema-baselines', 'v0.0.9', 'manifest.yaml'),
'utf8',
),
).format_version,
3,
);
});
test('基线清单中的提交必须等于发布标签指向的提交', () => { test('基线清单中的提交必须等于发布标签指向的提交', () => {
const root = makeTemporaryDirectory(); const root = makeTemporaryDirectory();
const target = path.join(root, 'v0.0.5'); const target = path.join(root, 'v0.0.5');