Batch 2R: 40250 pack inventory (epk_scan) and delivery decision
Read-only scanner for Client/pack following PackInitialize -> CEterPackManager
-> CEterPack::__BuildIndex/Get2. Keys are read from the reference EterPack.cpp
at run time; none are stored. Pure-Python TEA + LZO1X.
- 136 registered packs, 54,891 entries: NONE/COMPRESS/SECURITY only, no
PANAMA/HybridCrypt, all entries decode
- first-registered pack wins; 2,282 shadowed entries (2,194 differ)
- metin2_patch_{sd,mineral,mundi} use a second packer layout; 4 SECURITY
files fail 40250's CRC check and never load in the original
- no case collisions; paths are already lowercase
- PORT-PLAN: read EPK on device via ported EterPack, base install minus
extra locales, rebuild dict in Index order on any pack change
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
shenlei
@@ -487,3 +487,4 @@
|
||||
{"time":"2026-09-22T12:00:00Z","event":"dead_code_removal","deleted_system_files":106,"deleted_tests":106,"lines_removed":36221,"manifest_refs_removed":76,"contracts_touched":18,"reason":"project/*_system.gd reachable only from their own test_*_parity.gd; not implementations of 40250 behavior (SKILL.md: unreachable code is not evidence)","tests":["full headless Godot suite: 192 PASS, 7 FAIL and 5 render tests hanging in headless, all identical on the pre-deletion tree"]}
|
||||
{"time":"2026-09-22T13:00:00Z","event":"port_map_rebaseline","unit":"UserInterface/PythonPlayerEventHandler.cpp","reset_to_todo":["CPythonPlayerEventHandler::GetSingleton","CPythonPlayerEventHandler::~CPythonPlayerEventHandler","CPythonPlayerEventHandler::OnMoving","CPythonPlayerEventHandler::OnMove","CPythonPlayerEventHandler::OnStop","CPythonPlayerEventHandler::CPythonPlayerEventHandler"],"reason":"new mirror architecture owns the CPythonPlayerEventHandler singleton in native code; prior GDScript counterparts are migration sources, not final mirror implementations","tests":["port_map.py check"]}
|
||||
{"time":"2026-09-22T15:00:00Z","event":"port_map_rebaseline","scope":"all units","totals":{"logic":3343,"python":5175,"platform":2091,"all":10609,"done":0,"divergent":1},"previous":{"logic":3285,"python":4890,"platform":2229,"all":10404},"changes":["cpp_functions: strip comments, accept indented qualified definitions inside namespace blocks, long parameter/initializer lists (40-line window), all-caps qualified ctors, skip constructor initializer entries","EterBase split by unit: tea/lzo/cipher/Random/Stl/Timer/Poly are logic, CPostIt/CRC32/Debug/FileBase/FileDir/FileLoader/MappedFile/TempFile/Utils/error stay platform; 8 obsolete platform stub files removed","audit/slices regenerated (layer fields; EterLib/Dimm.h now platform)"],"reason":"PORT-PLAN 2A step 6; inventory cross-checked against the clang-derived platform skeleton: every stub is inventoried except 22 declarations 40250 never defines and namespace/typedef spelling variants","tests":["port_map.py check","script/port_gate.sh (macos/android/ios/windows PASS, linux BLOCKED)"]}
|
||||
{"time":"2026-09-22T16:00:00Z","event":"pack_inventory","scope":"Client/pack (batch 2R)","tool":"tools/epk_scan/epk_scan.py","report":"audit/packs/2R-scan.json","totals":{"registered_packs":136,"entries":54891,"unique_paths":52609,"NONE":20727,"COMPRESS":29812,"SECURITY":4352,"PANAMA":0,"HYBRIDCRYPT":0,"decode_failures":0,"overrides":2282,"overrides_different":2194,"case_collisions":0},"findings":["first-registered pack wins (CEterFileDict multimap, VS2022 STL)","short-layout packer in metin2_patch_{sd,mineral,mundi}: 4 SECURITY files rejected by 40250 CRC check","no server-delivered pack keys needed"]}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+76
-18
@@ -81,7 +81,7 @@ extension/third_party/cpython-2.7.18/ # 静态库（2P 批次
|
||||
| 1 | 删除运行时不加载的 106 个 `*_system.gd` 及其测试 | 完成（a989b8f1） |
|
||||
| — | 定下目录结构，评估内嵌 Python；Android 独立程序验证 | 完成（9d0e50de、b02c49bb） |
|
||||
| **2A** | 基础：Win32 类型层、参考公共头的最小闭包、`port_logic` CMake 目标、platform 接口骨架、头文件可编译门禁、port-map 重新基线 | **进行中**（剩 Python 库闭包头，等 2P） |
|
||||
| **2R** | 资源包能力盘点：EPK 类型/密钥/覆盖顺序/路径大小写/移动端交付 | **下一步**，可与 2A 并行（只读分析 + 独立工具） |
|
||||
| 2R | 资源包能力盘点：EPK 类型/密钥/覆盖顺序/路径大小写/移动端交付 | 完成：只有 NONE/COMPRESS/SECURITY，不需要服务器密钥；设备上直接读 EPK |
|
||||
| 2P | CPython 2.7.18 编进 libmtgodot，五个平台分别配置和验证 | 2A 之后 |
|
||||
| 2D | 数据源切到 40250（msm 路径、proto、资源根、严格资源测试） | proto 部分在 2A 之后；资源根在 2R 之后 |
|
||||
| **2V0** | UI 壳：真实 `PythonLauncher/wndMgr/grp/app` + platform UI，其他玩法模块用桩，显示并操作 Logo/Popup | 需要 2A、2P 和 root/uiscript 子集 |
|
||||
@@ -202,23 +202,80 @@ extension/third_party/cpython-2.7.18/ # 静态库（2P 批次
|
||||
已知的无关问题：完整 Android 扩展构建在 `extension/src/net/text_codec.cpp` 失败（bionic 从 API 28 才声明 `iconv`，
|
||||
而构建用 API 24；c9389431 引入）；仓库里的 `build-android/` 缓存还指向旧路径 `mtgodot-poc`。
|
||||
|
||||
### 批次 2R：资源包能力盘点
|
||||
### 批次 2R：资源包能力盘点（完成，2026-09-22）
|
||||
|
||||
40250 的读取链不只是 `EterPack.cpp`，还有 `EterPackManager.cpp`（多包覆盖顺序、路径归一化）、
|
||||
`EterPackPolicy_CSHybridCrypt.cpp`（HybridCrypt，密钥可能来自登录/握手阶段的服务器下发）、`CMappedFile`、`CLZO`，
|
||||
以及 TEA/Panama/Camellia/Twofish/XTEA。压缩类型有六种：`NONE`、`COMPRESS`、`SECURITY`、`PANAMA`、
|
||||
`HYBRIDCRYPT`、`HYBRIDCRYPT_WITHSDB`。`Client/pack` 有 217 个文件、约 1.3 GB。
|
||||
工具：`tools/epk_scan/epk_scan.py`（只读）。`summary` 统计并写 `audit/packs/2R-scan.json`；`verify` 解码全部可本地解的条目。
|
||||
索引密钥和 SECURITY 密钥运行时从参考树 `EterPack/EterPack.cpp` 读取，仓库里没有密钥。读取链照 40250：`PackInitialize`
|
||||
（`pack/Index`）→ `CEterPackManager::RegisterPack`（每个包再注册 `<name>_texcache`，最后 `RegisterRootPack("pack/root")`）
|
||||
→ `CEterPack::__BuildIndex`（`.eix` 全部是 `MCOZ`：TEA + LZO 包着 `EPKD` v2，每条 192 字节）→ `Get2`。
|
||||
|
||||
1. 写一个只读的扫描工具（Python，放在 `tools/`），解析所有 `.eix`，统计每个包、每种 `compressed_type` 的文件数和字节数。
|
||||
2. 按统计结果决定：
|
||||
- 只有 `NONE/COMPRESS/SECURITY` 的包可以离线解出；
|
||||
- `PANAMA`/`HYBRIDCRYPT*` 如存在，确认密钥来源：本地的 `Index`/配置，还是服务器 `GC_HYBRIDCRYPT_KEYS`/`SDB` 下发。
|
||||
一次真服登录抓取仅可用于确认格式和密钥来源，不能作为构建或发布依赖；正式方案必须是可重复的运行时密钥链，或经过
|
||||
授权、可重复并有来源清单的重新打包流程，密钥本身不进仓库。
|
||||
3. 验证 `Index` 文件列出的包顺序和同名文件的覆盖优先级，与 `CEterPackManager` 一致。
|
||||
4. 路径归一化：40250 在 Windows 上大小写不敏感，并把 `d:/ymir work/` 等前缀映射到包内路径。确定统一的小写化规则，扫描大小写冲突。
|
||||
5. 移动端交付：决定最终形式（例如解包后重新打成我们自己的 `mtpack`，或直接在设备上读 EPK），给出 Android/iOS 的包体积、首包与按需下载的划分，以及更新方式。`MT_ASSETS` 只是开发期覆盖。
|
||||
6. 输出写进本文第 5 节，再决定 2D 的资源根切换方式。
|
||||
1. **类型统计**（Index 注册的 136 个包，54,891 条，52,609 个不同路径）：
|
||||
|
||||
| 类型 | 文件数 | 存储 | 解码后 |
|
||||
| --- | --- | --- | --- |
|
||||
| `NONE` | 20,727 | 737.2 MiB | 737.2 MiB |
|
||||
| `COMPRESS`（LZO） | 29,812 | 592.9 MiB | 1296.8 MiB |
|
||||
| `SECURITY`（TEA + LZO） | 4,352 | 2.0 MiB | 6.6 MiB |
|
||||
| `PANAMA` / `HYBRIDCRYPT*` | **0** | — | — |
|
||||
|
||||
`verify --per-type 1000000` 全部 54,891 条解码成功。每个包的类型分布见 JSON 的 `per_pack`。
|
||||
2. **密钥来源**：这个客户端只用编译进客户端的两组 TEA 密钥（`s_adwEterPackKey` 解索引，`s_adwEterPackSecurityKey` 解
|
||||
SECURITY 数据）。没有 PANAMA 和 HybridCrypt 条目，所以**不需要服务器下发 IV/密钥**，也不需要真服抓包。
|
||||
`CAccountConnector` 的 `RegisterPack(..., abIV)` 和 `GC_HYBRIDCRYPT_KEYS/SDB` 照抄移植即可，但这批资源用不到。
|
||||
移植后的 `EterPack.cpp` 需要这两组常量；按"密钥不进仓库"，由构建时从参考树生成一个 gitignore 的头文件，不直接写进源码。
|
||||
3. **注册顺序与覆盖**：
|
||||
- `Index` 第 0 行是 `PACK`，后面是 (目录, 名字) 对，共 124 个名字。
|
||||
- 实际存在的包有 136 个：Index 里的包，加上两个 `metin2_patch_dragon_rock*_texcache`，再加 `root`。
|
||||
- 缺失的包：106 个 `_texcache`，以及 `locale_{ae,bg,ca,mx,pa,uk,us}`。40250 遇到缺失的包只记日志、继续执行。
|
||||
- 目录里有、但没注册的文件，40250 永远不会读到：`Outdoor`、`Psyco`（0 字节）、`outdoorempirebattle1`、两个 `.rdch`。
|
||||
`.rdch` 在源码里没有任何引用。
|
||||
- 查找由 `CEterFileDict` 完成，它是以 `filename_crc` 为键的 `unordered_multimap`。`GetItem` 在 `equal_range` 里返回第一个
|
||||
同名项。40250 用 VS2022（`ClientVS22`）构建，MSVC STL 的 multi 插入会把新元素放在同键已有元素的后面，所以**先注册的
|
||||
包优先**。`UpdateItem`（后注册的覆盖先注册的）只在带 IV 的注册里用到，这批资源没有。
|
||||
- 同名条目共 2,282 个，其中 2,194 个内容不同。主要是 `patch1`/`patch2`/`season3_eu`/`metin2_patch_*` 覆盖后面的
|
||||
`zone`、`pc*`、`season2`、`outdoor*`、`icon`。**覆盖顺序会影响画面，必须 1:1。**
|
||||
- 包里有两种打包器留下的格式：
|
||||
- Ymir 格式：`data_size` 不小于整个 CLZObject，常补到 256 字节的倍数；`data_crc` 按存储字节算。
|
||||
- 另一种格式出现在 `metin2_patch_sd`、`metin2_patch_mineral`、`metin2_patch_mundi` 三个包里：`data_size` 比 CLZObject
|
||||
短 16 字节，`data_crc` 按解码后的内容算。
|
||||
- 在第二种格式下，40250 的 COMPRESS 条目仍能读出来：`CLZO::Decompress` 按 LZO 头里的长度读，不看 `data_size`，读到
|
||||
映射视图里 `data_size` 之后的字节。但 SECURITY 条目过不了 `Get2` 的 CRC 检查。
|
||||
- 结果是 4 个文件在原版里**加载失败**：`npc/mineral2/wait.msa`、`effect/background/metinstone_loop_redblack.mse`、
|
||||
`monster2/spider_spawn/{wait.msa,line1.mse}`。第 5 个 `effect/etc/buff/buff_symbol1.mse` 由先注册的 `metin2_patch_eu4`
|
||||
提供，不受影响。
|
||||
- 移植的读取器要照这个行为实现：按头部长度读，保留 CRC 拒绝。
|
||||
4. **路径归一化**：
|
||||
- `ConvertFileName` 只做两件事：ASCII `tolower`，以及 `\` 转 `/`。CP949 字节原样保留。
|
||||
- 包内名字全部已经是小写、只用 `/`；52,609 个路径没有大小写冲突，`filename_crc` 与 `crc32(name)` 全部一致。
|
||||
- `d:/ymir work/...` 是包内的真实路径，一共 26,955 个。40250 的相对路径映射已经注释掉，所以调用方必须使用 40250 原样的
|
||||
路径名，不能再用 m2dev `assets/` 的相对路径。
|
||||
- 包文件名与 `Index` 大小写不一致，例如 `Etc` 对应磁盘上的 `ETC.eix`。NTFS 下能打开；在移动端和 Linux 上，按包名查找
|
||||
磁盘文件时要做大小写不敏感匹配。
|
||||
5. **移动端交付**（已注册包的存储大小共 1337 MiB）：
|
||||
|
||||
| 分组 | 大小 |
|
||||
| --- | --- |
|
||||
| `metin2_patch_*`/`patch1`/`patch2`/`season3_eu` | 692.7 MiB |
|
||||
| 核心 `d:/ymir work`（pc/monster/zone/effect/npc/item…）+ icon/property/textureset | 405.7 MiB |
|
||||
| sound/bgm | 87.6 MiB |
|
||||
| season1/2 | 71.5 MiB |
|
||||
| 其他 14 种语言的 locale | 49.9 MiB |
|
||||
| 地图包 | 25.9 MiB |
|
||||
| `locale_en` | 3.5 MiB |
|
||||
| root + uiscript | 0.6 MiB |
|
||||
|
||||
决定：
|
||||
- **设备上直接读 EPK**，用移植的 `EterPack`/`EterPackManager` 经 `asset_io` 读取。不重新打包成 `mtpack`。这样覆盖顺序、
|
||||
CRC 拒绝等行为天然 1:1，更新时直接替换包文件。
|
||||
- 首包包含 Index 里除其他语言外的全部包，约 1.29 GB。其他语言按需下载。
|
||||
- patch 包在覆盖链的最前面，按需下载会改变已有文件的内容，所以不做细粒度按需下载。
|
||||
- 如果以后要按需下载，任何新到的包都必须**按 Index 顺序重建整个字典**，不能追加到末尾。否则先注册者优先的结果就变了。
|
||||
- 更新方式是整包替换 `.eix/.epk`，由我们自己的清单记录版本号和 sha256。
|
||||
6. **2D 的资源根切换**：
|
||||
- 新增 pack 后端：`asset_io` 在 `MT_ASSETS` 之外，接上移植的 `CEterPackManager`，按 `pack/Index` 注册。
|
||||
- 渲染和 GDScript 的资源请求改用 40250 路径，经过 `ConvertFileName` 查找。
|
||||
- m2dev `assets/` 的散文件保留作开发期覆盖，直到对应读取方切到新路径。
|
||||
- 严格门禁（第 5 节步骤 3）的来源清单改为记录各个 `.eix/.epk` 的 sha256。
|
||||
|
||||
### 批次 2P：内嵌 Python 集成
|
||||
|
||||
@@ -276,7 +333,7 @@ extension/third_party/cpython-2.7.18/ # 静态库（2P 批次
|
||||
| `mob_proto` | `proto.cpp` | **读不了**：每条 255 字节 vs 335 字节，加密同上 | 同上 |
|
||||
| `item_list.txt` | `item_list.gd` | 177 行不同（m2dev 有 dummy 条目等）；40250 放在 `locale/en/`，m2dev 在 `locale/common/` | 按 40250 路径读取 |
|
||||
| `locale_interface.txt`、`itemdesc.txt`、`skilldesc.txt`、`skilltable.txt` | UI、技能表 | 20 行 / 2 行 / 2 行 / 位置不同 | 按 40250 路径读取 |
|
||||
| 模型、贴图、地图（PC、Monster、Outdoor 等） | 渲染 | **未核对**：40250 是经典 `.eix/.epk` 包 | 由 2R 决定 |
|
||||
| 模型、贴图、地图（PC、Monster、Outdoor 等） | 渲染 | 40250 是经典 `.eix/.epk`，136 个包、52,609 个路径，全部可以本地解码（2R） | 移植 `EterPack`/`EterPackManager` 接到 `asset_io`，请求改用 40250 路径（见 2R 第 6 条） |
|
||||
|
||||
切换步骤（批次 2D）：
|
||||
|
||||
@@ -288,7 +345,8 @@ extension/third_party/cpython-2.7.18/ # 静态库（2P 批次
|
||||
- 为 40250 的 root、locale、proto 和关键包生成来源清单（路径 + sha256），提交到 `audit/`，测试开始时核对。
|
||||
4. 新增并在 CI/发布前强制运行 `script/run_40250_asset_gate.sh`；它设置/校验 `MT_ASSETS_STRICT=1`，运行 native proto/pack/formats
|
||||
测试和 Godot root/locale/render 测试。脚本在没有 40250 资源时必须失败，不能把跳过报告成 PASS。
|
||||
5. 资源根切换方式按 2R 的结论执行，跑严格门禁，更新本节状态。
|
||||
5. 资源根切换按 2R 第 6 条执行：移植 `EterPack`（`CLZObject`/TEA 已经在第 2 步）和 `EterPackManager`，接上
|
||||
`asset_io` 的 pack 后端，读取方逐个改用 40250 路径，跑严格门禁，更新本节状态。
|
||||
|
||||
## 6. 进度与 port-map 基线
|
||||
|
||||
|
||||
@@ -0,0 +1,450 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only scan of the 40250 client packs (batch 2R, docs/PORT-PLAN.md).
|
||||
|
||||
Follows the 40250 read chain: `UserInterface.cpp` PackInitialize (pack/Index), `CEterPackManager`
|
||||
(RegisterPack, the `_texcache` companion, the root pack, `ConvertFileName`), `CEterPack::__BuildIndex`
|
||||
(.eix: plain `EPKD` or an `MCOZ` TEA+LZO object) and `CEterPack::Get2` (per-entry compressed_type).
|
||||
|
||||
epk_scan.py summary [--json OUT] per-pack / per-type counts and bytes, registration, overrides, paths
|
||||
epk_scan.py verify [--per-type N] decode N sample entries of every (pack, type) that needs no server key
|
||||
|
||||
The index and security keys are read from the reference `EterPack/EterPack.cpp` at run time; nothing
|
||||
key-like is stored here. Nothing is written except the optional --json report.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
import zlib
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO / ".agents/skills/metin2-40250-parity-audit/scripts"))
|
||||
import refroot # noqa: E402
|
||||
|
||||
TYPE_NAMES = {0: "NONE", 1: "COMPRESS", 2: "SECURITY", 3: "PANAMA", 4: "HYBRIDCRYPT", 5: "HYBRIDCRYPT_WITHSDB"}
|
||||
# Types CEterPack::Get2 can decode with keys compiled into the client; 3 needs the IV from the server
|
||||
# (CAccountConnector RegisterPack(..., abIV)), 4/5 need GC_HYBRIDCRYPT_KEYS / GC_HYBRIDCRYPT_SDB.
|
||||
LOCAL_TYPES = {0, 1, 2}
|
||||
EPKD = b"EPKD"
|
||||
MCOZ = b"MCOZ"
|
||||
# TEterPackIndex, #pragma pack(4), 32-bit long: id, filename[161], crc, real_data_size, data_size, data_crc,
|
||||
# data_position, compressed_type. Only data_size (the stored bytes) is read by CEterPack::Get2; real_data_size is
|
||||
# packer bookkeeping (free-block reuse) and is smaller than data_size in ~half the COMPRESS entries here, so it is
|
||||
# ignored. The decoded size lives only in the CLZObject header of COMPRESS/SECURITY data.
|
||||
INDEX = struct.Struct("<i161s3xIiiIib3x")
|
||||
assert INDEX.size == 192
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 40250 primitives
|
||||
|
||||
def reference_keys(ref: Path) -> dict[str, tuple[int, ...]]:
|
||||
text = (ref / "EterPack/EterPack.cpp").read_bytes().decode("latin-1")
|
||||
keys = {}
|
||||
for name in ("s_adwEterPackKey", "s_adwEterPackSecurityKey"):
|
||||
m = re.search(name + r"\[\]\s*=\s*\{([^}]*)\}", text)
|
||||
if not m:
|
||||
raise SystemExit(f"{name} not found in reference EterPack.cpp")
|
||||
keys[name] = tuple(int(v) for v in re.findall(r"\d+", m.group(1)))
|
||||
return keys
|
||||
|
||||
|
||||
def tea_decrypt(data: bytes, key: tuple[int, ...]) -> bytes:
|
||||
"""EterBase/tea.cpp tea_decrypt: 32 rounds, DELTA 0x9E3779B9, 32-bit words, size rounded up to 8."""
|
||||
if len(data) % 8:
|
||||
data = data + bytes(8 - len(data) % 8)
|
||||
words = struct.unpack(f"<{len(data) // 4}I", data)
|
||||
out = []
|
||||
k0, k1, k2, k3 = key
|
||||
kk = (k0, k1, k2, k3)
|
||||
M = 0xFFFFFFFF
|
||||
for i in range(0, len(words), 2):
|
||||
y, z = words[i], words[i + 1]
|
||||
s = (0x9E3779B9 * 32) & M
|
||||
for _ in range(32):
|
||||
z = (z - ((((y << 4) ^ (y >> 5)) + y) ^ (s + kk[(s >> 11) & 3]))) & M
|
||||
s = (s - 0x9E3779B9) & M
|
||||
y = (y - ((((z << 4) ^ (z >> 5)) + z) ^ (s + kk[s & 3]))) & M
|
||||
out += (y, z)
|
||||
return struct.pack(f"<{len(out)}I", *out)
|
||||
|
||||
|
||||
def lzo1x_decompress(src: bytes, out_len: int) -> bytes:
|
||||
"""lzo1x_decompress (lzo1x_d.ch), the decoder 40250 calls in CLZObject::Decompress."""
|
||||
op = bytearray()
|
||||
ip = 0
|
||||
|
||||
def copy_match(dist_back: int, n: int) -> None:
|
||||
start = len(op) - dist_back
|
||||
if start < 0:
|
||||
raise ValueError("lzo: match before start")
|
||||
if dist_back >= n:
|
||||
op.extend(op[start:start + n])
|
||||
else:
|
||||
for j in range(n):
|
||||
op.append(op[start + j])
|
||||
|
||||
state = "loop"
|
||||
t = 0
|
||||
if src[0] > 17:
|
||||
t = src[0] - 17
|
||||
ip = 1
|
||||
if t < 4:
|
||||
state = "match_next"
|
||||
else:
|
||||
op += src[ip:ip + t]
|
||||
ip += t
|
||||
state = "first_literal_run"
|
||||
while True:
|
||||
if state == "loop":
|
||||
t = src[ip]; ip += 1
|
||||
if t >= 16:
|
||||
state = "match"
|
||||
continue
|
||||
if t == 0:
|
||||
while src[ip] == 0:
|
||||
t += 255; ip += 1
|
||||
t += 15 + src[ip]; ip += 1
|
||||
op += src[ip:ip + t + 3]
|
||||
ip += t + 3
|
||||
state = "first_literal_run"
|
||||
if state == "first_literal_run":
|
||||
t = src[ip]; ip += 1
|
||||
if t >= 16:
|
||||
state = "match"
|
||||
else:
|
||||
dist = 1 + 0x0800 + (t >> 2) + (src[ip] << 2); ip += 1
|
||||
copy_match(dist, 3)
|
||||
state = "match_done"
|
||||
if state == "match":
|
||||
if t >= 64:
|
||||
dist = 1 + ((t >> 2) & 7) + (src[ip] << 3); ip += 1
|
||||
copy_match(dist, (t >> 5) + 1)
|
||||
elif t >= 32:
|
||||
t &= 31
|
||||
if t == 0:
|
||||
while src[ip] == 0:
|
||||
t += 255; ip += 1
|
||||
t += 31 + src[ip]; ip += 1
|
||||
dist = 1 + (src[ip] >> 2) + (src[ip + 1] << 6); ip += 2
|
||||
copy_match(dist, t + 2)
|
||||
elif t >= 16:
|
||||
dist = (t & 8) << 11
|
||||
t &= 7
|
||||
if t == 0:
|
||||
while src[ip] == 0:
|
||||
t += 255; ip += 1
|
||||
t += 7 + src[ip]; ip += 1
|
||||
dist += (src[ip] >> 2) + (src[ip + 1] << 6); ip += 2
|
||||
if dist == 0:
|
||||
break # end of stream
|
||||
copy_match(dist + 0x4000, t + 2)
|
||||
else:
|
||||
dist = 1 + (t >> 2) + (src[ip] << 2); ip += 1
|
||||
copy_match(dist, 2)
|
||||
state = "match_done"
|
||||
if state == "match_done":
|
||||
t = src[ip - 2] & 3
|
||||
if t == 0:
|
||||
state = "loop"
|
||||
continue
|
||||
state = "match_next"
|
||||
if state == "match_next":
|
||||
op += src[ip:ip + t]
|
||||
ip += t
|
||||
t = src[ip]; ip += 1
|
||||
state = "match"
|
||||
if len(op) != out_len:
|
||||
raise ValueError(f"lzo: size {len(op)} != {out_len}")
|
||||
return bytes(op)
|
||||
|
||||
|
||||
def lz_object(buf: bytes, key: tuple[int, ...] | None) -> bytes:
|
||||
"""CLZO::Decompress: THeader {fourcc, encrypt, compressed, real}, then MCOZ + LZO (TEA'd when encrypt != 0)."""
|
||||
fourcc, enc, comp, real = struct.unpack_from("<4sIII", buf)
|
||||
if fourcc != MCOZ:
|
||||
raise ValueError("not an MCOZ object")
|
||||
if enc:
|
||||
if key is None:
|
||||
raise ValueError("encrypted object without key")
|
||||
body = tea_decrypt(buf[16:16 + enc], key)
|
||||
if body[:4] != MCOZ:
|
||||
raise ValueError("TEA key incorrect")
|
||||
return lzo1x_decompress(body[4:4 + comp], real)
|
||||
return lzo1x_decompress(buf[20:20 + comp], real)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- packs
|
||||
|
||||
def read_registration(pack_dir: Path) -> list[tuple[str, str]]:
|
||||
"""PackInitialize: line 0 is PACK/FILE, then (folder, name) pairs for i in [1, count-1)."""
|
||||
lines = (pack_dir / "Index").read_bytes().decode("latin-1").splitlines()
|
||||
if lines[0] not in ("PACK", "FILE"):
|
||||
raise SystemExit("pack/Index: first line must be PACK or FILE")
|
||||
pairs = []
|
||||
for i in range(1, len(lines) - 1, 2):
|
||||
pairs.append((lines[i], lines[i + 1]))
|
||||
return pairs
|
||||
|
||||
|
||||
def read_index(eix: Path, key: tuple[int, ...]) -> tuple[list[dict], bool]:
|
||||
raw = eix.read_bytes()
|
||||
encrypted = raw[:4] == MCOZ
|
||||
data = lz_object(raw, key) if encrypted else raw
|
||||
if data[:4] != EPKD:
|
||||
raise ValueError(f"{eix.name}: fourcc {data[:4]!r}")
|
||||
ver, count = struct.unpack_from("<Ii", data, 4)
|
||||
if ver != 2 or len(data) < 12 + INDEX.size * count:
|
||||
raise ValueError(f"{eix.name}: version {ver} / size")
|
||||
entries = []
|
||||
for i in range(count):
|
||||
eid, name, crc, real, size, dcrc, pos, ctype = INDEX.unpack_from(data, 12 + INDEX.size * i)
|
||||
if not crc:
|
||||
continue # free index slot (PushFreeIndex)
|
||||
entries.append({"name": name.split(b"\0", 1)[0].decode("latin-1"), "crc": crc, "block": real,
|
||||
"size": size, "data_crc": dcrc, "pos": pos, "type": ctype})
|
||||
return entries, encrypted
|
||||
|
||||
|
||||
def find_file(pack_dir: Path, stem: str, ext: str) -> Path | None:
|
||||
"""40250 runs on case-insensitive NTFS: `Etc` in Index opens `ETC.eix`."""
|
||||
want = (stem + ext).lower()
|
||||
for p in pack_dir.iterdir():
|
||||
if p.name.lower() == want:
|
||||
return p
|
||||
return None
|
||||
|
||||
|
||||
def load(client: Path, ref: Path):
|
||||
pack_dir = client / "pack"
|
||||
keys = reference_keys(ref)
|
||||
order = [] # (registered name, folder) in CEterPackManager insertion order
|
||||
for folder, name in read_registration(pack_dir):
|
||||
order.append((name, folder))
|
||||
order.append((name + "_texcache", folder))
|
||||
order.append(("root", "")) # RegisterRootPack after the loop
|
||||
packs = {}
|
||||
for name, folder in order:
|
||||
if name in packs:
|
||||
continue # m_PackMap.find: registered once
|
||||
eix, epk = find_file(pack_dir, name, ".eix"), find_file(pack_dir, name, ".epk")
|
||||
if not eix or not epk:
|
||||
packs[name] = {"folder": folder, "missing": True}
|
||||
continue
|
||||
entries, encrypted = read_index(eix, keys["s_adwEterPackKey"])
|
||||
packs[name] = {"folder": folder, "missing": False, "eix": eix, "epk": epk, "index_encrypted": encrypted,
|
||||
"entries": entries, "epk_bytes": epk.stat().st_size}
|
||||
known = {p.name.lower() for n, v in packs.items() if not v["missing"] for p in (v["eix"], v["epk"])}
|
||||
stray = sorted(p.name for p in pack_dir.iterdir() if p.name.lower() not in known and p.name != "Index")
|
||||
return pack_dir, keys, order, packs, stray
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- commands
|
||||
|
||||
def decoded_sizes(p: dict) -> dict[int, int]:
|
||||
"""data_position -> decoded bytes (CLZObject THeader.dwRealSize for types 1/2, data_size otherwise)."""
|
||||
out = {}
|
||||
with open(p["epk"], "rb") as f:
|
||||
for e in p["entries"]:
|
||||
if e["type"] in (1, 2):
|
||||
f.seek(e["pos"])
|
||||
fourcc, _, _, real = struct.unpack("<4sIII", f.read(16))
|
||||
if fourcc != MCOZ:
|
||||
raise ValueError(f"{e['name']}: no MCOZ header")
|
||||
out[e["pos"]] = real
|
||||
else:
|
||||
out[e["pos"]] = e["size"]
|
||||
return out
|
||||
|
||||
|
||||
def summary(args) -> dict:
|
||||
client = Path(args.client) if args.client else refroot.reference_root().parents[1] / "Client"
|
||||
pack_dir, keys, order, packs, stray = load(client, refroot.reference_root())
|
||||
|
||||
per_type = defaultdict(lambda: {"files": 0, "stored_bytes": 0, "decoded_bytes": 0})
|
||||
per_pack = {}
|
||||
crc_mismatch = 0
|
||||
not_lower = 0
|
||||
backslash = 0
|
||||
prefixes = Counter()
|
||||
dict_first: dict[str, tuple[str, dict]] = {}
|
||||
overridden = []
|
||||
for name, _ in order:
|
||||
p = packs[name]
|
||||
if p["missing"] or name in per_pack:
|
||||
continue
|
||||
counts = Counter()
|
||||
size = Counter()
|
||||
decoded = decoded_sizes(p)
|
||||
for e in p["entries"]:
|
||||
t = TYPE_NAMES.get(e["type"], str(e["type"]))
|
||||
counts[t] += 1
|
||||
size[t] += e["size"]
|
||||
agg = per_type[t]
|
||||
agg["files"] += 1
|
||||
agg["stored_bytes"] += e["size"]
|
||||
agg["decoded_bytes"] += decoded[e["pos"]]
|
||||
fn = e["name"]
|
||||
if zlib.crc32(fn.encode("latin-1")) != e["crc"]:
|
||||
crc_mismatch += 1
|
||||
if any("A" <= c <= "Z" for c in fn): # ConvertFileName is an ASCII tolower; cp949 bytes pass through
|
||||
not_lower += 1
|
||||
if "\\" in fn:
|
||||
backslash += 1
|
||||
prefixes[fn.split("/")[0] + "/" + (fn.split("/")[1] + "/" if fn.count("/") > 1 else "")] += 1
|
||||
# CEterFileDict::InsertItem into an unordered_multimap; GetItem returns the first equal element.
|
||||
if fn in dict_first:
|
||||
w_pack, w = dict_first[fn]
|
||||
same = (w["size"], w["data_crc"]) == (e["size"], e["data_crc"])
|
||||
overridden.append({"file": fn, "kept": w_pack, "shadowed": name, "same_content": same})
|
||||
else:
|
||||
dict_first[fn] = (name, e)
|
||||
per_pack[name] = {"folder": p["folder"], "files": sum(counts.values()), "epk_bytes": p["epk_bytes"],
|
||||
"index_encrypted": p["index_encrypted"], "types": dict(counts), "type_bytes": dict(size)}
|
||||
|
||||
lower_groups = defaultdict(set)
|
||||
for fn in dict_first:
|
||||
lower_groups[fn.lower()].add(fn)
|
||||
case_collisions = sorted(v for v in (sorted(g) for g in lower_groups.values()) if len(v) > 1)
|
||||
|
||||
report = {
|
||||
"pack_dir": "Client/pack", # relative to the 40250 tree; no machine paths in committed reports
|
||||
"registered": [n for n, _ in order],
|
||||
"missing": sorted(n for n, v in packs.items() if v["missing"]),
|
||||
"stray_files": stray,
|
||||
"per_type": dict(per_type),
|
||||
"per_pack": per_pack,
|
||||
"unique_files": len(dict_first),
|
||||
"overrides": {"count": len(overridden), "different_content": sum(1 for o in overridden if not o["same_content"]),
|
||||
"by_pair": Counter(f'{o["kept"]} > {o["shadowed"]}' for o in overridden).most_common(),
|
||||
"different_examples": [o for o in overridden if not o["same_content"]][:20]},
|
||||
"paths": {"filename_crc_mismatch": crc_mismatch, "not_lowercase": not_lower, "backslash": backslash,
|
||||
"case_collisions": case_collisions, "top_prefixes": prefixes.most_common(25)},
|
||||
}
|
||||
print_summary(report)
|
||||
if args.json:
|
||||
Path(args.json).write_text(json.dumps(report, indent=1, ensure_ascii=False) + "\n", encoding="utf-8")
|
||||
print(f"\nwrote {args.json}")
|
||||
return report
|
||||
|
||||
|
||||
def print_summary(r: dict) -> None:
|
||||
print(f"pack dir: {r['pack_dir']} (40250 tree)")
|
||||
print(f"registered: {len(r['registered'])} (Index pairs x2 with _texcache, + root); "
|
||||
f"missing on disk: {len(r['missing'])}; stray files: {r['stray_files']}")
|
||||
print("\nper compressed_type:")
|
||||
for t, v in sorted(r["per_type"].items()):
|
||||
print(f" {t:<22} {v['files']:>7} files {v['stored_bytes'] / 2**20:>9.1f} MiB stored"
|
||||
f" {v['decoded_bytes'] / 2**20:>9.1f} MiB decoded")
|
||||
print(f"\nunique paths: {r['unique_files']}; overridden entries: {r['overrides']['count']}"
|
||||
f" ({r['overrides']['different_content']} with different content)")
|
||||
for pair, n in r["overrides"]["by_pair"][:15]:
|
||||
print(f" {n:>6} {pair}")
|
||||
p = r["paths"]
|
||||
print(f"\npaths: crc mismatch {p['filename_crc_mismatch']}, not lowercase {p['not_lowercase']}, "
|
||||
f"backslash {p['backslash']}, case collisions {len(p['case_collisions'])}")
|
||||
for pre, n in p["top_prefixes"][:12]:
|
||||
print(f" {n:>7} {pre}")
|
||||
print("\nper pack (types):")
|
||||
for name, v in r["per_pack"].items():
|
||||
ts = ", ".join(f"{t}={n}" for t, n in sorted(v["types"].items()))
|
||||
print(f" {name:<40} {v['folder']:<14} {v['files']:>6} {v['epk_bytes'] / 2**20:>7.1f} MiB {ts}")
|
||||
|
||||
|
||||
def lz_object_len(head: bytes) -> int:
|
||||
"""Bytes a CLZObject really occupies: THeader + (TEA block when encrypted, else fourcc + LZO)."""
|
||||
fourcc, enc, comp, _ = struct.unpack_from("<4sIII", head)
|
||||
if fourcc != MCOZ:
|
||||
raise ValueError("no MCOZ header")
|
||||
return 16 + (enc if enc else 4 + comp)
|
||||
|
||||
|
||||
def verify(args) -> int:
|
||||
"""Decode every sampled NONE/COMPRESS/SECURITY entry and classify how its pack was written.
|
||||
|
||||
Two packers wrote these files. The Ymir one stores data_size >= the whole CLZObject (often padded to a
|
||||
256-byte multiple) and data_crc over the stored bytes, which is what CEterPack::Get2 checks for SECURITY/PANAMA. The other ("short") stores
|
||||
data_size 16 bytes short of the object and data_crc over the decoded bytes: 40250 still decodes its
|
||||
COMPRESS entries (CLZO::Decompress trusts the header and reads past data_size inside the mapped view) but
|
||||
rejects its SECURITY entries in the CRC check, so those files never load in the original client.
|
||||
"""
|
||||
client = Path(args.client) if args.client else refroot.reference_root().parents[1] / "Client"
|
||||
_, keys, _, packs, _ = load(client, refroot.reference_root())
|
||||
ok = bad = 0
|
||||
layout = Counter()
|
||||
crc_stats = Counter()
|
||||
rejected = []
|
||||
for name, p in packs.items():
|
||||
if p["missing"]:
|
||||
continue
|
||||
by_type = defaultdict(list)
|
||||
for e in p["entries"]:
|
||||
by_type[e["type"]].append(e)
|
||||
with open(p["epk"], "rb") as f:
|
||||
for t, es in sorted(by_type.items()):
|
||||
if t not in LOCAL_TYPES:
|
||||
continue
|
||||
es = sorted(es, key=lambda e: e["size"])
|
||||
picks = {id(e): e for e in (es[:: max(1, len(es) // args.per_type)][: args.per_type] + [es[-1]])}
|
||||
for e in picks.values():
|
||||
try:
|
||||
f.seek(e["pos"])
|
||||
stored = f.read(e["size"])
|
||||
if len(stored) != e["size"]:
|
||||
raise ValueError("short read")
|
||||
if t == 0:
|
||||
out, kind = stored, "none"
|
||||
else:
|
||||
f.seek(e["pos"])
|
||||
obj = f.read(lz_object_len(stored[:16]))
|
||||
gap = len(obj) - e["size"]
|
||||
kind = "ymir" if gap <= 0 else "short" if gap == 16 else None
|
||||
if kind is None:
|
||||
raise ValueError(f"data_size {e['size']} vs object {len(obj)}")
|
||||
out = lz_object(obj, keys["s_adwEterPackSecurityKey"] if t == 2 else None)
|
||||
# Only SECURITY/PANAMA are CRC-checked by the client; tally what the CRC covers.
|
||||
crc = e["data_crc"]
|
||||
covers = ("stored" if zlib.crc32(stored) == crc else
|
||||
"decoded" if zlib.crc32(out) == crc else "neither")
|
||||
layout[f"{name}:{kind}"] += 1
|
||||
crc_stats[(TYPE_NAMES[t], kind, covers)] += 1
|
||||
if t == 2 and zlib.crc32(stored) != e["data_crc"]:
|
||||
rejected.append(f"{name}:{e['name']}")
|
||||
ok += 1
|
||||
except (ValueError, IndexError, struct.error) as exc:
|
||||
bad += 1
|
||||
print(f"FAIL {name}:{e['name']} ({TYPE_NAMES[t]}): {exc}")
|
||||
short = sorted({k.split(":")[0] for k in layout if k.endswith(":short")})
|
||||
print(f"short-layout packs: {short}")
|
||||
print("data_crc covers (type, layout, covers): " + ", ".join(f"{k}={v}" for k, v in sorted(crc_stats.items())))
|
||||
print(f"SECURITY entries the 40250 CRC check rejects: {len(rejected)}")
|
||||
for r in rejected:
|
||||
print(f" {r}")
|
||||
print(f"verify: {ok} decoded, {bad} failed")
|
||||
return 1 if bad else 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--client", default=os.environ.get("MT_40250_CLIENT"),
|
||||
help="40250 Client dir (default: <reference root>/../../Client)")
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
s = sub.add_parser("summary")
|
||||
s.add_argument("--json")
|
||||
v = sub.add_parser("verify")
|
||||
v.add_argument("--per-type", type=int, default=3)
|
||||
args = ap.parse_args()
|
||||
if args.cmd == "summary":
|
||||
summary(args)
|
||||
return 0
|
||||
return verify(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user