feat: 独立词库、三向认词与当日快闪复习
## 独立词库 理解词原先只能跟着课程单元走，学完 A0 十课词汇量只增加约 22 个实词， 不足以解决"记不住单词"。新增一份独立词库 assets/words/wordbank.json （2748 词，A1–B1），挂进 receptiveWordRegistry 的合成单元 bank-A1/A2/B1， 完全复用理解词已有的状态机，不依赖课程进度，第一天就能用。 数据来源、许可与合成规则记在 tool/words/DATA-NOTE.md：CEFR-J 定等级、 公开词书提供音标、AI 重写全部释义并生成例句、OpenSubtitles 提供口语词频。 词书部分为 CC BY-NC-SA 4.0 且上游权利不明，仅供个人非商用； 若要分发或上架，须替换音标那一列。 ## 背单词机制 - 间隔阶梯 1/3/7/15/30/60/120 天，连续答对上一级，答错回第一级。 原先首次答对后要等 7 天才复习，正是"第二天就忘"的成因。 - 每日新词上限（10 分钟 8 个 / 20 分钟 15 个 / 30 分钟 20 个）。 阶梯第一级是次日，今天引入的新词就是明天的工作量。 - 新词按口语频率发放，不再按字母序 —— A1 从 a.m./ability 变成 no/not/know/just。 - 三个方向按层级轮转：看词（英→中）→ 听词（音→中）→ 想词（中→英）。 想词题仍是选择题，不要求产出，理解词定位不变，不进升级分母。 - 单词页独立成 tab，首页今日任务卡下方给一张认词入口卡。 ## 用法对照 课程 JSON 增加 usage 字段（when/reply/swap/confuse）：一个句型用在什么场合、 对方通常怎么答、还能怎么说、跟哪个学过的句型容易混。 知道 How are you? 的意思，不等于知道它不是用来问名字的。 ## 复习流 - 当日快闪（recap）独立成队列，不占复习预算，也不计入积压。 - 只发放当日预算内的量，其余保持到期状态等下次，不悄悄丢弃或改期。 - 答错的项隔几题后回来，而不是立刻重问。 测试 296 通过，flutter analyze 干净。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
shenlei
@@ -0,0 +1,91 @@
|
||||
# 词库数据来源与许可
|
||||
|
||||
`assets/words/wordbank.json` 由两份来源合成，两份的地位不同，处置方式也不同。
|
||||
|
||||
## 1. 等级与词表范围：CEFR-J Vocabulary Profile 1.5
|
||||
|
||||
- 决定**哪些词进词库、算哪一级**。
|
||||
- 登记在 `tool/courses/lexicon/SOURCES.md`（版本、下载地址、SHA-256、许可原文），
|
||||
与 `tool/courses/pools/{a1,a2,b1}-word-pool.csv` 的生成用的是同一份数据。
|
||||
- CEFR-J 本身**只有分级，没有释义、音标、例句、词频** —— 它是分级表，不是词典。
|
||||
|
||||
## 2. 音标：公开词书（有许可风险，未解决）
|
||||
|
||||
- 来源：<https://github.com/lilinji/English>，由 `tool/words/fetch.sh` 下载其中的
|
||||
入门级词书（KET、PET、小学/初中教材词表）。
|
||||
- 许可：仓库标 CC BY-NC-SA 4.0，README 自称「整理自公开来源，仅供个人学习，严禁商用」。
|
||||
上游权利不明，**低于本项目 `SOURCES.md` 对课程数据的标准**。
|
||||
- 结论：**自用、不分发，风险基本为零；一旦要分发或上架，这部分必须换掉。**
|
||||
换的时候只需要替换音标一列 —— 词表范围和等级由 CEFR-J 决定，释义和例句已经由 AI 重写，
|
||||
都不受影响。
|
||||
|
||||
## 3. 释义与例句：AI 重写（deepseek-flash）
|
||||
|
||||
- `tool/words/enrich_wordbank.py` 把词书释义当**参考**喂给模型，明确告诉它词书质量不稳定、
|
||||
可以推翻；模型返回主释义、次要义项、一个例句和例句中文。
|
||||
- `tool/words/apply_enrichment.py` 校验后合并：释义为空、带括号、超过 10 个字的一律丢弃，
|
||||
保留词书原值。实际全部 2748 条都通过了校验。
|
||||
- 结果缓存在 `ENRICH_CACHE` 指定的 jsonl 里，按 id 去重，中断可续跑，不会重复付费。
|
||||
- 一次全量：输入约 10 万 token，输出约 63 万 token（模型会先推理，输出占大头）。
|
||||
|
||||
## 4. 词频：OpenSubtitles 计数（MIT）
|
||||
|
||||
- 来源：<https://github.com/hermitdave/FrequencyWords> 的 `content/2018/en/en_50k.txt`，
|
||||
仓库 MIT 许可，计数来自 OpenSubtitles 字幕语料。
|
||||
- 决定**新词按什么顺序发给用户**。同等级内从最常用的开始，不按字母序。
|
||||
- 为什么不用网页语料：先试了 `first20hours/google-10000-english`（公共领域），
|
||||
对词库只有 78% 覆盖，缺的全是 `ankle`、`ambulance`、`asleep`、`aspirin` 这类日常词 ——
|
||||
网页语料对口语词覆盖不好。字幕语料覆盖 97%，缺的只有 `bus stop` 这类多词短语。
|
||||
- 多词短语取组成词里**最罕见**那个词的名次：短语不会比它最难的组成词更容易。
|
||||
- `tool/words/add_frequency.py` 把名次写进 `rank` 字段；表里完全查不到的 11 个词
|
||||
排在本等级末尾。
|
||||
|
||||
排序效果：
|
||||
|
||||
| 等级 | 前 12 个词 |
|
||||
| --- | --- |
|
||||
| A1 | no not know just there here like get go right out about |
|
||||
| A2 | it yeah even first name ever enough understand next dead actually bit |
|
||||
| B1 | working rest able stupid sometimes sort serious scared hang bet beat calm |
|
||||
|
||||
## 合成规则
|
||||
|
||||
`tool/words/build_wordbank.py`：
|
||||
|
||||
1. 读 `tool/courses/pools/*-word-pool.csv`，只取 `status == pool` 且非虚词的词条；
|
||||
2. 词书里的词只有落在词池里才进词库，**词书不能往词库里加词**；
|
||||
3. 释义按词性拆分，优先匹配 CEFR-J 标注的词性，丢掉 `[美俚]` `<美>` 这类限定义项，
|
||||
去掉全部括号，每个义项取第一个逗号项；
|
||||
4. 第一个义项进 `zh`（选择题答案），其余进 `more`（只在词条详情里显示）；
|
||||
5. `add_frequency.py` 最后写入 `rank`，决定发放顺序。
|
||||
|
||||
## 现状
|
||||
|
||||
| 项 | 数值 |
|
||||
| --- | --- |
|
||||
| 词条总数 | 2748（A1 807 / A2 912 / B1 1029）|
|
||||
| A1 词池覆盖 | 807 / 828 = 97% |
|
||||
| A2 词池覆盖 | 912 / 1065 = 86% |
|
||||
| B1 词池覆盖 | 1029 / 1557 = 66% |
|
||||
| 有例句 | 2748（100%）|
|
||||
| 有词频名次 | 2737（99.6%）|
|
||||
| 有次要义项 | 2304 |
|
||||
| 无音标 | 37 条 |
|
||||
|
||||
规格 5.4 要求覆盖率 ≥85%：A1、A2 达标，**B1 偏低**，因为只下了入门级词书，
|
||||
补 B1 需要再下四六级一类的中高级词书。
|
||||
|
||||
## 释义缺陷的处置
|
||||
|
||||
词书原始数据有相当一部分词条只收了罕见义，规则清洗修不掉，全部由 AI 重写解决：
|
||||
|
||||
| 词 | 词书原值 | 现值 |
|
||||
| --- | --- | --- |
|
||||
| ticket | 加标签于；指派 | 票（次要义：罚单）|
|
||||
| it | 信息技术information technology | 它 |
|
||||
| check | 制止；检验 | 检查（次要义：核对；支票）|
|
||||
| accept | 承认；同意；承兑 | 接受（次要义：同意）|
|
||||
|
||||
`apply_enrichment.py` 还会打印一份「需要人眼看一下」的例句清单：它用词干前缀判断例句里
|
||||
是否真的用到了这个词，而 `teach → taught`、`bus stop` 这类变形和词组它判断不了，
|
||||
所以只报告、不丢弃。最近一次是 9 条，全部核对无误。
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Writes a frequency rank onto every bank word.
|
||||
|
||||
Without this the bank is handed out in alphabetical order, so the first two
|
||||
weeks are nothing but words starting with `a`. Rank 0 is the most common word.
|
||||
|
||||
A multi-word entry (`bus stop`, `air conditioning`) is not in a word frequency
|
||||
list at all, so it takes the rank of its rarest part: a phrase is no easier
|
||||
than the hardest word in it.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
BANK = os.path.join(ROOT, 'assets/words/wordbank.json')
|
||||
|
||||
# Words the list does not have at all go last, but stay in their level.
|
||||
UNRANKED = 99999
|
||||
|
||||
|
||||
def load_ranks(path):
|
||||
ranks = {}
|
||||
with open(path) as handle:
|
||||
for position, line in enumerate(handle):
|
||||
parts = line.split()
|
||||
if len(parts) == 2:
|
||||
ranks.setdefault(parts[0], position)
|
||||
return ranks
|
||||
|
||||
|
||||
def rank_of(word, ranks):
|
||||
direct = ranks.get(word.lower())
|
||||
if direct is not None:
|
||||
return direct
|
||||
parts = [re.sub(r"[^a-z']", '', part) for part in word.lower().split()]
|
||||
parts = [part for part in parts if part]
|
||||
found = [ranks[part] for part in parts if part in ranks]
|
||||
if len(found) == len(parts) and found:
|
||||
return max(found)
|
||||
return UNRANKED
|
||||
|
||||
|
||||
def main():
|
||||
frequency = sys.argv[1]
|
||||
ranks = load_ranks(frequency)
|
||||
with open(BANK) as handle:
|
||||
bank = json.load(handle)
|
||||
unranked = 0
|
||||
for word in bank['words']:
|
||||
rank = rank_of(word['en'], ranks)
|
||||
word['rank'] = rank
|
||||
if rank == UNRANKED:
|
||||
unranked += 1
|
||||
with open(BANK, 'w') as handle:
|
||||
json.dump(bank, handle, ensure_ascii=False, separators=(',', ':'))
|
||||
handle.write('\n')
|
||||
print('ranked %d, unranked %d' % (len(bank['words']) - unranked, unranked))
|
||||
for level in ('A1', 'A2', 'B1'):
|
||||
words = sorted(
|
||||
(w for w in bank['words'] if w['level'] == level),
|
||||
key=lambda w: w['rank'],
|
||||
)
|
||||
print('%s first 12: %s' % (level, ' '.join(w['en'] for w in words[:12])))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Merges the AI rewrite back into assets/words/wordbank.json.
|
||||
|
||||
Anything the model returned is checked before it lands: a gloss that is empty,
|
||||
parenthesised or sentence-long is worse than the word book's version, so the
|
||||
old value is kept and reported instead of being written over.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
BANK = os.path.join(ROOT, 'assets/words/wordbank.json')
|
||||
|
||||
BAD_GLOSS = re.compile(r'[()（）\[\]【】<>]')
|
||||
|
||||
|
||||
def usable_gloss(text):
|
||||
text = (text or '').strip()
|
||||
return bool(text) and len(text) <= 10 and not BAD_GLOSS.search(text)
|
||||
|
||||
|
||||
def mentions(word, sentence):
|
||||
"""Whether the sentence visibly uses the word.
|
||||
|
||||
This cannot be a hard check: `teach` shows up as "taught" and `bus stop`
|
||||
as two words, so a prefix match has no way to confirm them. It only flags
|
||||
a sentence as worth a human glance -- nothing is dropped over it.
|
||||
"""
|
||||
parts = [re.sub(r'[^a-z]', '', part) for part in word.lower().split()]
|
||||
stem = max(parts, key=len)[:3]
|
||||
return bool(stem) and stem in sentence.lower()
|
||||
|
||||
|
||||
def main():
|
||||
cache_path = os.environ.get(
|
||||
'ENRICH_CACHE', os.path.join(tempfile.gettempdir(), 'enrich_cache.jsonl')
|
||||
)
|
||||
rows = {}
|
||||
with open(cache_path) as handle:
|
||||
for line in handle:
|
||||
row = json.loads(line)
|
||||
rows[row['id']] = row
|
||||
with open(BANK) as handle:
|
||||
bank = json.load(handle)
|
||||
|
||||
kept = {'gloss': 0, 'example': 0, 'missing': 0, 'bad_gloss': 0, 'no_example': 0}
|
||||
unclear = []
|
||||
for word in bank['words']:
|
||||
row = rows.get(word['id'])
|
||||
if row is None:
|
||||
kept['missing'] += 1
|
||||
continue
|
||||
if usable_gloss(row.get('zh')):
|
||||
word['zh'] = row['zh'].strip()
|
||||
more = (row.get('more') or '').strip()
|
||||
senses = [s for s in more.split('；') if usable_gloss(s) and s != word['zh']]
|
||||
if senses:
|
||||
word['more'] = '；'.join(senses[:2])
|
||||
else:
|
||||
word.pop('more', None)
|
||||
kept['gloss'] += 1
|
||||
else:
|
||||
kept['bad_gloss'] += 1
|
||||
sentence = (row.get('ex') or '').strip()
|
||||
if not sentence:
|
||||
kept['no_example'] += 1
|
||||
continue
|
||||
word['ex'] = sentence
|
||||
word['exZh'] = (row.get('ex_zh') or '').strip()
|
||||
kept['example'] += 1
|
||||
if not mentions(word['en'], sentence):
|
||||
unclear.append('%s | %s' % (word['en'], sentence))
|
||||
|
||||
print(json.dumps(kept, indent=2))
|
||||
if unclear:
|
||||
print('%d sentences to eyeball:' % len(unclear))
|
||||
for line in unclear:
|
||||
print(' ' + line)
|
||||
if '--dry-run' in sys.argv:
|
||||
return
|
||||
bank['source'] = (
|
||||
'CEFR-J Vocabulary Profile 1.5 (levels) + 公开词书 (音标) + AI 重写 (释义与例句)'
|
||||
)
|
||||
with open(BANK, 'w') as handle:
|
||||
json.dump(bank, handle, ensure_ascii=False, separators=(',', ':'))
|
||||
handle.write('\n')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,131 @@
|
||||
"""Builds an independent word bank: CEFR-J decides which words and at what
|
||||
level, the downloaded word books supply the IPA and the Chinese gloss.
|
||||
|
||||
CEFR-J carries no glosses at all, so nothing here overrides it -- a word only
|
||||
enters the bank if the level pool already contains it.
|
||||
"""
|
||||
import csv, glob, json, os, re, sys
|
||||
import openpyxl
|
||||
|
||||
POOLS = '/Users/shenlei/Work/English/kouyu_english/tool/courses/pools'
|
||||
|
||||
# The books write parts of speech as `n.` / `vt.`; CEFR-J spells them out.
|
||||
POS_MAP = {
|
||||
'n': 'noun', 'pron': 'pronoun', 'v': 'verb', 'vt': 'verb', 'vi': 'verb',
|
||||
'aux': 'verb', 'modal': 'verb', 'adj': 'adjective', 'adv': 'adverb',
|
||||
'prep': 'preposition', 'conj': 'conjunction', 'art': 'determiner',
|
||||
'det': 'determiner', 'num': 'number', 'int': 'interjection',
|
||||
}
|
||||
|
||||
|
||||
def load_pool():
|
||||
"""{word form: (headword, pos set, level)} for A1..B1, content words only."""
|
||||
pool = {}
|
||||
for level in ('A1', 'A2', 'B1'):
|
||||
for row in csv.DictReader(open(f'{POOLS}/{level.lower()}-word-pool.csv')):
|
||||
if row.get('status') != 'pool' or row.get('function_word') == 'yes':
|
||||
continue
|
||||
head = row['headword'].strip()
|
||||
for form in re.split(r'[/,]', head):
|
||||
form = form.strip().lower()
|
||||
if not form:
|
||||
continue
|
||||
entry = pool.setdefault(form, {'headword': head, 'pos': set(), 'level': level})
|
||||
if row.get('pos'):
|
||||
entry['pos'].add(row['pos'].strip())
|
||||
return pool
|
||||
|
||||
|
||||
def clean_gloss(raw, wanted_pos):
|
||||
"""Picks one part of speech out of a stacked gloss and splits it into senses a
|
||||
beginner needs. The books ship dictionary dumps: `apple` arrives as
|
||||
"苹果，苹果树，苹果似的东西；[美俚]炸弹，手榴弹..."."""
|
||||
if not raw:
|
||||
return ''
|
||||
lines = [l.strip() for l in str(raw).split('\n') if l.strip()]
|
||||
picked = None
|
||||
for line in lines:
|
||||
m = re.match(r'^([a-z]+)\.\s*(.+)$', line, re.I)
|
||||
if not m:
|
||||
continue
|
||||
pos = POS_MAP.get(m.group(1).lower())
|
||||
if pos and wanted_pos and pos in wanted_pos:
|
||||
picked = m.group(2)
|
||||
break
|
||||
if picked is None:
|
||||
picked = m.group(2)
|
||||
if picked is None:
|
||||
picked = lines[0]
|
||||
senses = []
|
||||
for sense in re.split(r'[；;]', picked):
|
||||
sense = sense.strip()
|
||||
# `[美俚]炸弹` and `<美>支票` are register labels on a sense a learner
|
||||
# will never need; drop the whole sense, not just the label.
|
||||
if re.match(r'^\s*[\[<【]', sense):
|
||||
continue
|
||||
sense = re.sub(r'[（(][^）)]*[）)]', '', sense)
|
||||
sense = re.sub(r'[\[【][^\]】]*[\]】]', '', sense)
|
||||
# "苹果，苹果树，苹果似的东西" is one sense listed three ways.
|
||||
sense = re.split(r'[，,]', sense)[0].strip(' 。.、')
|
||||
if sense:
|
||||
senses.append(sense)
|
||||
return list(dict.fromkeys(senses))[:3]
|
||||
|
||||
|
||||
def main():
|
||||
pool = load_pool()
|
||||
bank, seen, skipped = {}, 0, 0
|
||||
for path in sorted(glob.glob('src/*.xlsx')):
|
||||
source = os.path.basename(path)
|
||||
wb = openpyxl.load_workbook(path, read_only=True)
|
||||
for row in wb.worksheets[0].iter_rows(min_row=2, values_only=True):
|
||||
if not row or not row[0]:
|
||||
continue
|
||||
seen += 1
|
||||
word = str(row[0]).strip()
|
||||
key = word.lower()
|
||||
hit = pool.get(key)
|
||||
if hit is None:
|
||||
skipped += 1
|
||||
continue
|
||||
senses = clean_gloss(row[3] if len(row) > 3 else '', hit['pos'])
|
||||
if not senses:
|
||||
skipped += 1
|
||||
continue
|
||||
uk = str(row[1] or '').strip()
|
||||
us = str(row[2] or '').strip()
|
||||
prev = bank.get(key)
|
||||
# Keep the first hit, but let a later book fill in a missing IPA.
|
||||
if prev:
|
||||
if not prev['ipa'] and (us or uk):
|
||||
prev['ipa'] = us or uk
|
||||
continue
|
||||
bank[key] = {
|
||||
'id': 'V-' + re.sub(r'[^a-z0-9]+', '-', key).strip('-'),
|
||||
'en': word,
|
||||
# One sense is what a quiz option should say; the rest are kept
|
||||
# for the word's detail line.
|
||||
'zh': senses[0],
|
||||
'more': '；'.join(senses[1:]),
|
||||
'ipa': us or uk,
|
||||
'level': hit['level'], 'source': source,
|
||||
}
|
||||
wb.close()
|
||||
|
||||
out = [bank[k] for k in sorted(bank)]
|
||||
json.dump(out, open('wordbank.json', 'w'), ensure_ascii=False, indent=1)
|
||||
|
||||
from collections import Counter
|
||||
by_level = Counter(w['level'] for w in out)
|
||||
print(f'读入词条 {seen}，未匹配/丢弃 {skipped}')
|
||||
print(f'词库产出 {len(out)} ' + ' '.join(f'{k} {by_level[k]}' for k in ('A1', 'A2', 'B1')))
|
||||
for level in ('A1', 'A2', 'B1'):
|
||||
total = len({f for f, v in pool.items() if v['level'] == level})
|
||||
print(f' {level} 词池覆盖 {by_level[level]}/{total} = {by_level[level]/total:.0%}')
|
||||
print(f' 无音标 {sum(1 for w in out if not w["ipa"])}')
|
||||
print('\n样例:')
|
||||
for w in out[:8]:
|
||||
print(f' {w["en"]:12} {w["ipa"]:18} {w["zh"]:10} [{w["level"]}] {w["more"]}')
|
||||
|
||||
|
||||
main()
|
||||
@@ -0,0 +1,171 @@
|
||||
"""Rewrites the Chinese gloss of every bank word and gives it an example
|
||||
sentence, using the AI service configured in assets/config/ai_config.json.
|
||||
|
||||
The downloaded word books are a dictionary dump: some entries carry only a rare
|
||||
sense (`ticket` as a verb and nothing else), some are unusable as a quiz answer.
|
||||
Cleaning rules cannot invent the missing sense, so the model rewrites the gloss
|
||||
outright, with the book's version passed only as a hint it may overrule.
|
||||
|
||||
Results are cached one JSON object per line, so an interrupted run resumes
|
||||
instead of paying for the same words twice.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
BANK = os.path.join(ROOT, 'assets/words/wordbank.json')
|
||||
CONFIG = os.path.join(ROOT, 'assets/config/ai_config.json')
|
||||
|
||||
# A word the model reasons about at length can truncate its whole batch; set
|
||||
# ENRICH_BATCH smaller to pick those stragglers up on a second run.
|
||||
BATCH = int(os.environ.get('ENRICH_BATCH', '12'))
|
||||
WORKERS = 8
|
||||
RETRIES = 3
|
||||
|
||||
SYSTEM = """你在为一个中国成年人用的英语学习 App 整理词库。对每个词输出：
|
||||
|
||||
- zh：最常用的那一个中文释义。2 到 6 个字，不要词性标注，不要括号，不要罗列同义词。
|
||||
- more：其他常用义，最多 2 个，用「；」分隔；没有别的常用义就给空字符串。
|
||||
- ex：一个英文例句。日常口语场景，6 到 12 个词，必须用到 zh 所指的那个义项，
|
||||
句中其他词不要超过这个词的 CEFR 等级。
|
||||
- ex_zh：例句的中文翻译，说人话，不要翻译腔。
|
||||
|
||||
输入里的「参考」来自一份质量不稳定的词书：有的只收了罕见义，有的缺了最常用的义项。
|
||||
不要盲从，以你自己对这个词最常用义的判断为准。
|
||||
|
||||
只输出 JSON 数组，每个元素形如
|
||||
{"id":"...","zh":"...","more":"...","ex":"...","ex_zh":"..."}
|
||||
不要输出任何别的文字，不要用 markdown 代码块。"""
|
||||
|
||||
|
||||
def load_config():
|
||||
with open(CONFIG) as handle:
|
||||
return json.load(handle)
|
||||
|
||||
|
||||
def call(config, prompt):
|
||||
"""One chat completion, through curl: this machine's Python does not trust
|
||||
the local certificate chain but curl does."""
|
||||
body = {
|
||||
'model': config['model'],
|
||||
'messages': [
|
||||
{'role': 'system', 'content': SYSTEM},
|
||||
{'role': 'user', 'content': prompt},
|
||||
],
|
||||
'temperature': 0.3,
|
||||
# The configured model reasons before it answers, and those tokens come
|
||||
# out of the same budget -- a tight cap truncates the JSON mid-array.
|
||||
'max_tokens': 8000,
|
||||
}
|
||||
with tempfile.NamedTemporaryFile('w', suffix='.json', delete=False) as payload:
|
||||
json.dump(body, payload, ensure_ascii=False)
|
||||
path = payload.name
|
||||
# The key goes through stdin, never argv, so it stays out of the process list.
|
||||
conf = (
|
||||
'url = "%s/v1/chat/completions"\n'
|
||||
'header = "Authorization: Bearer %s"\n'
|
||||
'header = "Content-Type: application/json"\n'
|
||||
'data-binary = "@%s"\n'
|
||||
'max-time = 300\n'
|
||||
) % (config['endpoint'].rstrip('/'), config['apiKey'], path)
|
||||
try:
|
||||
done = subprocess.run(
|
||||
['curl', '-sS', '--config', '-'], input=conf, capture_output=True, text=True
|
||||
)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
if done.returncode != 0:
|
||||
raise RuntimeError(done.stderr[:200])
|
||||
answer = json.loads(done.stdout)
|
||||
if 'choices' not in answer:
|
||||
raise RuntimeError(done.stdout[:200])
|
||||
choice = answer['choices'][0]
|
||||
if choice.get('finish_reason') == 'length':
|
||||
raise RuntimeError('truncated at max_tokens')
|
||||
return choice['message']['content'], answer.get('usage', {})
|
||||
|
||||
|
||||
def parse(text):
|
||||
"""The model is told to return bare JSON; strip a code fence when it adds one."""
|
||||
text = text.strip()
|
||||
if text.startswith('```'):
|
||||
text = text.split('\n', 1)[1].rsplit('```', 1)[0]
|
||||
start, end = text.find('['), text.rfind(']')
|
||||
if start < 0 or end < 0:
|
||||
raise ValueError(text[:200])
|
||||
return json.loads(text[start : end + 1])
|
||||
|
||||
|
||||
def ask(config, batch):
|
||||
lines = [
|
||||
'%s | %s | %s | 参考：%s'
|
||||
% (word['id'], word['en'], word['level'], '；'.join(
|
||||
filter(None, [word.get('zh', ''), word.get('more', '')])) or '（无）')
|
||||
for word in batch
|
||||
]
|
||||
text, usage = call(config, '\n'.join(lines))
|
||||
wanted = {word['id'] for word in batch}
|
||||
rows = [row for row in parse(text) if row.get('id') in wanted]
|
||||
return rows, usage
|
||||
|
||||
|
||||
def main():
|
||||
limit = int(sys.argv[1]) if len(sys.argv) > 1 else 0
|
||||
cache_path = os.environ.get('ENRICH_CACHE', os.path.join(tempfile.gettempdir(), 'enrich_cache.jsonl'))
|
||||
done = {}
|
||||
if os.path.exists(cache_path):
|
||||
with open(cache_path) as handle:
|
||||
for line in handle:
|
||||
row = json.loads(line)
|
||||
done[row['id']] = row
|
||||
with open(BANK) as handle:
|
||||
bank = json.load(handle)
|
||||
todo = [word for word in bank['words'] if word['id'] not in done]
|
||||
if limit:
|
||||
todo = todo[:limit]
|
||||
print('cached %d, to do %d' % (len(done), len(todo)), flush=True)
|
||||
|
||||
config = load_config()
|
||||
batches = [todo[i : i + BATCH] for i in range(0, len(todo), BATCH)]
|
||||
spent = {'in': 0, 'out': 0, 'fail': 0}
|
||||
with open(cache_path, 'a') as cache:
|
||||
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
|
||||
for index, result in enumerate(pool.map(lambda b: safe(config, b), batches)):
|
||||
rows, usage, error = result
|
||||
if error:
|
||||
spent['fail'] += 1
|
||||
print(' batch %d failed: %s' % (index, error), flush=True)
|
||||
continue
|
||||
for row in rows:
|
||||
cache.write(json.dumps(row, ensure_ascii=False) + '\n')
|
||||
cache.flush()
|
||||
spent['in'] += usage.get('prompt_tokens', 0)
|
||||
spent['out'] += usage.get('completion_tokens', 0)
|
||||
if index % 10 == 0:
|
||||
print(' %d/%d batches' % (index + 1, len(batches)), flush=True)
|
||||
print('tokens in %d out %d, failed batches %d' % (spent['in'], spent['out'], spent['fail']))
|
||||
print('cache: %s' % cache_path)
|
||||
|
||||
|
||||
def safe(config, batch):
|
||||
"""Retries a batch a few times; a truncated or malformed answer is common
|
||||
enough that giving up on the first one would leave holes everywhere."""
|
||||
last = 'no attempt'
|
||||
for attempt in range(RETRIES):
|
||||
try:
|
||||
rows, usage = ask(config, batch)
|
||||
if rows:
|
||||
return rows, usage, None
|
||||
last = 'empty result'
|
||||
except Exception as error:
|
||||
last = '%s: %s' % (type(error).__name__, error)
|
||||
return [], {}, last
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/bin/bash
|
||||
# Downloads the beginner-level word books. The repo is CC BY-NC-SA 4.0 and its
|
||||
# own README calls the contents "compiled from public sources, personal use
|
||||
# only" -- see the note in DATA-NOTE.md before shipping any of this.
|
||||
set -u
|
||||
BASE="https://github.com/lilinji/English/raw/main"
|
||||
enc() { python3 -c "import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1]))" "$1"; }
|
||||
get() {
|
||||
local path="$1" out="src/$2"
|
||||
[ -s "$out" ] && { echo " 已有 $2"; return; }
|
||||
curl -sL -o "$out" "$BASE/$(enc "$path")"
|
||||
if [ -s "$out" ] && head -c2 "$out" | grep -q PK; then echo " 取得 $2 ($(wc -c <"$out") 字节)"
|
||||
else echo " 失败 $2"; rm -f "$out"; fi
|
||||
}
|
||||
echo "剑桥 KET/PET:"
|
||||
get "9.其他（更多）/14天攻克KET核心词汇.xlsx" ket-core.xlsx
|
||||
get "9.其他（更多）/KET核心词 巧记速练.xlsx" ket-drill.xlsx
|
||||
get "9.其他（更多）/21天攻克PET核心词汇.xlsx" pet-core.xlsx
|
||||
get "9.其他（更多）/PET核心词 巧记速练.xlsx" pet-drill.xlsx
|
||||
get "9.其他（更多）/突破英文基础词汇.xlsx" basic.xlsx
|
||||
echo "人教版小学:"
|
||||
for g in 一 二 三 四 五 六; do
|
||||
for t in 上 下; do
|
||||
get "1.全国各大教材版本中小学同步/人教版/人教版一年级起点${g}年级${t}.xlsx" "rj-p1-${g}${t}.xlsx"
|
||||
done
|
||||
done
|
||||
echo "人教版初中(七年级):"
|
||||
get "1.全国各大教材版本中小学同步/人教版/人教版初中英语七年级上册.xlsx" rj-m7a.xlsx
|
||||
get "1.全国各大教材版本中小学同步/人教版/人教版初中英语七年级下册.xlsx" rj-m7b.xlsx
|
||||
|
||||
# Word frequency, for the order new words are handed out in. These are
|
||||
# OpenSubtitles counts, i.e. spoken language: a web corpus ranks `ankle` and
|
||||
# `asleep` far below where a learner of spoken English needs them.
|
||||
# hermitdave/FrequencyWords is MIT, a cleaner licence than the word books above.
|
||||
echo "词频表:"
|
||||
if [ -s src/en_50k.txt ]; then
|
||||
echo " 已有 en_50k.txt"
|
||||
else
|
||||
curl -sL -o src/en_50k.txt \
|
||||
"https://raw.githubusercontent.com/hermitdave/FrequencyWords/master/content/2018/en/en_50k.txt"
|
||||
echo " 取得 en_50k.txt ($(wc -l <src/en_50k.txt) 行)"
|
||||
fi
|
||||
Reference in New Issue
Block a user