Compare commits

...
10 Commits
Author SHA1 Message Date
iacore 07ebbc90c6 ++ 2026-08-11 11:23:43 +08:00
iacore 6144d63b68 Add translation check gate and herdr batch workflow docs
check-translation.py: 11-check deterministic gate (line/paragraph/heading
parity, emphasis preservation, CJK and Chinese-punctuation leakage, bold
leakage, 万/亿-aware digit fidelity, terminology vs term map, bilingual
freshness). Built and calibrated on the 9-book review batch (2026-08-07).

AGENTS.md: add Workflow C (batch translate/review with herdr) — pane setup,
the three canonical prompts (translate/review/apply), and gotchas learned on
the 9-book run.

readme.md: document the gate in common tasks and point to Workflow C.
2026-08-08 12:53:48 +08:00
iacore da0ee57834 docs+skills: preserve speaker's voice (语气), not just tone
Reviewer feedback: translations must carry Master Jiqun's 语气, not only
register. Add a 'Speaker's Voice (语气)' section to mpi-translation and
expand check D3 in mpi-translation-review. Also note the idea of distilling
his way of talk into a dedicated skill (docs/idea-jiqun-voice-skill.md).
2026-07-20 14:55:21 +08:00
iacore ee15159856 chore(terms-database): declare flask dependency via uv script metadata
Convert server.py from a plain Python shebang to a uv-managed script

so users can run it directly without manually installing flask first.
2026-07-16 09:48:35 +08:00
iacore 40abd7abe3 docs: add README and align skills readme for Oh My Pi
Add a workspace setup guide for new translation volunteers so they can

clone the public toolkit, load skills, and run a first project without

guessing the directory layout. Update skills/readme.dj to remove the

Hermes-specific instructions and describe generic agent loading so it

works for Oh My Pi users.
2026-07-16 09:48:35 +08:00
iacore ef7deb36d8 add(skills): Simplify state machine 2026-07-15 13:09:42 +08:00
iacore bc17471635 feat(skills): make skill scripts self-executing with uv hashbang
The Python scripts in mpi-pptx-translate, mpi-pdf-to-docx-conversion, and
mpi-chinese-text-normalize previously required users to type
manually. That is easy to forget and adds friction every time the skill runs.

Switch all four scripts to a  shebang,
so they can be invoked directly: . The
 PEP 723 metadata blocks remain, so uv still installs the
dependencies automatically. Also made the scripts executable and updated
their usage messages and SKILL.md instructions to match the direct-execution
style.
2026-07-15 13:05:34 +08:00
iacore 894d769051 refactor(skills): align MPI skills with Agent Skills best practices
The skill metadata had drifted: every SKILL.md name field lacked the
mpi- prefix, contradicting the directory names and the Agent Skills
specification. Descriptions were also missing negative triggers, making
it easy for the agent to load the wrong skill.

Rewrote the pdf-to-docx skill to follow progressive disclosure: the main
SKILL.md dropped from 318 lines to 80, with detailed code examples moved
to on-demand references. Added uv run instructions and /// script PEP 723
metadata so dependencies are declared inline and installed automatically.
Fixed the pptx skill's script paths and added CLI usage messages to both
pptx scripts and the normalize script.

Removed the empty self-review directory that was superseded by the unified
translation-review skill.
2026-07-14 23:18:55 +08:00
iacore 216a7658ac Migrate from duckdb to sqlite 2026-07-14 11:57:53 +08:00
iacore 8bf19a8d18 Remove MPI_PROJECT_ROOT mentions 2026-07-14 11:22:54 +08:00
32 changed files with 1427 additions and 457 deletions
+102 -31
View File
@@ -39,41 +39,35 @@ Generated files (`bilingual.dj`) are not committed either.
## Translation State Machine
All translation work follows this deterministic workflow. Non-deterministic LLM work (drafting, reviewing) happens at the edges; the states and transitions are fixed.
Non-deterministic LLM work happens at the states; transitions are fixed.
```mermaid
stateDiagram-v2
[*] --> idle
idle --> translating: SOURCE_LOADED
idle --> other_reviewing: BILINGUAL_LOADED
translating --> bilingual_ready: TRANSLATION_DRAFTED
bilingual_ready --> self_reviewing: BILINGUAL_GENERATED
self_reviewing --> translating: SELF_REJECTED
self_reviewing --> other_reviewing: SELF_APPROVED [peer review required]
self_reviewing --> approved: SELF_APPROVED [no peer review]
note right of self_reviewing
peer_review_required flag decides the branch
end note
other_reviewing --> translating: PEER_REJECTED
other_reviewing --> approved: PEER_APPROVED
approved --> typesetting: TYPESET_REQUESTED
approved --> done: COMPLETE
typesetting --> done: TYPESET_COMPLETE
done --> [*]
```
| Current state | Event / condition | Next state | Notes |
|---|---|---|---|
| `*start*` | source loaded | `idle` | Begin from a new source. |
| `*start*` | bilingual loaded | `idle` | Begin from an existing review file. |
| `idle` | `SOURCE_LOADED` | `translating` | |
| `idle` | `BILINGUAL_LOADED` | `other_reviewing` | |
| `translating` | `TRANSLATION_DRAFTED` | `bilingual_ready` | |
| `bilingual_ready` | `BILINGUAL_GENERATED` | `self_reviewing` | |
| `self_reviewing` | `SELF_REJECTED` | `translating` | |
| `self_reviewing` | `SELF_APPROVED` and peer review required | `other_reviewing` | `peer_review_required` flag decides the branch. |
| `self_reviewing` | `SELF_APPROVED` and no peer review | `approved` | |
| `other_reviewing` | `PEER_REJECTED` | `translating` | |
| `other_reviewing` | `PEER_APPROVED` | `approved` | |
| `approved` | `TYPESET_REQUESTED` | `typesetting` | Optional. |
| `approved` | `COMPLETE` | `done` | |
| `typesetting` | `TYPESET_COMPLETE` | `done` | |
States:
| State | Meaning | Output artifact |
|---|---|---|
| `idle` | Waiting for source or an existing bilingual file. | — |
| `translating` | Agent loads `mpi-translation` + `mpi-terms-search` skills and drafts `target.dj`. | `target.dj` |
| `bilingual_ready` | `bilingual.dj` generated from `source.dj` + `target.dj`. | `bilingual.dj` |
| `self_reviewing` | Self-review with unified ruleset (self mode). Edit target.dj. | `target.dj` (edited) |
| `other_reviewing` | Peer review with unified ruleset (other mode). Write review-comments.dj. | `review-comments.dj` |
| `approved` | Translation accepted. May typeset or finish. | — |
| `typesetting` | Producing PDF/DOCX from approved bilingual content. | `.pdf` / `.docx` |
| `done` | Complete. | — |
: `idle` — Waiting for source or an existing bilingual file.
: `translating` — Draft `target.dj`.
: `bilingual_ready``bilingual.dj` generated from `source.dj` + `target.dj`.
: `self_reviewing` — Self-review with `mpi-translation-review` (self mode); edit `target.dj`.
: `other_reviewing` — Peer review with `mpi-translation-review` (other mode); write `review-comments.dj`.
: `approved` — Translation accepted; may typeset or finish.
: `typesetting` Produce PDF/DOCX.
: `done` — Complete.
## Workflow A: Translation(翻译)
@@ -165,6 +159,82 @@ else?" after a direct-edit pass, do one more full systematic read and batch agai
---
## Workflow C: Batch Translate / Review with Herdr(批量翻译/审阅)
Run one book/article per omp session in its own herdr pane. Proven on the
9-book batch (2026-08-07): 9 review panes (`omp --model slow`), each moved to
its own tab, then one apply pass per pane, all 9 green on the check suite.
### Setup
1. One herdr workspace; the main omp session in the root pane orchestrates.
2. Rename each book dir with a shared batch prefix so they sort and zip
together: `batch0-21【《心经》的人生智慧】`, `batch0-55【…】`, …
3. Split one pane per book. `herdr pane split --cwd <dir>` does NOT stick
(panes launch in the workspace root) — pass the absolute book dir to omp's
own `--cwd` at launch instead.
```fish
# 5 right of the main pane, then 4 below it
r=$(herdr pane split --current --direction right --no-focus)
p=$(echo "$r" | jq -r '.result.pane.pane_id')
# repeat: herdr pane split --pane $prev --direction down --no-focus
herdr pane rename <pane> review-<book> # label each pane
```
4. Launch the slow model in every pane (background all, then `wait`):
```fish
herdr pane run w7:p2 "omp --model slow --cwd /abs/path/to/book-dir" &
# ... one line per book
wait
```
Verify each pane landed in its book dir:
`herdr pane read <pane> --source recent-unwrapped --lines 8` — the TUI title
shows the dir.
5. Submit the review prompt (B below) to all panes at once. Poll
`herdr pane get <pane> | jq -r '.result.pane.agent_status'` until every
pane is `idle`/`done` (allow ~1 h for long books).
6. Move each pane to its own tab so the batch is watchable while it runs:
`herdr pane move <pane> --new-tab --label <name> --no-focus`.
7. After all reviews finish, submit the apply prompt (C below) to every pane
again, poll to completion, then gate with `check-translation.py` and
spot-check that fixes actually landed in `target.dj`.
8. Package: `zip -r <batch>-reviewed.zip batch0-*/` and verify the entry count
(9 books × 6 files each = 63 entries).
### The three prompts
**A — Translate a book** (one session per book, Workflow A):
> Translate the book <NAME> (file: <NAME>.docx) from Chinese to English for the MPI translation project. Follow Workflow A in ../../toolkit/AGENTS.md: (1) load the mpi-translation and mpi-terms-search skills from ../../toolkit/skills/; (2) extract the Chinese source with ../../toolkit/scripts/docx2dj.fish '<NAME>.docx' into source.dj; (3) translate the ENTIRE book into target.dj (English; line count matches source; you ARE the model — no external translation APIs; look up key Buddhist terms with ../../toolkit/terms-database/search.py); (4) generate bilingual.dj: ../../toolkit/scripts/gen-bilingual.py source.dj target.dj > bilingual.dj; (5) self-review with mpi-translation-review (self mode), edit target.dj, and write edit-suggestions.dj for terminology issues; (6) regenerate bilingual.dj and verify source/target line counts match. Deliverables in this folder: source.dj, target.dj, bilingual.dj, edit-suggestions.dj. Do not commit binaries or bilingual.dj. Report when done.
**B — Review a book** (herdr pane, slow model; writes `review-findings.dj` only):
> Review the translation in this directory (your cwd is the book dir). Files: source.dj (Chinese source), target.dj (English translation), bilingual.dj (bilingual), edit-suggestions.dj (prior edit suggestions, may be stale). Read source and target fully and review the English translation for: (1) accuracy vs source — mistranslations, omissions, additions, meaning drift; (2) Buddhist terminology — consistent, standard renderings; (3) fluency and register — natural, idiomatic English appropriate to the genre; (4) completeness — every source section covered. Write findings to review-findings.dj in this directory, organized by severity (critical/major/minor), each with location and a concrete fix. Do NOT modify source.dj, target.dj, or bilingual.dj. End your final message with a one-paragraph summary.
**C — Apply findings** (same pane, direct-edit mode):
> Apply your review findings now. This is direct-edit mode per project convention. 1) Read review-findings.dj and target.dj fully. 2) Apply EVERY actionable finding (all must-fix and considerations) to target.dj with exact-string replacements, batched in one pass. 3) CRITICAL: do not add or remove any line — source.dj and target.dj line counts must remain identical. 4) Regenerate bilingual.dj: <abs path>/gen-bilingual.py source.dj target.dj > bilingual.dj 5) Run <abs path>/check-translation.py . and report which checks pass/fail. Report what you changed and the check result.
### Gotchas(踩过的坑)
- `herdr pane split --cwd <dir>` doesn't stick — launch omp with `--cwd <absolute path>` instead.
- Write the poll loop carefully: wait for `agent_status` to reach `idle`/`done` with a deadline. The naive first version inverted the logic and reported "done" instantly.
- Line-count parity is load-bearing: `gen-bilingual.py` pairs lines by index and the check gate FAILs on drift. Apply fixes with exact-string replacements; never insert or delete lines (a blank-line fix was deliberately skipped in the batch for exactly this reason).
- `check-translation.py` calibration facts (all learned on the 9-book run):
- CJK leakage ignores djot anchors/links (`{#...}`, `(...)`) — structural markup legitimately contains Chinese.
- Only strictly-Chinese punctuation flags (`,。、;:?!《》【】()`); `—` `“”` `` `·` are legitimate English.
- Emphasis = preservation on the same line (source `*…*` must survive in target), not count parity — targets legitimately add italics for titles/Sanskrit.
- Digit fidelity understands 万/亿 scaling, 多, word and comma forms (180亿 → "18 billion", 1300多万 → "13+ million"), and excludes TOC page numbers (`[N](#...)`), which the convention drops.
- Use `--allow-cjk 人` for intentional Chinese (e.g. a character whose strokes the text explains) and `--term-map term-map.md` to check terminology fidelity; without a term map that check is skipped.
- `--model slow` is omp's model-role flag for the slow/reasoning model; confirm the exact flag with `omp --help` if unsure.
- Reviews are the quality gate: the apply pass is what lands findings in `target.dj` (book 21 alone took 26 exact-string replacements). The gate proves mechanics, not quality.
---
## Djot
- Comments: `{% ... %}`
@@ -196,6 +266,7 @@ regenerating the same Python in execute_code each turn.
- `toolkit/scripts/dj2docx.fish <target.dj>` — pandoc .dj → .docx in `/tmp/`
- `toolkit/scripts/proofread-pdf.py <docx> <pdf>` — word-level diff between manuscript and typeset PDF
- `toolkit/scripts/gen-bilingual.py <source.dj> <target.dj>` — produce `bilingual.dj` on stdout; run as `gen-bilingual.py source.dj target.dj > bilingual.dj`
- `toolkit/scripts/check-translation.py <book_dir>` — deterministic translation gate: line/paragraph/heading parity, emphasis preservation, CJK & Chinese-punctuation leakage, digit fidelity (万/亿-aware), terminology vs term map, bilingual freshness. Exit 1 on any FAIL. Run before delivering a translation; keep green as a regression suite.
- `toolkit/scripts/gen-bilingual-<name>-<hash>.py` — article-specific extraction from DOCX or source/target pairing
- `toolkit/scripts/compile-typst.fish <typ>` — compile a Typst file to PDF
+42
View File
@@ -0,0 +1,42 @@
# Idea: distill Master Jiqun's way of talk (语气) into a skill
Date: 2026-07-20
Status: idea / not scheduled
## Background
Reviewer feedback on a translated article: translations must preserve Master
Jiqun's 语气 — the speaker's voice and manner — not just the tone (register,
formality). A translation can have the right register and still flatten the
teacher's voice.
As a first step, a "Speaker's Voice (语气)" section was added to
`skills/mpi-translation/SKILL.md`, and check D3 in
`skills/mpi-translation-review/SKILL.md` was expanded to verify 语气 markers.
That covers the immediate need, but the knowledge is currently a short bullet
list inside a general skill.
## Idea
Distill Master Jiqun's way of talk into a dedicated skill (or a reference doc
under `skills/mpi-translation/references/`), built from evidence across many
of his talks:
- Collect characteristic passages (source + approved translations) from the
terms DB (`loc` fields) and finished articles in `translate-files/`.
- Catalog his recurring devices: rhetorical question chains, first-person
asides (我经常说 / 我曾在讲座中 / 由此我想到), reasoning connectives
(可见 / 所以说 / 事实上), homely analogies (rotting apple, leaking boat,
teacup, face mask), gentle admonition with humor, measured unhurried
authority, inclusive we/you address.
- For each device, give approved English renderings and common failure modes
(e.g. question → declaration, "我经常说" → "it is said").
- Possibly also cover delivery/register differences between his oral talks,
essays, and micro-blog posts.
## Open questions
- Standalone skill vs. reference doc under `mpi-translation`? A reference doc
is probably enough; a full skill may be overkill.
- Who approves the example translations used as evidence?
- Should it also cover other MPI teachers, or stay Jiqun-specific?
+119
View File
@@ -0,0 +1,119 @@
# MPI Translation Toolkit
This is the public translation toolkit for Mindful Peace International (MPI).
It contains agent skills, a Buddhist/Dharma term database, helper scripts, and
project conventions. You clone it into a private workspace and work on your
translations in sibling directories.
## Who this is for
Volunteers who want to translate MPI articles using the Oh My Pi agent
workflow. You do not need to know Oh My Pi yet; this guide covers the setup.
## Workspace layout
Create a workspace directory and clone this repo into `toolkit/` inside it.
Your translations and reference materials live next to `toolkit/`, not inside
it. Treat `toolkit/` as read-only unless you are intentionally contributing to
the toolkit itself.
```text
mpi-workspace/
├── toolkit/ ← this repository (public, read-only for translators)
│ ├── skills/
│ ├── terms-database/
│ ├── scripts/
│ └── AGENTS.md
├── translate-files/ ← your translation projects
└── references/ ← your private reference materials
```
Example:
```sh
mkdir ~/mpi-workspace
cd ~/mpi-workspace
git clone https://codeberg.org/eastwind/translation-toolkit.git toolkit
mkdir translate-files references
```
## Dependencies
Install these before you start:
- **Oh My Pi** — the agent harness you will talk to.
- **Python 3** and **uv** — used by the scripts.
- **fish** — the scripts in `toolkit/scripts/` are written for fish.
- **pandoc** — converts between `.docx` and `.dj` (djot).
- **typst** — compiles bilingual layouts to PDF.
## Load the skills
Configure your agent to load skills from the `toolkit/skills/` directory. The
skills use the standard Agent Skills format. If your agent supports an
`external_dirs`-style skill path, add the absolute path:
```yaml
skills:
external_dirs:
- /home/yourname/mpi-workspace/toolkit/skills
```
Replace `/home/yourname/mpi-workspace` with the actual absolute path to your
workspace.
## First walkthrough
Use the shortest existing article to verify your setup:
```sh
cd ~/mpi-workspace/toolkit
ls ../translate-files/展示
# source.dj target.dj
```
A minimal project has these files:
```text
../translate-files/展示/
├── source.dj ← Chinese original, one paragraph per line
└── target.dj ← English translation, matching source line count
```
Generate the bilingual file:
```sh
./scripts/gen-bilingual.py \
../translate-files/展示/source.dj \
../translate-files/展示/target.dj \
> ../translate-files/展示/bilingual.dj
```
The output interleaves source, target, and blank lines. `bilingual.dj` is
generated; do not edit it by hand or commit it.
## Common tasks
| Task | Command |
|---|---|
| Convert `.docx` to `.dj` | `./scripts/docx2dj.fish input.docx output.dj` |
| Convert `target.dj` to `.docx` | `./scripts/dj2docx.fish ../translate-files/展示/target.dj` |
| Compile a Typst file to PDF | `./scripts/compile-typst.fish ../translate-files/my-article/my-article.typ` |
| Run the translation gate | `./scripts/check-translation.py ../translate-files/my-article/` — 11 deterministic checks; exit 1 on any FAIL |
| Search the term database | `./terms-database/search.py 空性` |
| Run the term database UI | `./terms-database/server.py` then open <http://127.0.0.1:8910> |
## Sharing your work
Keep your workspace private. Do not share translation files through this
repository. Share finished translations with the team through WeChat or Google
Docs, as arranged by your project coordinator.
## Learn more
- `AGENTS.md` — full MPI project conventions, translation workflows, review rules, and Workflow C: the herdr batch workflow for translating or reviewing many books in parallel (one omp pane per book).
- `skills/readme.dj` — how the skills are organized.
- `references/` — design notes, formatting guides, and other reference materials.
If you improve the toolkit itself, contributions back to the Codeberg repo are
welcome. If you are only translating, leave `toolkit/` unchanged.
+411
View File
@@ -0,0 +1,411 @@
#!/usr/bin/env python3
"""Deterministic translation checks for the MPI project (TDD-style gate).
Checks the mechanical invariants of a Chinese->English djot translation.
Semantic quality (fluency, register, tone) is NOT checked here — that is the
LLM review layer. This script is the hard gate: it must go green before a
translation is delivered, and it stays in toolkit/scripts/ as a regression
suite so later edits cannot silently break parity.
Severity:
FAIL — hard invariant broken; gate is red.
WARN — possible drift worth a reviewer's eye; does not fail the gate.
PASS — clean.
Checks:
1. line-count parity — source.dj and target.dj must have equal lines
2. paragraph parity — equal number of blank-separated blocks
3. heading parity — equal # of lines starting with each heading level
4. emphasis preservation — every *...* in a source line survives on the
same-index target line (D5). Target may ADD
italics (titles, Sanskrit) — that is fine.
5. comment parity — equal {% ... %} blocks (D5)
6. CJK leakage — no Chinese characters in target (whitelistable)
7. Chinese punctuation — no strictly-Chinese punctuation in target
(,。、;:?!《》【】()). Em dash, curly
quotes, middot are legal English — not flagged.
8. bold leakage — no Markdown ** in target (D5)
9. digit fidelity — every arabic number in source appears in target
10. terminology — source terms from a term map must appear in target
with an allowed English rendering (A3 / terms DB).
FAIL: term present in source, none of its
renderings found in target. WARN: renderings
found but fewer times than the source term.
11. bilingual freshness — bilingual.dj, if present, equals a regeneration
from source+target
Usage:
check-translation.py <book_dir>
Uses <book_dir>/source.dj and <book_dir>/target.dj; auto-detects
bilingual.dj and term-map.md in the same directory.
check-translation.py <source.dj> <target.dj> [--bilingual FILE]
Explicit files.
Options:
--term-map FILE Term map (default: <book_dir>/term-map.md if present).
Accepted formats:
- markdown table rows: | 菩提心 | bodhicitta |
- plain lines: CN<TAB>EN or CN|EN1|EN2
Multiple Chinese terms separated by "/" or "" share
one English side; English renderings separated by "/"
are alternatives, any of which satisfies the check.
--allow-cjk LIST Comma-separated CJK strings permitted in target
(e.g. quoted book titles like 《心经》).
--json Emit machine-readable JSON results.
Exit code: 0 when no check FAILs (WARNs allowed), 1 otherwise.
"""
import argparse
import json
import re
import sys
from pathlib import Path
CJK_RE = re.compile(r"[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]")
# Strictly-Chinese punctuation only. Em dash (—), curly quotes (“” ‘’),
# middot (·), and ellipsis are legitimate in English prose.
CN_PUNCT_RE = re.compile(r"[,。、;:?!《》【】()]")
EMPHASIS_RE = re.compile(r"\*[^*\n]+\*")
COMMENT_RE = re.compile(r"\{%[\s\S]*?%\}")
HEADING_RE = re.compile(r"^(#{1,6})(?:\s|$)")
BOLD_RE = re.compile(r"\*\*")
DIGIT_RE = re.compile(r"\d+")
PASS, WARN, FAIL = "PASS", "WARN", "FAIL"
def read_lines(path):
return Path(path).read_text(encoding="utf-8").splitlines()
def count_paras(lines):
"""Blank-separated blocks; leading/trailing blanks ignored."""
count = 0
in_block = False
for line in lines:
if line.strip():
if not in_block:
count += 1
in_block = True
else:
in_block = False
return count
def heading_counts(lines):
counts = {i: 0 for i in range(1, 7)}
for line in lines:
m = HEADING_RE.match(line)
if m:
counts[len(m.group(1))] += 1
return counts
def term_map_from_markdown(text):
"""Parse a term map into {chinese_term: [english_renderings]}."""
terms = {}
for raw in text.splitlines():
line = raw.strip()
if not line or line.startswith("#"):
continue
if "\t" in line:
cn, _, en = line.partition("\t")
terms[cn.strip()] = [r.strip() for r in en.split("/") if r.strip()]
continue
if "|" not in line:
continue
cells = [c.strip() for c in line.strip("|").split("|")]
if len(cells) < 2:
continue
cn_cell, en_cell = cells[0], cells[1]
if not cn_cell or not en_cell or set(cn_cell) <= {"-", " "}:
continue
for cn in (c.strip() for c in re.split(r"[/、]", cn_cell) if c.strip()):
terms[cn] = [r.strip() for r in en_cell.split("/") if r.strip()]
return terms
def check_line_count(src, tgt):
ok = len(src) == len(tgt)
return (PASS if ok else FAIL,
f"source={len(src)} target={len(tgt)}", [])
def check_paras(src, tgt):
s, t = count_paras(src), count_paras(tgt)
return (PASS if s == t else FAIL,
f"source={s} target={t}", [])
def check_headings(src, tgt):
s, t = heading_counts(src), heading_counts(tgt)
diffs = [f"H{i}: {s[i]} vs {t[i]}" for i in range(1, 7) if s[i] != t[i]]
return (PASS if not diffs else FAIL,
"; ".join(diffs) if diffs else "all levels match", diffs)
def check_emphasis(src, tgt):
"""D5 preservation: source emphasis must survive on the same line.
Target may add emphasis for titles/Sanskrit, so counts need not match.
"""
missing = []
for i, (s, t) in enumerate(zip(src, tgt), 1):
if EMPHASIS_RE.search(s) and not EMPHASIS_RE.search(t):
missing.append((i, s, t))
return (PASS if not missing else FAIL,
"all source emphasis preserved"
if not missing else
f"{len(missing)} source line(s) lost emphasis: "
+ ", ".join(f"S{i}" for i, _, _ in missing[:10]),
missing)
def check_comments(src, tgt):
s, t = len(COMMENT_RE.findall("\n".join(src))), len(COMMENT_RE.findall("\n".join(tgt)))
return (PASS if s == t else FAIL,
f"source={s} target={t}", [])
ANCHOR_RE = re.compile(r"\{#[^{}]*\}|\(\s*#?[^{}\n]*\)")
LINK_TGT_RE = re.compile(r"\[\d+\]\(\s*#")
def strip_structural(text):
"""Remove djot anchors {#...}, link destinations (...), and image paths —
structural markup that may legitimately contain Chinese."""
return ANCHOR_RE.sub("", text)
def check_cjk(tgt, allow=()):
bad = []
for i, line in enumerate(tgt, 1):
stripped = strip_structural(line)
for token in allow:
stripped = stripped.replace(token, "")
if CJK_RE.search(stripped):
bad.append((i, line))
return (PASS if not bad else FAIL,
"clean" if not bad else f"{len(bad)} line(s) contain CJK outside anchors/links: "
+ ", ".join(f"L{i}" for i, _ in bad[:10]),
bad)
def check_cn_punct(tgt):
bad = []
for i, line in enumerate(tgt, 1):
if CN_PUNCT_RE.search(line):
bad.append((i, line))
return (PASS if not bad else FAIL,
"clean" if not bad else f"{len(bad)} line(s) contain Chinese punctuation: "
+ ", ".join(f"L{i}" for i, _ in bad[:10]),
bad)
def check_bold(tgt):
bad = []
for i, line in enumerate(tgt, 1):
if BOLD_RE.search(line):
bad.append((i, line))
return (PASS if not bad else FAIL,
"clean" if not bad else f"{len(bad)} line(s) contain ** : "
+ ", ".join(f"L{i}" for i, _ in bad[:10]),
bad)
_ONES = ["", "one", "two", "three", "four", "five", "six", "seven", "eight",
"nine", "ten", "eleven", "twelve", "thirteen", "fourteen", "fifteen",
"sixteen", "seventeen", "eighteen", "nineteen"]
_TENS = ["", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy",
"eighty", "ninety"]
def number_to_words(n):
if n < 20:
return _ONES[n]
if n < 100:
return (_TENS[n // 10] + ("-" + _ONES[n % 10] if n % 10 else ""))
if n < 1000:
return _ONES[n // 100] + " hundred" + (
(" " + number_to_words(n % 100)) if n % 100 else "")
if n < 1000000:
return number_to_words(n // 1000) + " thousand" + (
(" " + number_to_words(n % 1000)) if n % 1000 else "")
return number_to_words(n // 1000000) + " million" + (
(" " + number_to_words(n % 1000000)) if n % 1000000 else "")
def accepted_number_spellings(n, unit):
"""All English spellings that legitimately render source number n.
unit: "" (plain), "" (×10^4), or "亿" (×10^8). The target may keep the
digits ("1,200"), spell them ("twelve hundred"), or scale the unit
("13 million" for 1300万, "18 billion" for 180亿).
"""
cands = {str(n), number_to_words(n)}
if 100 <= n < 10000 and n % 100 == 0: # "twelve hundred"
cands.add(f"{n // 100} hundred")
cands.add(number_to_words(n // 100) + " hundred")
value = n * (10 ** 4 if unit == "" else 10 ** 8 if unit == "亿" else 1)
if value != n:
cands.add(str(value))
cands.add(number_to_words(value))
for divisor, suffix in ((10 ** 9, "billion"), (10 ** 6, "million"),
(10 ** 3, "thousand")):
if value % divisor == 0 and value // divisor > 0:
cands.add(f"{value // divisor} {suffix}")
cands.add(number_to_words(value // divisor) + " " + suffix)
return cands
def source_content_nums(src_text):
"""(number, unit) pairs from the source, excluding TOC page numbers
([N](#...)) that the project convention intentionally drops."""
stripped = LINK_TGT_RE.sub("", src_text)
out = []
for m in re.finditer(r"\d+(?:\s*(?:多\s*)?[万亿])?", stripped):
tok = m.group(0)
unit = tok[-1] if tok[-1] in "万亿" else ""
out.append((int(re.sub(r"\D", "", tok)), unit))
return out
def check_digits(src, tgt):
src_nums = source_content_nums("\n".join(src))
tgt_text = " ".join(tgt).lower().replace(",", "")
missing = []
for n, unit in src_nums:
if any(s.lower() in tgt_text
for s in accepted_number_spellings(n, unit)):
continue
missing.append(str(n) + unit)
return (PASS if not missing else FAIL,
"all present" if not missing else f"missing in target: {', '.join(missing)}",
missing)
def check_terminology(src, tgt, terms):
"""A3: source term present -> some allowed rendering present in target.
FAIL when no rendering is found at all; WARN when found but under-counted
(inflections, line wraps, or a genuine drift the reviewer should verify).
"""
if not terms:
return PASS, "no term map provided; skipped", []
src_text = "\n".join(src)
tgt_text = " ".join(tgt).lower()
fails, warns = [], []
checked = 0
for cn, renderings in sorted(terms.items()):
n = src_text.count(cn)
if n == 0:
continue
checked += 1
hits = sum(tgt_text.count(r.lower()) for r in renderings)
if hits == 0:
fails.append(f"{cn} ({n}× in source) — none of {renderings} found in target")
elif hits < n:
warns.append(f"{cn} ({n}× in source, {hits}× rendered) — verify")
if fails:
status, detail = FAIL, f"{checked} term(s) checked; " + "; ".join(fails)
elif warns:
status, detail = WARN, f"{checked} term(s) checked; " + "; ".join(warns)
else:
status, detail = PASS, f"{checked} term(s) checked; all consistent"
return status, detail, fails + warns
def check_bilingual(src, tgt, bilingual_path):
if bilingual_path is None or not Path(bilingual_path).exists():
return PASS, "no bilingual.dj present; skipped", []
actual = Path(bilingual_path).read_text(encoding="utf-8").splitlines()
expected = []
for s, t in zip(src, tgt):
if s == "":
expected.append("")
else:
expected.extend([s, t, ""])
if expected and expected[-1] != "":
expected.append("")
if actual == expected:
return PASS, f"{len(actual)} lines match a regeneration", []
return FAIL, f"stale: {Path(bilingual_path)} differs from source+target regeneration", []
def run_checks(src, tgt, bilingual=None, term_map=None, allow_cjk=()):
return [
("line-count parity", *check_line_count(src, tgt)),
("paragraph parity", *check_paras(src, tgt)),
("heading parity", *check_headings(src, tgt)),
("emphasis preservation", *check_emphasis(src, tgt)),
("comment parity", *check_comments(src, tgt)),
("CJK leakage", *check_cjk(tgt, allow_cjk)),
("Chinese punctuation", *check_cn_punct(tgt)),
("bold leakage", *check_bold(tgt)),
("digit fidelity", *check_digits(src, tgt)),
("terminology", *check_terminology(src, tgt, term_map)),
("bilingual freshness", *check_bilingual(src, tgt, bilingual)),
]
def main():
ap = argparse.ArgumentParser(description="Deterministic translation checks")
ap.add_argument("paths", nargs="+", help="book dir, or source.dj target.dj")
ap.add_argument("--bilingual", default=None, help="bilingual.dj to verify")
ap.add_argument("--term-map", default=None, help="term map file")
ap.add_argument("--allow-cjk", default="", help="comma-separated CJK whitelist")
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
if len(args.paths) == 1 and Path(args.paths[0]).is_dir():
d = Path(args.paths[0])
src, tgt = d / "source.dj", d / "target.dj"
bilingual = args.bilingual or d / "bilingual.dj"
term_map = args.term_map or d / "term-map.md"
label = str(d)
elif len(args.paths) == 2:
src, tgt = Path(args.paths[0]), Path(args.paths[1])
bilingual = Path(args.bilingual) if args.bilingual else None
term_map = Path(args.term_map) if args.term_map else None
label = f"{src} -> {tgt}"
else:
ap.error("pass a book directory, or source.dj target.dj")
if not src.exists() or not tgt.exists():
ap.error(f"missing source or target: {src} / {tgt}")
src_lines = read_lines(src)
tgt_lines = read_lines(tgt)
allow = [t for t in args.allow_cjk.split(",") if t.strip()]
terms = {}
if term_map and Path(term_map).exists():
terms = term_map_from_markdown(Path(term_map).read_text(encoding="utf-8"))
checks = run_checks(src_lines, tgt_lines, bilingual, terms, allow)
failed = [name for name, status, *_ in checks if status == FAIL]
warned = [name for name, status, *_ in checks if status == WARN]
if args.json:
print(json.dumps({
"target": label,
"passed": [c[0] for c in checks if c[1] == PASS],
"warned": warned,
"failed": failed,
"details": {c[0]: {"status": c[1], "detail": c[2]} for c in checks},
}, ensure_ascii=False, indent=2))
else:
print(f"check-translation.py — {label}\n")
for name, status, detail, *_ in checks:
print(f"[{status:4}] {name}: {detail}")
summary = "ALL CHECKS PASSED"
if warned:
summary = f"PASSED with warnings: {', '.join(warned)}"
if failed:
summary = f"FAILED: {', '.join(failed)}"
print(f"\n{summary}")
sys.exit(0 if not failed else 1)
if __name__ == "__main__":
main()
+10 -3
View File
@@ -1,6 +1,7 @@
---
name: chinese-text-normalize
description: Normalize Chinese markdown files remove extraneous mid-sentence line breaks from fixed-width exports while preserving TOC structures, section headers, and intentional paragraph breaks.
name: mpi-chinese-text-normalize
description: Normalize Chinese markdown files by removing extraneous mid-sentence line breaks from fixed-width exports while preserving TOC structures, section headers, and intentional paragraph breaks. Do not use for English prose, wiki-link index files, or mixed CJK/English documents without manual review.
compatibility: Requires Python 3.9+ and uv. The script's shebang invokes `uv run --script`; it uses only the standard library.
---
When Chinese text has hard line breaks at a fixed width (~20-25 chars) — common in PDF exports, OCR output, or poorly-converted documents — use this skill to join them into flowing paragraphs.
@@ -13,7 +14,13 @@ When Chinese text has hard line breaks at a fixed width (~20-25 chars) — commo
## Approach
Run `scripts/normalize_breaks.py <directory>` — it processes all .md files in the directory.
Run the script directly:
```bash
skills/mpi-chinese-text-normalize/scripts/normalize_breaks.py <directory>
```
The shebang invokes `uv run --script`. It processes all `.md` files in the directory.
The script handles three file patterns:
+11 -1
View File
@@ -1,3 +1,9 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.9"
# dependencies = []
# ///
"""
Fix extraneous line breaks in Chinese markdown files.
@@ -6,8 +12,9 @@ Three file patterns:
2. Mostly-paragraph with stray breaks + outline TOC -> join broken lines, preserve list items
3. Already fine -> skip (idempotent)
Usage: python3 normalize_breaks.py <directory>
Usage: ./normalize_breaks.py <directory>
"""
import re
import sys
from pathlib import Path
@@ -155,6 +162,9 @@ def process_file(filepath):
def main():
if len(sys.argv) != 2:
print("Usage: ./normalize_breaks.py <directory>", file=sys.stderr)
sys.exit(1)
workdir = Path(sys.argv[1])
files = sorted(workdir.glob('*.md'))
+45 -287
View File
@@ -1,253 +1,45 @@
---
name: pdf-to-docx-conversion
description: "Convert flowing text PDFs (Chinese or multi-language) to DOCX with proper fonts, styles, native bullets, lists, and embedded images. Preserves visual hierarchy from PDF font/size/color data."
version: 1.0.0
author: Claude
name: mpi-pdf-to-docx-conversion
description: Convert flowing text PDFs (Chinese or multi-language) to DOCX with proper fonts, styles, native bullets, lists, and embedded images. Use for text-based PDFs. Do not use for scanned/image PDFs, form-heavy PDFs, or documents where exact page layout must be preserved.
license: MIT
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [PDF, DOCX, Documents, python-docx, pymupdf]
compatibility: Requires Python 3.9+ and uv. The script's shebang invokes `uv run --script`; dependencies (pymupdf, python-docx) are declared in the `/// script` metadata.
---
# PDF-to-DOCX Conversion
Convert PDF documents (especially flowing text documents in any language, including CJK) into well-structured DOCX files that preserve fonts, sizes, colors, and layout intent.
Convert text-based PDF documents (including CJK) into structured DOCX files that preserve fonts, sizes, colors, and layout intent.
## Prerequisites
## Quick start
For most PDFs, run the bundled converter directly:
```bash
pip install pymupdf python-docx
skills/mpi-pdf-to-docx-conversion/scripts/convert_pdf_to_docx.py input.pdf output.docx
```
The shebang invokes `uv run --script`, which reads the `/// script` metadata block and installs `pymupdf` and `python-docx` automatically.
For documents with unusual fonts or structure, inspect first and pass a config dict. See `references/config-patterns.md` for the config schema and common patterns.
## Workflow
1. **Inspect the PDF.** Dump fonts, sizes, bullets, and header hierarchy. See `references/inspection-guide.md`.
2. **Configure.** Build a config dict matching the PDF's patterns (fonts, bullet fonts, skip fonts, numbered patterns, header sizes). See `references/config-patterns.md`.
3. **Convert.** Run `scripts/convert_pdf_to_docx.py` or import `PDFToDOCXConverter` in Python.
4. **Verify.** Check paragraph count, styles, bullets, images, and page-number leakage. See the checklist below.
## Features
- **Style-aware**: reads actual font, size, bold, color from PDF spans
- **Native bullets**: uses Word `List Bullet` style instead of Wingdings glyphs
- **Native numbering**: uses numbered list style for sequential items
- **Image extraction**: detects and embeds PDF images into the DOCX
- **Verse/poetry handling**: splits merged verse lines at semantic boundaries
- **Multi-language**: works with CJK, RTL, and mixed-script documents
- **Flowing text**: text flows across pages; no forced page breaks
- Style-aware extraction (font, size, bold, color)
- Native Word bullets and numbering
- Image extraction and embedding
- Verse/poetry line splitting
- Multi-language support (CJK, RTL, mixed scripts)
- Flowing text across pages (no forced page breaks)
## Quick Start
## Verification checklist
```python
from pdf_to_docx import convert_pdf_to_docx
convert_pdf_to_docx("input.pdf", "output.docx")
```
## Step-by-Step Workflow
### 1. Inspect the PDF
First, dump the PDF to understand its structure:
```bash
python3 << 'PY'
import pymupdf
doc = pymupdf.open("input.pdf")
for pi in range(len(doc)):
page = doc[pi]
blocks = page.get_text("dict")["blocks"]
for block in blocks:
if block["type"] != 0: continue
for line in block["lines"]:
for span in line["spans"]:
bbox = span["bbox"]
flags = span["flags"]
attrs = []
if flags & 2**1: attrs.append("I")
if flags & 2**4: attrs.append("B")
print(f" Y={bbox[1]:.0f} [{span['size']:.1f}pt {'+'.join(attrs) or '-'}] {span['font']} | {span['text']}")
PY
```
Key things to identify:
- **Fonts used** (map to DOCX fonts)
- **Bullet mechanism** (Wingdings? Unicode?)
- **Header hierarchy** (what size = section header vs sub-header)
- **Numbered lists** (what delimiter: `1)` `1.` `1`)
- **Images** (check `page.get_images()`)
- **Special sections** (tables, verses, forms)
### 2. Configure the Converter
Create a config dict matching your PDF's patterns:
```python
config = {
"fonts": {
"title": "STHeitiSC-Medium",
"body": "HYShuSongErKW",
"page_number": "HelveticaNeue",
},
"header_sizes": {"section": 18, "sub": 15},
"body_size": 12,
"bullet_fonts": ["Wingdings", "Wingdings 2", "Wingdings 3"],
"page_number_font": "HelveticaNeue",
"numbered_patterns": [r'^\d+\)', r'^\d+\.'], # detect numbered items
"skip_fonts": ["HelveticaNeue"], # fonts to skip (page numbers)
}
```
### 3. Run the Conversion
```python
from pdf_to_docx import PDFToDOCXConverter
converter = PDFToDOCXConverter(config)
converter.convert("input.pdf", "output.docx")
```
## Core Classes
### PDFToDOCXConverter
```python
class PDFToDOCXConverter:
def __init__(self, config=None):
self.cfg = config or self._default_config()
def convert(self, pdf_path: str, docx_path: str):
"""Main entry point."""
# 1. Extract all spans
# 2. Merge Wingdings bullets with body text
# 3. Group lines into paragraphs by Y-gap and style changes
# 4. Post-process: split compact lists, verses, merged steps
# 5. Build DOCX with proper styles
# 6. Embed images
pass
def extract_spans(self, pdf_path: str) -> list[dict]:
"""Extract all text spans with full style info."""
doc = pymupdf.open(pdf_path)
raw = []
for pi in range(len(doc)):
for block in doc[pi].get_text("dict")["blocks"]:
if block["type"] != 0: continue
for line in block["lines"]:
for span in line["spans"]:
raw.append({
"text": span["text"], "font": span["font"],
"size": span["size"], "flags": span["flags"],
"color": span["color"], "bbox": span["bbox"],
})
return raw
def detect_bullets(self, line: list) -> bool:
"""Check if first span in line is a Wingdings bullet."""
return "Wingdings" in line[0]["font"]
def group_paragraphs(self, lines: list) -> list[dict]:
"""Group raw lines into logical paragraphs."""
# Group by Y-gap threshold (typically 20-30pt)
# Break on style change (font change, size > threshold, bold toggle)
# Break on attribution lines (——节选自...)
# Break on special fonts (STHeitiSC-Light etc.)
pass
def post_process(self, paras: list) -> list[dict]:
"""Split merged compact lists and verses."""
# See examples below for common patterns
pass
```
## Common Post-Processing Patterns
### Compact Numbered Lists
When the PDF flows list items together in one paragraph:
```python
def split_compact_list(text: str) -> list[str]:
"""Split '1) foo 2) bar 3) baz' into separate items."""
parts = re.split(r'(?=\d+\))', text)
return [p for p in parts if p.strip()]
```
### Verse / Poetry Lines
When the PDF merges verse lines that should be on separate lines:
```python
def split_verse(text: str, split_markers: list[str]) -> list[str]:
"""Split verse at semantic phrase boundaries.
Example markers: ['感恩', '', '更愿']"""
pattern = '|'.join(f'(?={m})' for m in split_markers)
return [p for p in re.split(pattern, text) if p.strip()]
```
### 小组交流流程 / Process Steps
Split Chinese process steps numbered 一、二、三、etc.:
```python
def split_chinese_steps(text: str) -> list[str]:
"""Split '一、foo 二、bar' into separate items."""
parts = re.split(r'(?=[一二三四五六七八九十]、)', text)
return [p.strip() for p in parts if p.strip()]
```
## DOCX Construction
### Font Setup (East Asian fonts)
```python
from docx.oxml.ns import qn
from docx.oxml import OxmlElement
def set_east_asian_font(run, name: str):
"""Set CJK font properly in python-docx."""
rPr = run._element.get_or_add_rPr()
rFonts = rPr.find(qn('w:rFonts'))
if rFonts is None:
rFonts = OxmlElement('w:rFonts')
rPr.insert(0, rFonts)
rFonts.set(qn('w:eastAsia'), name)
rFonts.set(qn('w:ascii'), name)
rFonts.set(qn('w:hAnsi'), name)
```
### Bullet Items
Use native Word bullets, NOT Wingdings characters:
```python
p = doc.add_paragraph(style='List Bullet')
p.clear()
run = p.add_run("Your bullet text here")
set_east_asian_font(run, font_name)
```
### Image Embedding
```python
from docx.shared import Inches
p = doc.add_paragraph()
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
run = p.add_run()
run.add_picture(image_path, width=Inches(3.5))
```
Extract images from PDF first:
```python
doc = pymupdf.open("input.pdf")
for pi in range(len(doc)):
images = doc[pi].get_images()
for idx, img in enumerate(images):
xref = img[0]
base = doc.extract_image(xref)
with open(f"extracted_{pi}_{idx}.{base['ext']}", 'wb') as f:
f.write(base['image'])
```
## Verification Checklist
After conversion, verify the DOCX:
After conversion, inspect the DOCX:
```bash
python3 -c "
@@ -256,67 +48,33 @@ doc = Document('output.docx')
print(f'Paragraphs: {len(doc.paragraphs)}')
for i, p in enumerate(doc.paragraphs):
style = p.style.name if p.style else '-'
txt = p.text[:80]
print(f'[{i:2d}] [{style:15s}] {txt}')
print(f'[{i:2d}] [{style:15s}] {p.text[:80]}')
"
```
Check for:
1. [ ] All sections present (count paragraphs)
1. [ ] All sections present (paragraph count matches expectations)
2. [ ] No merged verses or lists
3. [ ] Headers are bold + larger size
4. [ ] Bullets use `List Bullet` style
3. [ ] Headers are bold and larger than body text
4. [ ] Bullets use the `List Bullet` style
5. [ ] Numbered items are separate paragraphs
6. [ ] Images present in `word/media/`
7. [ ] Attribution lines right-aligned/indented
8. [ ] No page numbers leaked into body
6. [ ] Images are embedded in `word/media/`
7. [ ] Attribution lines are right-aligned or indented
8. [ ] Page numbers are not leaked into body text
## Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| Text over-merged | Y-gap threshold too high | Lower gap threshold (e.g. 20 → 15) |
| Missing sections | Skipped by font filter | Add font to skip_fonts or remove filter |
| Over-split lines | Y-gap threshold too low | Raise gap threshold (e.g. 20 → 30) |
| Wingdings boxes | Unicode bullet inserted | Use `style='List Bullet'` instead |
| CJK font wrong | East Asian font not set | Use `set_east_asian_font()` helper |
| Image missing | Not extracted before DOCX build | Run `extract_images()` first |
| Verse mangled | Regex too aggressive | Tune verse splitting pattern |
If output is wrong, see `references/troubleshooting.md` for a full symptom/cause/fix table. Common first checks:
## Full Example Script
- Over-merged text → lower the Y-gap threshold.
- Over-split lines → raise the Y-gap threshold.
- Wingdings boxes → use native `List Bullet` style instead.
- Missing images → ensure images are extracted before DOCX construction.
See `scripts/convert_pdf_to_docx.py` for a production-ready converter with all patterns pre-configured.
## References
```python
# scripts/convert_pdf_to_docx.py
import pymupdf, re
from docx import Document
from docx.shared import Pt, Cm, Inches, RGBColor
from docx.enum.text import WD_ALIGN_PARAGRAPH
from docx.oxml.ns import qn
from docx.oxml import OxmlElement
def convert_pdf_to_docx(pdf_path: str, docx_path: str, config: dict = None):
"""Convert a flowing text PDF to DOCX."""
cfg = config or {}
# Extract
doc_pdf = pymupdf.open(pdf_path)
raw = []
for pi in range(len(doc_pdf)):
for block in doc_pdf[pi].get_text("dict")["blocks"]:
if block["type"] != 0: continue
for line in block["lines"]:
ss = line["spans"]
is_b = any("Wingdings" in s["font"] for s in ss[:1])
text = "".join(s["text"] for s in ss)
dom = ss[1] if (is_b and len(ss) > 1) else ss[0]
raw.append(dict(text=text, font=dom["font"], size=dom["size"],
bold=bool(dom["flags"] & 2**4), color=dom["color"],
x=dom["bbox"][0], y=dom["bbox"][1], is_bullet=is_b))
# ... (merge bullets, group paragraphs, post-process, build DOCX)
# Run:
# python scripts/convert_pdf_to_docx.py input.pdf output.docx
```
- `references/inspection-guide.md` — dump PDF structure and interpret spans
- `references/config-patterns.md` — config dict, compact lists, verses, numbered steps
- `references/docx-construction.md` — East Asian fonts, bullets, image embedding
- `references/troubleshooting.md` — symptom/cause/fix table
- `scripts/convert_pdf_to_docx.py` — production-ready converter
@@ -0,0 +1,60 @@
# PDF-to-DOCX Config Patterns
The converter accepts a config dict that maps PDF-specific patterns to DOCX behavior.
## Example config
```python
config = {
"fonts": {
"title": "STHeitiSC-Medium",
"body": "HYShuSongErKW",
"page_number": "HelveticaNeue",
},
"header_sizes": {"section": 18, "sub": 15},
"body_size": 12,
"bullet_fonts": ["Wingdings", "Wingdings 2", "Wingdings 3"],
"page_number_font": "HelveticaNeue",
"numbered_patterns": [r'^\d+\)', r'^\d+\.'],
"skip_fonts": ["HelveticaNeue"],
}
```
## Common post-processing patterns
### Compact numbered lists
Split items that flowed together in one PDF paragraph:
```python
def split_compact_list(text: str) -> list[str]:
parts = re.split(r'(?=\d+\))', text)
return [p for p in parts if p.strip()]
```
### Verse / poetry lines
Split merged verse at semantic phrase boundaries:
```python
def split_verse(text: str, split_markers: list[str]) -> list[str]:
pattern = '|'.join(f'(?={m})' for m in split_markers)
return [p for p in re.split(pattern, text) if p.strip()]
```
### Chinese process steps
Split `一、foo 二、bar` into separate items:
```python
def split_chinese_steps(text: str) -> list[str]:
parts = re.split(r'(?=[一二三四五六七八九十]、)', text)
return [p.strip() for p in parts if p.strip()]
```
## Tuning thresholds
- **Y-gap threshold** (typically 2030 pt): controls how aggressively lines are grouped into paragraphs.
- **Style-change threshold**: break paragraphs on font change, size jump, or bold toggle when the difference exceeds this value.
See `references/troubleshooting.md` for threshold adjustment guidance.
@@ -0,0 +1,59 @@
# DOCX Construction Notes
Low-level notes for building the DOCX output with python-docx.
## East Asian fonts
Set CJK fonts properly on a run:
```python
from docx.oxml.ns import qn
from docx.oxml import OxmlElement
def set_east_asian_font(run, name: str):
rPr = run._element.get_or_add_rPr()
rFonts = rPr.find(qn('w:rFonts'))
if rFonts is None:
rFonts = OxmlElement('w:rFonts')
rPr.insert(0, rFonts)
rFonts.set(qn('w:eastAsia'), name)
rFonts.set(qn('w:ascii'), name)
rFonts.set(qn('w:hAnsi'), name)
```
## Native bullets
Use Word's `List Bullet` style, not Wingdings characters:
```python
p = doc.add_paragraph(style='List Bullet')
p.clear()
run = p.add_run("Bullet text")
set_east_asian_font(run, font_name)
```
## Image embedding
Extract images from the PDF first, then embed them:
```python
from docx.shared import Inches
# Extract
doc = pymupdf.open("input.pdf")
for pi in range(len(doc)):
for idx, img in enumerate(doc[pi].get_images()):
xref = img[0]
base = doc.extract_image(xref)
path = f"extracted_{pi}_{idx}.{base['ext']}"
with open(path, 'wb') as f:
f.write(base['image'])
# Embed
p = doc.add_paragraph()
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
run = p.add_run()
run.add_picture(image_path, width=Inches(3.5))
```
See `scripts/convert_pdf_to_docx.py` for the production-ready implementation.
@@ -0,0 +1,42 @@
# PDF Inspection Guide
Before converting a PDF, dump its text spans to understand fonts, sizes, bullets, and structure.
## Dump spans
```bash
python3 << 'PY'
import pymupdf
doc = pymupdf.open("input.pdf")
for pi in range(len(doc)):
page = doc[pi]
blocks = page.get_text("dict")["blocks"]
for block in blocks:
if block["type"] != 0: continue
for line in block["lines"]:
for span in line["spans"]:
bbox = span["bbox"]
flags = span["flags"]
attrs = []
if flags & 2**1: attrs.append("I")
if flags & 2**4: attrs.append("B")
print(f" Y={bbox[1]:.0f} [{span['size']:.1f}pt {'+'.join(attrs) or '-'}] {span['font']} | {span['text']}")
PY
```
## What to identify
- **Fonts used** — map PDF font names to DOCX font names.
- **Bullet mechanism** — Wingdings glyphs, Unicode bullets, or something else.
- **Header hierarchy** — which font size marks section vs. sub-section headers.
- **Numbered lists** — delimiter style: `1)`, `1.`, `1`, `一、`.
- **Images** — check `page.get_images()` on each page.
- **Special sections** — tables, verses, attribution lines, page numbers, forms.
## Page numbers
Page-number fonts are usually small and repeated on every page. Add them to `skip_fonts` in the config so they do not leak into body text.
## Attribution lines
Lines like `——节选自...` often use a different font or indentation. The converter breaks paragraphs on these; verify the break point after conversion.
@@ -0,0 +1,18 @@
# PDF-to-DOCX Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| Text over-merged | Y-gap threshold too high | Lower gap threshold (e.g. 20 → 15) |
| Missing sections | Skipped by font filter | Add font to `skip_fonts` or remove filter |
| Over-split lines | Y-gap threshold too low | Raise gap threshold (e.g. 20 → 30) |
| Wingdings boxes | Unicode bullet inserted | Use `style='List Bullet'` instead |
| CJK font wrong | East Asian font not set | Use `set_east_asian_font()` helper |
| Image missing | Not extracted before DOCX build | Run image extraction first |
| Verse mangled | Regex too aggressive | Tune verse splitting pattern |
## General debugging steps
1. Re-inspect the PDF with the span dump from `references/inspection-guide.md`.
2. Compare the config against the actual fonts and sizes in the dump.
3. Run the verification script in `SKILL.md` and check paragraph styles.
4. Adjust one threshold at a time and re-convert.
+12 -4
View File
@@ -1,13 +1,21 @@
#!/usr/bin/env python3
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.9"
# dependencies = [
# "pymupdf",
# "python-docx",
# ]
# ///
"""
Production-ready PDF-to-DOCX converter.
Handles multi-language text, mixed fonts, bullets, numbered lists, verses,
attributions, images, and flowing text across pages.
Usage:
python convert_pdf_to_docx.py input.pdf output.docx
./convert_pdf_to_docx.py input.pdf output.docx
Requires: pip install pymupdf python-docx
Requires: uv (dependencies are declared in the /// script block above)
"""
import sys
@@ -544,6 +552,6 @@ def convert_pdf_to_docx(pdf_path: str, docx_path: str, config: dict = None):
# CLI
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: python convert_pdf_to_docx.py <input.pdf> <output.docx>")
print("Usage: ./convert_pdf_to_docx.py <input.pdf> <output.docx>")
sys.exit(1)
convert_pdf_to_docx(sys.argv[1], sys.argv[2])
+15 -6
View File
@@ -1,7 +1,8 @@
---
name: pptx-translate
description: Translate PowerPoint files between Chinese and English — extract strings to YAML, translate, quality review, and write back with font-shrink + auto-fit for layout.
name: mpi-pptx-translate
description: Translate PowerPoint files between Chinese and English — extract strings to YAML, translate, quality review, and write back with font-shrink + auto-fit for layout. Use only for .pptx files. Do not use for .ppt, Google Slides exports, or PDFs.
category: productivity
compatibility: Requires Python 3.9+ and uv. The scripts' shebang invokes `uv run --script`; dependencies (python-pptx, pyyaml) are declared in the `/// script` metadata.
---
# PPTX Translation
@@ -12,7 +13,13 @@ Translate `.pptx` files between Chinese and English. Covers the full pipeline: e
### 1. Extract strings to YAML
Run `toolkit/scripts/extract.py original.pptx strings.yaml`. Produces YAML with entries:
Run the script directly:
```bash
skills/mpi-pptx-translate/scripts/extract.py original.pptx strings.yaml
```
The shebang invokes `uv run --script`, which reads the `/// script` metadata block and installs `python-pptx` and `pyyaml` automatically. Produces YAML with entries:
```yaml
- slide: 1
@@ -57,7 +64,9 @@ Scan for:
### 4. Write back with layout fixes
Run `toolkit/scripts/build.py strings.yaml original.pptx translated.pptx`.
```bash
skills/mpi-pptx-translate/scripts/build.py strings.yaml original.pptx translated.pptx
```
The script:
- Replaces text in matching paragraphs (clears all runs, sets first run)
@@ -80,5 +89,5 @@ The absorbed `pptx-translation` skill had alternate script names: `extract_pptx.
## Scripts
- `toolkit/scripts/extract.py` — extract strings from PPTX to YAML
- `toolkit/scripts/build.py` — write translations back with font shrink + auto-fit
- `skills/mpi-pptx-translate/scripts/extract.py` — extract strings from PPTX to YAML
- `skills/mpi-pptx-translate/scripts/build.py` — write translations back with font shrink + auto-fit
+28 -13
View File
@@ -1,3 +1,12 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.9"
# dependencies = [
# "python-pptx",
# "pyyaml",
# ]
# ///
import sys, yaml
from pptx import Presentation
from pptx.util import Pt
@@ -5,15 +14,6 @@ from pptx.enum.text import MSO_AUTO_SIZE
FONT_SCALE = 0.82 # shrink ~18%
yaml_path, src_path, out_path = sys.argv[1], sys.argv[2], sys.argv[3]
with open(yaml_path) as f:
entries = yaml.safe_load(f)
index = {}
for e in entries:
index[(e["slide"], e["shape"], e["run"])] = e["en"]
def shrink_font_tf(tf):
for para in tf.paragraphs:
@@ -26,9 +26,17 @@ def shrink_font_tf(tf):
pass
prs = Presentation(src_path)
def build(yaml_path, src_path, out_path):
with open(yaml_path) as f:
entries = yaml.safe_load(f)
for slide_num, slide in enumerate(prs.slides, 1):
index = {}
for e in entries:
index[(e["slide"], e["shape"], e["run"])] = e["en"]
prs = Presentation(src_path)
for slide_num, slide in enumerate(prs.slides, 1):
for shape_idx, shape in enumerate(slide.shapes):
if shape.has_text_frame:
has_translation = False
@@ -64,5 +72,12 @@ for slide_num, slide in enumerate(prs.slides, 1):
ns.notes_text_frame.clear()
ns.notes_text_frame.paragraphs[0].add_run().text = index[key]
prs.save(out_path)
print(f"Saved {out_path}")
prs.save(out_path)
print(f"Saved {out_path}")
if __name__ == "__main__":
if len(sys.argv) != 4:
print("Usage: ./build.py <strings.yaml> <input.pptx> <output.pptx>", file=sys.stderr)
sys.exit(1)
build(sys.argv[1], sys.argv[2], sys.argv[3])
+12
View File
@@ -1,3 +1,12 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.9"
# dependencies = [
# "python-pptx",
# "pyyaml",
# ]
# ///
import sys, yaml
from pptx import Presentation
@@ -62,6 +71,9 @@ def extract(pptx_path):
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: ./extract.py <input.pptx> <output.yaml>", file=sys.stderr)
sys.exit(1)
entries = extract(sys.argv[1])
with open(sys.argv[2], "w") as f:
yaml.dump(entries, f, allow_unicode=True, default_flow_style=False, sort_keys=False)
+14 -14
View File
@@ -1,19 +1,20 @@
---
name: terms-search
description: Full-text search across the MPI term database. Use when translating or looking up Chinese-English Buddhist/MPI terminology.
name: mpi-terms-search
description: Full-text search across the MPI Buddhist/Dharma term database. Use when translating or reviewing Chinese-English Buddhist terminology. Do not use for general Chinese-English dictionary lookup outside MPI conventions.
category: research
compatibility: Requires Python 3; SQLite database is bundled.
---
# Terms Search
Database: `$MPI_PROJECT_ROOT/toolkit/terms-database/termlib.duckdb`
CLI: `$MPI_PROJECT_ROOT/toolkit/terms-database/search.py`
Server: `$MPI_PROJECT_ROOT/toolkit/terms-database/server.py`
Database: `toolkit/terms-database/termlib.sqlite` (SQLite)
CLI: `toolkit/terms-database/search.py`
Server: `toolkit/terms-database/server.py`
## CLI (preferred)
```
$MPI_PROJECT_ROOT/toolkit/terms-database/search.py <query> [limit]
toolkit/terms-database/search.py <query> [limit]
```
Multi-word queries are ANDed. Searches both `zh` and `en` columns.
@@ -22,7 +23,7 @@ Multi-word queries are ANDed. Searches both `zh` and `en` columns.
```python
import sys
sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
sys.path.insert(0, 'toolkit/terms-database')
from search import search
results = search("空性", limit=5, src="DoT定稿")
# → list of {zh, en, loc, source} dicts
@@ -32,7 +33,7 @@ Use this inside `execute_code` scripts for batch lookups — no subprocess neede
## HTTP API (use only when CLI is insufficient)
Start: `python3 $MPI_PROJECT_ROOT/toolkit/terms-database/server.py` (port 8910)
Start: `python3 toolkit/terms-database/server.py` (port 8910)
- `GET /` — plain HTML UI (form + results table, no CSS)
- `GET /` — plain HTML UI (form + results table, no CSS)
@@ -61,20 +62,19 @@ Errors return `{"error": "..."}` with HTTP 500 (API) or shown inline (UI).
| 禅意项目 | 14+11 | Zen program terms |
| 公案 | 8 | Chan koans |
## Direct DuckDB
## Direct SQLite
```
duckdb $MPI_PROJECT_ROOT/toolkit/terms-database/termlib.duckdb
sqlite3 toolkit/terms-database/termlib.sqlite
```
Key tables: `unified_terms_flat` (zh, en, loc, source), individual source tables, `unified_terms` view.
Key table: `terms` (zh, en, loc, source).
## Rebuilding
Terms data comes from `$MPI_PROJECT_ROOT/guide/03 术语库/`. To rebuild:
Terms data comes from `guide/03 术语库/`. To rebuild:
1. Convert source xlsx/ods → CSV+YAML in `_output/`
2. Rebuild DuckDB from CSVs
3. Materialize `unified_terms_flat` view → table for performance
2. Load CSVs into SQLite as the `terms` table (zh, en, loc, source)
**Full rebuild pipeline:** See `references/termbase-rebuild.md` (absorbed from the `termbase-management` skill).
@@ -1,11 +1,11 @@
# Termbase Rebuild (from absorbed termbase-management)
How to rebuild the terms DuckDB from source spreadsheets. This is the full pipeline from the now-archived `termbase-management` skill.
How to rebuild the terms SQLite database from source spreadsheets. This is the full pipeline from the now-archived `termbase-management` skill.
## Prerequisites
```bash
pip install openpyxl odfpy pyyaml duckdb
pip install openpyxl odfpy pyyaml
```
## Step 1: Inspect spreadsheet structure
@@ -69,17 +69,23 @@ def dedup_headers(headers):
- **CSV**: `csv.writer` — column-major, preserves all raw data
- **YAML**: `yaml.dump(data, allow_unicode=True, default_flow_style=False, sort_keys=False, width=200)` — list of dicts
## Step 3: Load into DuckDB
## Step 3: Load into SQLite
```python
import duckdb
con = duckdb.connect('termlib.duckdb')
import sqlite3
import csv
# Simple CSVs work with auto-detect:
con.execute("""
CREATE TABLE table_name AS
SELECT * FROM read_csv_auto('file.csv', header=true, all_varchar=true)
""")
con = sqlite3.connect('termlib.sqlite')
# Simple CSVs work with csv.reader:
with open('file.csv', 'r', encoding='utf-8') as f:
rows = list(csv.reader(f))
headers = rows[0]
data = rows[1:]
col_defs = ', '.join(f'"{h}" TEXT' for h in headers)
con.execute(f'CREATE TABLE "table_name" ({col_defs})')
con.executemany(f'INSERT INTO "table_name" VALUES ({", ".join(["?"] * len(headers))})', data)
con.commit()
```
### Pitfall: Multiline CSV fields
@@ -104,10 +110,10 @@ for i in range(0, len(data), batch_size):
con.execute(f'INSERT INTO "{table}" VALUES {placeholders}', flat)
```
### Pitfall: DuckDB CLI opens in-memory by default
Running plain `duckdb` gives an empty database. Always pass the file path:
### SQLite CLI
Open the database file directly:
```
duckdb path/to/termlib.duckdb
sqlite3 path/to/termlib.sqlite
```
## Step 4: Create unified views
@@ -122,6 +128,6 @@ See `references/unified-view.sql` for the pattern. Key patterns:
- **ODS reading**: Must traverse `odf.text.P` child elements, not direct text nodes
- **Duplicate headers**: JSON/YAML dict silently overwrites duplicate keys — always deduplicate
- **Multiline CSV + DuckDB**: `read_csv_auto` fails on CSVs with quoted newlines — use Python csv.reader
- **DuckDB path**: Always explicit file path; `duckdb` alone is in-memory
- **Multiline CSV**: Use Python `csv.reader` for CSVs with embedded newlines
- **SQLite path**: Always pass the file path to `sqlite3`
- **`execute_code` sandbox**: Does NOT share pip-installed packages — use `terminal` for Python scripts
@@ -10,7 +10,7 @@ After producing a first-pass translation, or when the user asks to check termino
1. Read the full translated file. Extract all Chinese terms from `{% "TERM" (pinyin) = ENGLISH ... %}` blocks.
2. Start the search server: `python3 $MPI_PROJECT_ROOT/toolkit/terms-database/server.py &` (port 8910). It may already be running — check with `curl -s http://localhost:8910/`.
2. Start the search server: `python3 toolkit/terms-database/server.py &` (port 8910). It may already be running — check with `curl -s http://localhost:8910/`.
3. Batch-search each term via the HTTP API:
```
+25 -16
View File
@@ -1,13 +1,9 @@
---
name: translation-review
name: mpi-translation-review
description: |
Unified review skill for Chinese-English Buddhist/Dharma translations.
Supports two modes:
- **self**: You translated the text. Edit target.dj directly.
- **other**: Someone else translated. Write review-comments.dj, do not edit target.dj.
What to review is identical in both modes — the same detection rules,
editorial standards, and terminology checks. Only the action differs: self
mode applies fixes directly; other mode records them for the translator.
Unified review skill for Chinese-English Buddhist/Dharma translations in djot format.
Use in self mode to edit your own target.dj, or in other mode to write review-comments.dj for a peer translator.
Do not use for non-djot formats or for non-Buddhist texts.
---
# Translation Review (unified)
@@ -34,12 +30,17 @@ Switch to self-mode output in that case, but keep a collegial tone.
1. Read `source.dj` and `target.dj` fully before changing anything.
(Other mode only: read the translator's note if present, and address them by name.)
2. Run a three-pass review using the detection rules below.
3. Apply the R1R14 editorial polish checklist.
4. Check terminology against the terms DB (see `mpi-terms-search` skill).
5. Verify that every source paragraph maps to a target paragraph with no missing
2. If the text is **meditation-practice content** — a guided meditation script,
posture/breathing exercise guide, or meditation-method explanation — load
`references/meditation-practice-translation.md` and run its summary checklist
as an extra pass. It distills the recurring issues a human reviewer flagged
across a whole meditation manuscript.
3. Run a three-pass review using the detection rules below.
4. Apply the R1R14 editorial polish checklist.
5. Check terminology against the terms DB (see `mpi-terms-search` skill).
6. Verify that every source paragraph maps to a target paragraph with no missing
or truncated content.
6. Produce the correct artifact for your mode.
7. Produce the correct artifact for your mode.
## Detection rules — three passes
@@ -89,9 +90,13 @@ issue with line/paragraph references.
- **D2 Cultural anachronism.** Modern concepts projected onto classical material.
Example: translating 般若 as "wisdom" in a scholarly context may flatten the
term; in a popular talk it may be exactly right.
- **D3 Tone / voice of the teacher.** For oral talks, preserve the speaker's
warmth, rhetorical questions, and direct address. Do not flatten into essay
prose.
- **D3 Tone AND voice (语气) of the teacher.** For oral talks, preserve the
speaker's warmth, rhetorical questions, and direct address. Beyond register,
check the 语气 markers in `../mpi-translation/SKILL.md` → "Speaker's Voice
(语气)": rhetorical questions kept as questions, first-person teacher asides
("我经常说") kept in first person, reasoning connectives (可见, 所以说)
preserved, inclusive we/you address, homely analogies left concrete, gentle
rather than scolding admonition. Do not flatten into essay prose.
- **D4 Implicit meaning / implicature.** What the source implies but does not say
(e.g., irony, conventional politeness, Gricean maxims). Ensure the implication
survives or is compensated.
@@ -207,6 +212,10 @@ Any DB queries or proposed term changes.
for this project's genres.
- `references/buddhist-terminology.md` — register and convention notes for
Buddhist/Dharma terms.
- `references/meditation-practice-translation.md` — recurring review patterns
for meditation-practice texts (person/subject consistency, sentence
splitting, filler deletion, fixed series terms, Chan-verse quotes), distilled
from a human-reviewed manuscript.
- `../mpi-translation/SKILL.md` — the upstream translation skill that produces the
`target.dj` this skill reviews.
- `../mpi-terms-search/SKILL.md` — skill for querying the terms database before and
@@ -35,6 +35,29 @@ Terms encountered in Chinese-English translation of Dharma study materials. Thes
| 愿心 | mind of vows, bodhicitta aspiration | Plural "vows" in English |
| 重要感、优越感、主宰欲 | sense of importance, superiority, desire to control | Three ego-driven motivations |
## Meditation-practice terms (from 初级正念禅修 review)
Terms from the *初级正念禅修* review (2026-08-10). These appear in
meditation-method / breathing / posture texts. See
`references/meditation-practice-translation.md` for the full review patterns.
| Chinese | English | Notes |
|---------|---------|-------|
| 初级正念(禅修) | Primary Mindfulness (Meditation) | NOT "elementary". 初级 → "primary" in this series. |
| 毗卢七支坐 | seven-point posture of Vairocana | terms DB: "Seven-Point Postures of Vairochana". Use consistently. |
| 八式动禅 | Eight Mindful Exercises for Lotus Posture | Fixed series term. |
| 七式呼吸 | Seven Mindful Exercises for Breathing | Fixed series term. |
| 观呼吸 | Mindful Breathing | As a practice name; not "Mindfulness of Breathing". |
| 打坐 / 坐禅 | sitting meditation | terms DB `坐禅`. Use uniformly ("seated meditation" OK). |
| 细(呼吸) | subtle | For 细 breath quality; "fine" reads unnatural to general readers. |
| 止 | settling the mind on an object | Terse source needs expansion; never bare "stopping". |
| 证悟 | enlightenment | terms DB. Not "Way". |
| 本心 / 真心 | true mind | Use consistently within a document (also for 心性 where intended). |
| 辅助(禅修) | supportive (mindfulness practice) | "supportive" preferred over "auxiliary" (warmer, conversational). |
| 觉知 | awareness | terms DB. |
| 观照 | attend to the mind / mindful observation | Contemplative, not passive "observe". |
| 缘起 | dependent origination | Prefer "follow the principle of dependent origination" collocation. |
## Structural patterns
- Section numbering: Chinese uses 一、二、三... English should pick one style (Part One/Two, First/Second, I/II) and stick with it.
@@ -0,0 +1,174 @@
# Meditation-Practice Text Translation — Patterns from Human Review
Human review of *初级正念禅修:从基础练习到觉知开启* (translated 代艺初翻,
reviewed 慈德审, 2026-08-10) produced 85 reviewer comments. They cluster into a
small set of recurring patterns. These are the issues a Chinese → English
translator/reviewer of meditation-practice texts most often gets wrong. Treat
this as a checklist for any guided-meditation script, posture/breathing
exercise guide, or meditation-method explanation.
## 1. Person and subject (the single most-repeated issue)
Chinese meditation prose drops subjects; English must not. The reviewer's #1
fix was to **supply and keep a consistent person as subject** — almost always
inclusive **"we"**, with **"you"** for direct step-by-step guidance. Two
failures dominate:
- **Missing subject / missing verb.** A bare clause that reads as an imperative
fragment where the source intended a statement.
- Source: `……入静` → target `, adjusting ...` → fix: give it a subject and a
full verb.
- **Drift into abstract third person / generic "people"/"one".** Meditation
practice is reader-facing. Render the practitioner as "we" (shared
experience) or "you" (guidance), never as "people," "one," or a dangling
noun phrase.
**Rule:** For every sentence, name the subject. Check the previous sentence's
subject and keep the reference chain consistent (don't hop between "the
practitioner," "we," "you," and an inanimate object). The reviewer repeatedly
asked to change the subject *back* to a consistent person for reader
engagement (参与感).
Examples from the review:
| Problem | Reviewed fix |
|---|---|
| `use` (no subject) | "In the Seven Mindful Exercises for Breathing, **we use** postures and various body movements to expand the space of breathing…" |
| "process" as subject | "**In this process, we need to keep alert**: when we realize that the mind has wandered off or grown drowsy, we bring it back." |
| missing subject at paragraph start | supply "we": "Through repeated practice, **we** gradually master it and make it a habit." |
| "when there is no awareness…" | "**When we** are not aware…" |
When the source uses an abstract/system noun (mind, meditation, the method),
match it only if the English sentence genuinely reads well that way; if it
reads stiff, re-cast around a person. Chinese topic-comment structure must not
be reproduced as a subjectless English sentence.
## 2. Split long sentences (one idea per breath)
Chinese runs clauses together; the reviewer repeatedly split one long sentence
into two or three, often introducing "we" as the subject of each.
| Before (literal) | After (reviewed) |
|---|---|
| "We need the view of the sudden teaching so that we can be filled with confidence in practice—knowing that within the life of every person lies a quality no different from that of all the Buddhas of the three periods of time and the successive patriarchs" | "We need the view of the sudden teaching because it gives us confidence in our practice. It helps us understand that within each of us lies a quality no different from that of all the Buddhas across the past, present, and future, as well as the successive patriarchs." |
| "These exercises rely not only on movement practice but also on coordinating with the breath—during each movement training, bringing the breath to every part of the body…" | "In these exercises, we not only rely on physical movement but also regulate the breath. During each movement, we bring the breath to every part of the body…" |
| "As long as we perform each movement with concentration and awareness, then whether we are dressing, eating, relating with others, walking…" | "As long as we engage in each action with concentration and awareness, even daily activities can become the actual practice of meditation—whether we are getting dressed, eating, walking, standing, sitting, lying down, interacting with others, or handling daily affairs." |
## 3. Delete redundant filler (the second most-repeated fix)
The reviewer deleted words that add nothing: `particular` (with `emphasis`),
`corresponding`, `especially`, `basically`, `essentially`, `like someone`,
`in fantasy`, `the`, `a process of`, `arising and passing`, `, the crucial
doorway to awakened nature`, `to heed during sitting meditation`.
**Rule:** When a sentence parses with and without a word, drop it. Common
Chinese-draft padding: "especially," "basically," "essentially,"
"particularly," "corresponding," "a process of." These are almost always
compressible or deletable.
## 4. Over-literal / word-for-word (字对字) renderings
The reviewer flagged machine-literal translations that needed to become
natural, person-led English:
| Over-literal | Reviewed fix |
|---|---|
| "life-form" (for 色身/生命载体) | "Our present body, made up of the five aggregates, is…" |
| "even dull faculties, provided the method is right…" | "Even those of dull faculties, if they practice with the right method and diligently wipe away the dust of the mind, can realize the true mind." |
| "when able, do abdominal breathing" (字对字 of 能则) | "When we are able to practice abdominal breathing, we simply do it." |
| "we climb on again" (for a car returning to a highway) | "we get back on" — the AI misread the metaphor; a car does not "climb." |
| "we are accustomed to 'not knowing'" | "we are used to being unaware" |
**Rule:** If the English only makes sense by re-translating it into the
Chinese, it is over-literal. Read for meaning, not word order.
## 5. Avoid exaggerated or unusual English
Chinese meditation prose is plain; keep the English plain. The reviewer
specifically softened:
- "an unending supply of vitality and energy" → "the vitality and energy
needed" / "a steady supply" (unending sounds exaggerated).
- "like a turbid pool slowly becoming clear" → "like a pool of muddy water
slowly becoming clear" (turbid is literary; muddy is plain).
- "fine" (for 细 breath) → "subtle" — meditative, natural for general readers.
- "deconstruct" → "see through" (plainer).
- "obscured" → "veiled" (fewer, simpler words).
- "transcend" → "go beyond" (especially when another "transcend" appears nearby
— avoid repetition).
- "accord with" → "align with" (more conversational for a general audience).
**Rule:** Prefer short, warm, everyday words over literary or Latinate ones
for a lay readership. If a word would need a dictionary look-up or a footnote
for a general reader, replace it.
## 6. Consistency of terms within the document
- Same concept, one rendering: 本心/真心 → **true mind** throughout (not
alternating "true mind" / "nature" / "original mind"). The reviewer picked
"true mind" as the clearest and asked for it to be used consistently.
- 初级 → **primary** (not "elementary"). 初级正念禅修 = **Primary Mindfulness
Meditation**. Pick one form for "elementary/primary/basic" and keep it.
- 打坐/坐禅 → **sitting meditation** (uniform; matches terms DB `坐禅`
"sitting meditation").
- Section-title style: 观呼吸 → **Mindful Breathing** (not "Mindfulness of
Breathing") when used as a practice name.
**Rule:** Build a one-term-per-concept map for the file and enforce it. Any
term changed mid-review must be updated everywhere (search the whole file).
## 7. Fixed series / internal terms (check the series, not just the DB)
This book belongs to a meditation series with established names. Do not invent
new renderings when a fixed name exists:
- 毗卢七支坐 → **seven-point posture of Vairocana** (terms DB: "Seven-Point
Postures of Vairochana").
- 八式动禅 → **Eight Mindful Exercises for Lotus Posture**.
- 七式呼吸 → **Seven Mindful Exercises for Breathing**.
- 辅助 (as in auxiliary mindfulness practice) → **supportive** (reviewer
preferred "supportive" over "auxiliary": warmer, more conversational).
**Rule:** For a multi-part practice system or series, check whether the names
are already established in the series / terms DB before coining new English.
## 8. Terse Dharma statements need expansion, not word-for-word
A terse Chinese phrase ("止就是安住", "只是转换一下") is often too compressed
to render literally. Expand it into a complete, teachable English sentence:
| Terse | Reviewed expansion |
|---|---|
| "stopping" means settling | "We need to rest the mind on the object of focus; *samatha* means settling the mind on an object." |
| "originally not a single thing" / "at all times diligently wipe it clean" (止观 contrast) | "Sudden teachings point out that the true mind is originally free from all attachment, while gradual teachings emphasize that we, as ordinary beings, need to diligently wipe away the inner dust." |
| "…anxious about…" → impermanence point | "Impermanence: It helps us understand that thoughts arise and cease. With awareness, we can observe them coming and going without seeing them as permanent." |
## 9. Chan verse / scripture quotes — reuse established renderings
When the text quotes a famous Chan verse, do NOT translate it fresh. Use the
established English rendering from the lineage, and if the project already
published this verse (an earlier book in the series), reuse that exact
wording.
Example (信心铭 / *Faith in Mind*, 三祖僧璨): the source quotes
"至道无难,唯嫌拣择,但莫憎爱,洞然明白". The reviewer noted the English
target mis-quoted the verse and that the correct rendering ("Without love or
hate, one sees things as they truly are") was already used in an earlier
translated book in the series. Check `references/translation-pitfalls.md`
"Chan verse / scripture quotes" and prior published books before translating.
## Summary checklist for meditation-practice texts
- [ ] Every sentence has an explicit, consistent person subject ("we"/"you"),
no abstract-third-person drift, no subjectless fragments.
- [ ] Long sentences split into breath-length units.
- [ ] No redundant filler ("especially," "basically," "essentially,"
"particularly," "corresponding").
- [ ] No word-for-word (字对字) renderings; English reads for meaning.
- [ ] No exaggerated/literary words; warm, plain vocabulary.
- [ ] One term per concept, enforced file-wide (true mind, primary, sitting
meditation, Mindful Breathing).
- [ ] Fixed series terms (Vairocana posture, Eight Mindful Exercises for Lotus
Posture, Seven Mindful Exercises for Breathing, supportive).
- [ ] Terse Dharma statements expanded into teachable sentences.
- [ ] Chan verse quotes use established published renderings, not fresh drafts.
@@ -17,9 +17,12 @@ correspondence, generate `bilingual.dj` directly from the DOCX:
4. Write bilingual.dj
The DOCX English is the authoritative target text. No PDF needed.
**Extraction approach**: start by adapting `toolkit/scripts/gen-bilingual-docx.py`.
For articles where the body has strict CN→EN→CN→EN alternation, the simple
extraction in that script (CN line, blank, EN line, blank) works directly.
**Extraction approach**: write a custom extraction script. The standard
`toolkit/scripts/gen-bilingual.py` expects separate `source.dj` and `target.dj` files;
for DOCX→bilingual extraction, adapt its pattern-matching logic to read from the
pandoc plain-text output instead. For articles where the body has strict
CN→EN→CN→EN alternation, the simple extraction (CN line, blank, EN line, blank)
works directly.
### A2. Bilingual from `.docx.md` (pandoc markdown output)
@@ -8,7 +8,7 @@ Import directly in `execute_code` scripts — no subprocess, no server, no text
```python
import sys
sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
sys.path.insert(0, 'toolkit/terms-database')
from search import search
results = search("三级修学", limit=5)
@@ -20,7 +20,7 @@ results = search("空性", loc="心经", src="DoT定稿", limit=5)
```python
import sys
sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
sys.path.insert(0, 'toolkit/terms-database')
from search import search
terms = ["三无漏学", "八步三禅", "闻思修", ...]
@@ -275,6 +275,37 @@ initial/middle/great scope."
佛教术语 has "Guanshiyin/Guanyin Bodhisattva." Use "Guanyin Bodhisattva."
## Chan verse / scripture quotes — reuse established renderings
When the source quotes a well-known Chan verse or sutra passage, do NOT draft
a fresh translation. Use the established English rendering from the lineage,
and check whether the project already published this verse in an earlier book
of the series — reuse that exact wording.
**Case (信心铭 / *Faith in Mind*, from 初级正念禅修 review):** the source
quoted "至道无难,唯嫌拣择,但莫憎爱,洞然明白". The draft target mis-quoted
the verse (anchoring the wrong lines) and the reviewer noted the correct
rendering ("Without love or hate, one sees things as they truly are.") was
already used in an earlier translated book in the series. The annotated
"才有是非,纷然失心" was a mis-annotation of the source; the passage the
source actually quotes is the four-line verse above.
**Detection:** For any quoted verse/idiom the source sets off (quotation
marks, 曰/云/云何 structures), identify the source work (e.g. 信心铭, 金刚经,
六祖坛经) and search prior published translations in the series before
translating. A famous Chan line almost always has a standard English form.
## AI misinterpretation of Chinese metaphor
LLMs sometimes mistranslate a concrete Chinese metaphor by guessing the wrong
image. When the English rendering of a metaphor is physically impossible or
odd ("we climb on again" for a car returning to a highway), the model
misread the source, not the register. Re-read the source image literally and
render the actual picture in plain English ("we get back on").
**Rule:** If the English metaphor is hard to visualize or sounds comical, it
is likely a mistranslation of the source image — check the Chinese.
## Workflow pitfall
### Translating before consulting terms DB
+43 -3
View File
@@ -1,8 +1,9 @@
---
name: translation
description: Translate Chinese↔English Buddhist/Dharma content — register guidance from Mindfulness Bell corpus, tone, voice, cultural bridging technique.
name: mpi-translation
description: Translate Chinese↔English Buddhist/Dharma content. Use for oral talks, written articles, guided meditations, Q&A, and sutra commentary. Do not use for non-religious general Chinese-English translation, technical documentation, or marketing copy.
inputs: source.dj (Chinese djot), or .docx via docx2dj.fish
outputs: target.dj (English djot), bilingual.dj, edit-suggestions.dj
compatibility: Requires pandoc for docx-to-djot conversion.
---
# Translation
@@ -12,7 +13,7 @@ terms DB query, workflows, output format) are in AGENTS.md.
## Source context
Before translating, understand the source's format and delivery context. Is it a transcript of an oral talk, a book excerpt, a guided meditation script, a Q&A, a written article, or another genre? The register shapes the translation. If the context is not clear from the file path or source content, ask the user before proceeding.
Before translating, understand the source's format and delivery context. Is it a transcript of an oral talk, a book excerpt, a guided meditation script, a Q&A, a written article, or another genre? The register shapes the translation. If the context is not clear from the file path or source content, ask the user before proceeding. If you cannot identify the author, write the target text with the style of 济群法师.
## Mindfulness Bell Corpus
@@ -40,6 +41,40 @@ Quick-find in index: Thầy talks → `"Thích Nhất Hạnh"` + page ≤ 10; te
4. **Voice**: Direct address ("you"), concrete images, and oral rhythm make Dharma land in English. Abstract noun chains (common in Chinese→English translationese) kill it.
5. **Sutra quotes**: Use standard English Buddhist idiom. Check terse-idiom conventions (e.g., Diamond Sutra "lives" not "bodies").
## Speaker's Voice (语气)
Tone (register, formality) is not enough — preserve the speaker's 语气: the
stance and manner carried by sentence mood. Reviewers have flagged translations
that got the tone right but flattened the teacher's voice. 语气 lives in:
- **Sentence mood.** Rhetorical questions stay questions ("幸福在哪里?" →
"Where is happiness?", not "Happiness is nowhere to be found.").
Exclamations and wonder stay exclamatory. Do not convert the speaker's
questioning into declarations.
- **First-person teacher asides.** 济群法师 often speaks in his own voice:
"我经常说……", "我曾在讲座中多次谈到……", "由此我想到……". Keep the
first person — do not flatten to "it is often said" or impersonal prose.
These asides are how he establishes presence with the audience.
- **Reasoning connectives.** His talks argue step by step: 可见, 所以说,
事实上, 问题在于. Render the logical gait ("So it is clear that…", "That
is why…", "In fact…") rather than dropping it — the reasoned,
unhurried persuasion IS his voice.
- **Inclusive address.** Default "we" for shared human condition, "you" when
he turns to the listener. Do not drift into abstract third person ("one",
"people") where the source speaks as teacher-to-audience.
- **Everyday analogies, plainly told.** Rotting apples, leaking boats,
teacups, face masks — keep the homely image concrete and unvarnished;
do not upgrade it to literary language or explain it away.
- **Gentle admonition, never scolding.** He points out folly with warmth and
a little humor (the "有点烦" pop song, Mo Yan's dodge). Keep the lightness;
avoid both sermonizing severity and jokey casualness.
- **Measured authority.** Calm, composed, unhurried. No hype, no exclamation-
point enthusiasm, no academic hedging ("arguably", "it could be said").
Check: read a paragraph aloud as if delivering a talk to a lay audience. If it
sounds like an essay, a lecture abstract, or a motivational speaker, the 语气
has been lost.
## Pitfalls
### Pre-flight accuracy check
@@ -241,6 +276,10 @@ Read the entire English target aloud. If anything stalls, rephrase it.
When translating guided meditation scripts, exercise guides, or posture instructions
(rather than Dharma talks), use a lighter workflow. See `references/meditation-translation.md`.
Before drafting, also read `../mpi-translation-review/references/meditation-practice-translation.md`
(patterns a human reviewer flagged across a meditation manuscript) so the recurring
issues — dropped/inconsistent person, over-long sentences, filler, 字对字
renderings, fixed series terms — are avoided up front rather than caught at review.
## Other Pitfalls
@@ -253,6 +292,7 @@ Create article-specific scripts per `references/proofread-pdf-workflow.md`.
## References
- `references/meditation-translation.md` — lighter workflow for meditation/mindfulness content
- `../mpi-translation-review/references/meditation-practice-translation.md` (cross-skill) — recurring patterns for meditation-practice texts, distilled from a human-reviewed manuscript
- `references/markdown-to-djot.md` — converting .docx.md to .dj for translation prep
- `references/bilingual-format.md` — bilingual.dj layout: source/target adjacent, blank separator between pairs
- `references/diacritics-convention.md` — diacritics rules
@@ -7,7 +7,8 @@ use a lighter workflow than the full dharma-translation pipeline.
## Register
Default to warm, direct instructional voice (Thầy-adjacent):
- Second-person address ("you")
- Inclusive first person ("we") for describing the practice and shared
experience; second person ("you") for direct step-by-step guidance
- Concrete images, sensory details
- Oral rhythm, short sentences
- Present tense, imperative mood
@@ -15,6 +16,18 @@ Default to warm, direct instructional voice (Thầy-adjacent):
MB corpus consultation is NOT needed for register — this content type has its own
well-established English conventions (yoga/meditation instructional voice).
### Person and subject (most-reviewed issue in this genre)
Chinese meditation prose drops subjects; English must not. The human reviewer's
#1 fix was to supply and keep a **consistent person as subject** — almost
always "we" for shared experience, "you" for direct guidance — and never to
drift into abstract third person ("people," "one") or leave a sentence
subjectless. Every sentence needs a named subject, and the reference chain
across sentences must stay consistent (don't hop between "we," "the
practitioner," and an inanimate noun). For the full pattern library from a
human-reviewed manuscript, read
`../mpi-translation-review/references/meditation-practice-translation.md`.
## Terms
Terms DB lookup for Buddhist-mindfulness vocabulary is useful but limited to key terms:
+21 -5
View File
@@ -1,18 +1,34 @@
# MPI Skills
These skills live in this directory and are loaded via `~/.hermes/config.yaml`.
These skills live in this directory and are loaded by your agent from this path.
## Install for Hermes
## Install
Add to `~/.hermes/config.yaml`:
Add the absolute path to this directory to your agent's external skill paths.
For Oh My Pi and compatible agents, this is usually a config file with an
`external_dirs` list:
```yaml
skills:
external_dirs:
- $MPI_PROJECT_ROOT/toolkit/skills
- /path/to/mpi/toolkit/skills
```
{% Edit config.yaml directly — `hermes config set` stores list values as strings. %}
{% Replace `/path/to/mpi` with the absolute path to your workspace. %}
## Structure
Each skill follows the Agent Skills layout:
```
skill-name/
├── SKILL.md # frontmatter + core instructions
├── scripts/ # tiny deterministic CLIs
├── references/ # docs loaded on demand
└── assets/ # templates and static files
```
The `name` field in each `SKILL.md` frontmatter matches the directory name.
## Skills
+1 -1
View File
@@ -2,7 +2,7 @@ FROM python:3.14-slim
WORKDIR /app
RUN pip install --no-cache-dir flask duckdb gunicorn
RUN pip install --no-cache-dir flask gunicorn
COPY . .
+16 -11
View File
@@ -1,26 +1,31 @@
#!/usr/bin/env python3
"""Full-text search over MPI term database. Queries unified_terms_flat via DuckDB LIKE.
#!/usr/bin/env -S uv run --script
# /// script
# dependencies = []
# ///
"""Full-text search over MPI term database (SQLite).
Module usage:
from search import search_terms
results = search_terms("空性")
results = search_terms("空性", limit=5, loc="心经", src="佛教术语")
from search import search
results = search("空性")
results = search("空性", limit=5, loc="心经", src="佛教术语")
# returns list of dicts: {zh, en, loc, source}
CLI usage:
python search.py <query> [limit]
python search.py 空性 loc:心经 src:公案
toolkit/terms-database/search.py <query> [limit]
toolkit/terms-database/search.py 空性 loc:心经 src:公案
"""
import sys
import os
import duckdb
import sqlite3
DB = os.path.join(os.path.dirname(os.path.abspath(__file__)), "termlib.duckdb")
DB = os.path.join(os.path.dirname(os.path.abspath(__file__)), "termlib.sqlite")
def _connect():
return duckdb.connect(DB, read_only=True)
con = sqlite3.connect(DB)
con.execute("PRAGMA journal_mode=WAL")
return con
def _search_rows(con, query, loc=None, src=None, limit=None):
@@ -42,7 +47,7 @@ def _search_rows(con, query, loc=None, src=None, limit=None):
where += " AND source = ?"
params.append(src)
sql = f"SELECT zh, en, loc, source FROM unified_terms_flat WHERE {where}"
sql = f"SELECT zh, en, loc, source FROM terms WHERE {where}"
if limit is not None:
sql += " LIMIT ?"
params.append(limit)
Regular → Executable
+16 -7
View File
@@ -1,22 +1,31 @@
#!/usr/bin/env python3
#!/usr/bin/env -S uv run --script
# /// script
# dependencies = ["flask"]
# ///
import sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import duckdb
import sqlite3
from flask import Flask, request, jsonify, render_template
import search as s
app = Flask(__name__)
def _connect():
con = sqlite3.connect(s.DB)
con.execute("PRAGMA journal_mode=WAL")
return con
def do_search(q, loc, src, limit):
with duckdb.connect(s.DB, read_only=True) as con:
rows = s.search(con, q, loc, src, limit)
return [{'zh': r[0], 'en': r[1], 'loc': r[2] or None, 'source': r[3]} for r in rows]
rows = s.search(q, loc=loc, src=src, limit=limit)
return [{'zh': r['zh'], 'en': r['en'], 'loc': r['loc'] or None, 'source': r['source']} for r in rows]
def do_sources():
with duckdb.connect(s.DB, read_only=True) as con:
with _connect() as con:
rows = con.execute(
'SELECT source, COUNT(*) AS cnt FROM unified_terms_flat GROUP BY source ORDER BY cnt DESC'
'SELECT source, COUNT(*) AS cnt FROM terms GROUP BY source ORDER BY cnt DESC'
).fetchall()
return [{'source': r[0], 'count': r[1]} for r in rows]
Binary file not shown.
Binary file not shown.