skills: update skills

2026-06-18 08:24:11 +08:00
parent 8f086cd3da
commit 12134faea9
16 changed files with 242 additions and 28 deletions
@@ -34,6 +34,49 @@ The script handles three file patterns:
 - Inline page numbers (standalone 1-2 digit lines)
 - Trailing blank lines

+## Beyond the script: bold fragments, conjoined paragraphs, encoding
+
+The script handles simple fixed-width body text. Some PDF→markdown conversions produce more complex artifacts that need manual multi-pass Python scripts via `execute_code`:
+
+### Bold marker fragmentation
+
+`**...text...**` blocks split across blank lines with stray `**` at fragment boundaries:
+
+```
+**第三条 特色——依据五大要素，构建次第修学。营造良好氛围，提供有效**
+
+引导。
+```
+
+**Fix**: Join fragments, remove stray `**` from join point, add closing `**` to final result. See `references/bold-fragments.md` for full pattern catalog and multi-pass workflow.
+
+**Critical pitfall**: Do NOT join lines where BOTH the first and second line are complete bold blocks (start+end with `**`). These are separate entries, not fragments:
+```
+**第一条 ...之道。**    ← complete bold item
+                        ← blank line
+**第二条 ...合一。**    ← complete bold item (DON'T JOIN)
+```
+
+### Conjoined paragraphs
+
+Separate paragraphs/sections merged into one line — opposite problem to the script. Common in song lyrics, dense instructional sections. Requires semantic splitting. See `references/bold-fragments.md`.
+
+### Encoding artifacts
+
+`川` (U+5DDD) replacing `"` (curly quote) — search-and-replace: `" 道理川` → `"道理"`, `" 自己的川` → `"自己的"`.
+
+### Multi-pass approach
+
+1. **Pass 1**: Join word fragments split by blank lines (conservative — only when current line doesn't end with `。！？` or is NOT a complete bold block)
+2. **Pass 2**: Split obviously conjoined paragraphs (manual string replacements for known patterns)
+3. **Pass 3**: Fix stray bold markers, encoding artifacts, stray page numbers
+4. Verify after each pass; revert with `git checkout` if over-aggressive
+
+### Heuristic pitfalls
+
+- **Short-line join** (< 15 chars): Over-joins section headers with body, Q&A pairs (`正念是什么？\n\n就是...`). Only use for clear word-fragment continuations.
+- **Bold-end join**: Lines ending with `**` are ambiguous — either broken bold fragment or complete bold item. Check if the content before `**` forms a complete sentence (ends with `。`).
+
 ## Pitfalls

 - **TOC detection boundaries**: The vertical TOC end is detected by finding the first line with 3+ CJK characters. If a page number like "2" sits between TOC and body, it lands in the TOC section — harmless but visible.
@@ -0,0 +1,110 @@
+# Bold fragments & conjoined paragraphs — fix patterns
+
+From session fixing `静心学堂学员手册.md` (1575→1478 lines, ~100 fixes).
+
+## Pattern A: Bold marker fragmentation
+
+**Problem**: `**...text...**` block split across blank line with stray `**` markers:
+
+```
+**第三条 特色——依据五大要素，构建次第修学。营造良好氛围，提供有效**
+
+引导。
+```
+
+**Detection**: Line ends with `**`, next non-blank line continues the sentence (does NOT start with `**`).
+
+**Fix** (Python):
+```python
+# curr ends with **, nxt is continuation (no leading **)
+curr_fixed = curr.rstrip()[:-2].rstrip()  # strip trailing **
+nxt_fixed = nxt.lstrip()
+if nxt_fixed.endswith('**'):
+    nxt_fixed = nxt_fixed[:-2].rstrip()
+    joined = curr_fixed + nxt_fixed + '**'
+else:
+    joined = curr_fixed + nxt_fixed  # lost closing ** — may need manual fix
+```
+
+### Anti-pattern: Complete bold items
+
+Do NOT join when BOTH lines are complete bold blocks (start+end with `**`):
+
+```
+**第一条 ...之道。**    ← DON'T JOIN
+                        ← blank line
+**第二条 ...合一。**    ← DON'T JOIN
+```
+
+**Detection**: Both `curr` and `nxt` start with `**` and end with `**`.
+
+## Pattern B: Conjoined paragraphs (Type 2)
+
+Separate sections merged into one line. Common cases:
+
+### Section headers merged with body
+```
+导言：这本指引怎么用这本指引是什么这是一本修学地图...
+```
+→ Split into:
+```
+导言：这本指引怎么用
+
+这本指引是什么
+
+这是一本修学地图...
+```
+
+### Song titles merged mid-lyrics
+```
+...生生世世不再久违《菩提花开》如果你渴求一滴水...
+```
+→ Split into:
+```
+...生生世世不再久违
+
+### 《菩提花开》
+
+如果你渴求一滴水...
+```
+
+### List items merged into one line
+```
+不在班级群发布...不从事违法活动不在班级平台拉拢...
+```
+→ Split into bullet list:
+```
+- 不在班级群发布...
+- 不从事违法活动
+- 不在班级平台拉拢...
+```
+
+**Approach**: Manual string replacements for known patterns. Regex is unreliable for semantic splits.
+
+## Pattern C: Stray page numbers
+
+Standalone digits at line ends, often from PDF page number artifacts:
+- `42`, `43`, `46`, `47` at end of content lines
+
+**Fix**: Strip trailing digits that aren't part of dates, durations, or course numbers.
+
+## Pattern D: Encoding artifacts
+
+`川` (U+5DDD) replacing curly quotes `"` (U+201C/U+201D):
+```
+把" 道理川变成" 自己的川    →    把"道理"变成"自己的"
+```
+
+**Fix**: Replace `" 道理川` → `"道理"`, `" 自己的川` → `"自己的"`.
+
+## Multi-pass workflow
+
+1. **Pass 1 — Join word fragments**: Scan for lines split by blank line where first line doesn't end with `。！？` and neither line is structural (header/list/table). Skip complete bold items.
+2. **Pass 2 — Split conjoined**: Apply known string replacements for merged sections, song transitions, list items.
+3. **Pass 3 — Clean artifacts**: Fix stray `**` markers, encoding issues, stray page numbers.
+4. **Verify**: `git diff` after each pass; `git checkout` if over-aggressive.
+
+## Rejected heuristics
+
+- **Short-line join** (< 15 chars): Over-joins section headers (`中级和高级（以后的事）`) with body, and Q&A pairs (`正念是什么？\n\n就是...`). Only use for clear mid-word fragments.
+- **Blind `**` stripping**: Removes valid bold formatting from complete bold items.
@@ -9,7 +9,7 @@ Add to `~/.hermes/config.yaml`:
 ```yaml
 skills:
  external_dirs:
-  - /home/user/documents/mpi/skills
+  - $MPI_PROJECT_ROOT/skills
 ```

 {% Edit config.yaml directly — `hermes config set` stores list values as strings. %}
@@ -6,14 +6,14 @@ category: research

 # Terms Search

-Database: `/home/user/documents/mpi/terms-search/termlib.duckdb`
-CLI: `/home/user/documents/mpi/terms-search/search.py`
-Server: `/home/user/documents/mpi/terms-search/server.py`
+Database: `$MPI_PROJECT_ROOT/terms-search/termlib.duckdb`
+CLI: `$MPI_PROJECT_ROOT/terms-search/search.py`
+Server: `$MPI_PROJECT_ROOT/terms-search/server.py`

 ## CLI (preferred)

 ```
-/home/user/documents/mpi/terms-search/search.py <query> [limit]
+$MPI_PROJECT_ROOT/terms-search/search.py <query> [limit]
 ```

 Multi-word queries are ANDed. Searches both `zh` and `en` columns.
@@ -22,7 +22,7 @@ Multi-word queries are ANDed. Searches both `zh` and `en` columns.

 ```python
 import sys
-sys.path.insert(0, '/home/user/documents/mpi/terms-search')
+sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
 from search import search
 results = search("空性", limit=5, src="DoT定稿")
 # → list of {zh, en, loc, source} dicts
@@ -32,7 +32,7 @@ Use this inside `execute_code` scripts for batch lookups — no subprocess neede

 ## HTTP API (use only when CLI is insufficient)

-Start: `python3 /home/user/documents/mpi/terms-search/server.py` (port 8910)
+Start: `python3 $MPI_PROJECT_ROOT/terms-search/server.py` (port 8910)

 - `GET /` — plain HTML UI (form + results table, no CSS)
 - `GET /` — plain HTML UI (form + results table, no CSS)
@@ -64,14 +64,14 @@ Errors return `{"error": "..."}` with HTTP 500 (API) or shown inline (UI).
 ## Direct DuckDB

 ```
-duckdb /home/user/documents/mpi/terms-search/termlib.duckdb
+duckdb $MPI_PROJECT_ROOT/terms-search/termlib.duckdb
 ```

 Key tables: `unified_terms_flat` (zh, en, loc, source), individual source tables, `unified_terms` view.

 ## Rebuilding

-Terms data comes from `/home/user/documents/mpi/guide/03 术语库/`. To rebuild:
+Terms data comes from `$MPI_PROJECT_ROOT/guide/03 术语库/`. To rebuild:
 1. Convert source xlsx/ods → CSV+YAML in `_output/`
 2. Rebuild DuckDB from CSVs
 3. Materialize `unified_terms_flat` view → table for performance
@@ -10,7 +10,7 @@ After producing a first-pass translation, or when the user asks to check termino

 1. Read the full translated file. Extract all Chinese terms from `{% "TERM" (pinyin) = ENGLISH ... %}` blocks.

-2. Start the search server: `python3 /home/user/documents/mpi/terms-search/server.py &` (port 8910). It may already be running — check with `curl -s http://localhost:8910/`.
+2. Start the search server: `python3 $MPI_PROJECT_ROOT/terms-search/server.py &` (port 8910). It may already be running — check with `curl -s http://localhost:8910/`.

 3. Batch-search each term via the HTTP API:
   ```
@@ -87,7 +87,7 @@ Use `terminal: cat` — `read_file` deduplicates within a session.

 **Terms database drift** (systematic):
 - Cross-reference glossary terms against the MPI terms database
- CLI preferred: `python3 /home/user/documents/mpi/terms-search/search.py <query>`. For a review, batch many queries in one `execute_code` script (subprocess loop) — one terminal call per term is slow and noisy.
+- CLI preferred: `python3 $MPI_PROJECT_ROOT/terms-search/search.py <query>`. For a review, batch many queries in one `execute_code` script (subprocess loop) — one terminal call per term is slow and noisy.
 - Source priority: DoT定稿 > 内部特色词 > 佛教术语 > 经论名
 - Fix both glossary comments AND body text
 - See `references/terms-db-alignment.md` for batch-lookup patterns
@@ -8,7 +8,7 @@ Import directly in `execute_code` scripts — no subprocess, no server, no text

 ```python
 import sys
-sys.path.insert(0, '/home/user/documents/mpi/terms-search')
+sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
 from search import search

 results = search("三级修学", limit=5)
@@ -20,7 +20,7 @@ results = search("空性", loc="心经", src="DoT定稿", limit=5)

 ```python
 import sys
-sys.path.insert(0, '/home/user/documents/mpi/terms-search')
+sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
 from search import search

 terms = ["三无漏学", "八步三禅", "闻思修", ...]
@@ -73,5 +73,5 @@ initial/middle/great scope."
 ### Translating before consulting terms DB

 Always search key terms BEFORE translating. The dharma-translation skill says to do
-this, but it's easy to skip. Use the CLI: `/home/user/documents/mpi/terms-search/search.py <query>`.
+this, but it's easy to skip. Use the CLI: `$MPI_PROJECT_ROOT/terms-search/search.py <query>`.
 Prioritize DoT定稿 > 内部特色词 > 佛教术语 > 经论名.
@@ -56,4 +56,5 @@ When translating guided meditation scripts, exercise guides, or posture instruct

 - `references/meditation-translation.md` — lighter workflow for meditation/mindfulness content
 - `references/markdown-to-djot.md` — converting .docx.md to .dj for translation prep
+- `references/bilingual-format.md` — bilingual.dj layout: source/target adjacent, blank separator between pairs
 - `references/diacritics-convention.md` — diacritics rules
@@ -0,0 +1,36 @@
+# Bilingual DJ Format
+
+## Layout
+
+Each pair: source line immediately followed by target line. Blank line separates pairs.
+
+```
+source-line
+target-line
+
+source-line
+target-line
+```
+
+NOT:
+```
+source-line
+              ← WRONG: extra blank between source and target
+target-line
+```
+
+## Creating initial bilingual from source only
+
+Every source line gets an empty target placeholder + blank separator:
+
+```
+source-A
+
+source-B
+```
+
+(2 blank lines between consecutive source lines: empty target + separator.)
+
+## Verification
+
+Source line count × 3 − 1 = bilingual line count (before trailing newline strip).
@@ -11,6 +11,23 @@ sed -n '1,218p' combined.md > a1.md
 sed -n '220,282p' combined.md > a2.md
 ```

+## TOC stripping
+
+Pandoc docx→md produces a markdown TOC with tab-separated page numbers:
+
+```markdown
+[一、对佛教的感悟\t1](#一、对佛教的感悟)
+[二、佛教与人类文明\t5](#二、佛教与人类文明)
+```
+
+Strip before conversion:
+
+```bash
+sed -i '/^\[.*\t.*\](#.*)$/d' input.md
+```
+
+Or in Python: skip lines matching `line.startswith("[") and "\t" in line and "](#" in line`.
+
 ## Heading anchor cleanup

 Pandoc's docx→md conversion adds `{#heading-id}` anchors to every heading: