skills: update skills
This commit is contained in:
@@ -1 +1,2 @@
|
||||
.codegraph/
|
||||
.firecrawl/
|
||||
|
||||
@@ -7,7 +7,7 @@ Skills in `skills/`. Loaded via `~/.hermes/config.yaml`:
|
||||
```yaml
|
||||
skills:
|
||||
external_dirs:
|
||||
- /home/user/documents/mpi/skills
|
||||
- $MPI_PROJECT_ROOT/skills
|
||||
```
|
||||
|
||||
{% `hermes config set` stringifies list values — edit config.yaml directly. %}
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
"""Generate bilingual.dj from DOCX manuscript only (no PDF).
|
||||
Source: Chinese from DOCX. Target: English from DOCX.
|
||||
"""
|
||||
import re, subprocess, os
|
||||
import re, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
DOCX = "/home/user/documents/mpi/translate-files/佛教徒的人生态度/定稿 佛教徒的人生态度 善鑫慧炬照禅道靖妙一观轩慈德20260527.docx"
|
||||
OUT_DIR = "/home/user/documents/mpi/translate-files/佛教徒的人生态度"
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
DOCX = ROOT / "translate-files/佛教徒的人生态度/定稿 佛教徒的人生态度 善鑫慧炬照禅道靖妙一观轩慈德20260527.docx"
|
||||
OUT_DIR = ROOT / "translate-files/佛教徒的人生态度"
|
||||
|
||||
def has_cjk(s):
|
||||
return any('\u4e00' <= c <= '\u9fff' for c in s)
|
||||
@@ -124,5 +126,5 @@ if __name__ == '__main__':
|
||||
pairs = extract_pairs(text)
|
||||
print(f" Pairs: {len(pairs)}")
|
||||
|
||||
out = os.path.join(OUT_DIR, 'bilingual.dj')
|
||||
out = OUT_DIR / "bilingual.dj"
|
||||
generate(pairs, out)
|
||||
|
||||
@@ -6,11 +6,13 @@ Target: English from PDF typeset.
|
||||
Strategy: find each DOCX English paragraph in PDF body, extract
|
||||
the PDF text region for that paragraph using position boundaries.
|
||||
"""
|
||||
import re, subprocess, os
|
||||
import re, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
DOCX = "/home/user/documents/mpi/translate-files/佛教徒的人生态度/定稿 佛教徒的人生态度 善鑫慧炬照禅道靖妙一观轩慈德20260527.docx"
|
||||
PDF = "/home/user/documents/mpi/translate-files/佛教徒的人生态度/0607-二排-果澄-佛教徒的人生态度-一校-多人-0607.pdf"
|
||||
OUT_DIR = "/home/user/documents/mpi/translate-files/佛教徒的人生态度"
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
DOCX = ROOT / "translate-files/佛教徒的人生态度/定稿 佛教徒的人生态度 善鑫慧炬照禅道靖妙一观轩慈德20260527.docx"
|
||||
PDF = ROOT / "translate-files/佛教徒的人生态度/0607-二排-果澄-佛教徒的人生态度-一校-多人-0607.pdf"
|
||||
OUT_DIR = ROOT / "translate-files/佛教徒的人生态度"
|
||||
|
||||
|
||||
def has_cjk(s):
|
||||
@@ -256,5 +258,5 @@ if __name__ == '__main__':
|
||||
print("Extracting segments...")
|
||||
segments = extract_segments(pdf_body, positions)
|
||||
|
||||
out = os.path.join(OUT_DIR, 'bilingual.dj')
|
||||
out = OUT_DIR / "bilingual.dj"
|
||||
generate(pairs, segments, out)
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
"""Generate bilingual.dj from DOCX for 「生命也可以被设计的」.
|
||||
One-pass approach: walk interleaved paragraphs, handle multi-CN sequences.
|
||||
"""
|
||||
import re, subprocess, os
|
||||
import re, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
DOCX = "/home/user/documents/mpi/translate-files/生命也可以被设计的/中英文定稿-260324-生命也是可以被设计的-妙一宽山静雅初翻 慈鎏妙一审议 宽山定稿.docx"
|
||||
OUT_DIR = "/home/user/documents/mpi/translate-files/生命也可以被设计的"
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
DOCX = ROOT / "translate-files/生命也可以被设计的/中英文定稿-260324-生命也是可以被设计的-妙一宽山静雅初翻 慈鎏妙一审议 宽山定稿.docx"
|
||||
OUT_DIR = ROOT / "translate-files/生命也可以被设计的"
|
||||
|
||||
def has_cjk(s):
|
||||
return any('\u4e00' <= c <= '\u9fff' for c in s)
|
||||
@@ -177,5 +179,5 @@ if __name__ == '__main__':
|
||||
pairs = extract_body_pairs(text)
|
||||
print(f" Pairs: {len(pairs)}")
|
||||
|
||||
out = os.path.join(OUT_DIR, 'bilingual.dj')
|
||||
out = OUT_DIR / "bilingual.dj"
|
||||
generate(toc_cn, toc_en, pairs, out)
|
||||
|
||||
@@ -34,6 +34,49 @@ The script handles three file patterns:
|
||||
- Inline page numbers (standalone 1-2 digit lines)
|
||||
- Trailing blank lines
|
||||
|
||||
## Beyond the script: bold fragments, conjoined paragraphs, encoding
|
||||
|
||||
The script handles simple fixed-width body text. Some PDF→markdown conversions produce more complex artifacts that need manual multi-pass Python scripts via `execute_code`:
|
||||
|
||||
### Bold marker fragmentation
|
||||
|
||||
`**...text...**` blocks split across blank lines with stray `**` at fragment boundaries:
|
||||
|
||||
```
|
||||
**第三条 特色——依据五大要素,构建次第修学。营造良好氛围,提供有效**
|
||||
|
||||
引导。
|
||||
```
|
||||
|
||||
**Fix**: Join fragments, remove stray `**` from join point, add closing `**` to final result. See `references/bold-fragments.md` for full pattern catalog and multi-pass workflow.
|
||||
|
||||
**Critical pitfall**: Do NOT join lines where BOTH the first and second line are complete bold blocks (start+end with `**`). These are separate entries, not fragments:
|
||||
```
|
||||
**第一条 ...之道。** ← complete bold item
|
||||
← blank line
|
||||
**第二条 ...合一。** ← complete bold item (DON'T JOIN)
|
||||
```
|
||||
|
||||
### Conjoined paragraphs
|
||||
|
||||
Separate paragraphs/sections merged into one line — opposite problem to the script. Common in song lyrics, dense instructional sections. Requires semantic splitting. See `references/bold-fragments.md`.
|
||||
|
||||
### Encoding artifacts
|
||||
|
||||
`川` (U+5DDD) replacing `"` (curly quote) — search-and-replace: `" 道理川` → `"道理"`, `" 自己的川` → `"自己的"`.
|
||||
|
||||
### Multi-pass approach
|
||||
|
||||
1. **Pass 1**: Join word fragments split by blank lines (conservative — only when current line doesn't end with `。!?` or is NOT a complete bold block)
|
||||
2. **Pass 2**: Split obviously conjoined paragraphs (manual string replacements for known patterns)
|
||||
3. **Pass 3**: Fix stray bold markers, encoding artifacts, stray page numbers
|
||||
4. Verify after each pass; revert with `git checkout` if over-aggressive
|
||||
|
||||
### Heuristic pitfalls
|
||||
|
||||
- **Short-line join** (< 15 chars): Over-joins section headers with body, Q&A pairs (`正念是什么?\n\n就是...`). Only use for clear word-fragment continuations.
|
||||
- **Bold-end join**: Lines ending with `**` are ambiguous — either broken bold fragment or complete bold item. Check if the content before `**` forms a complete sentence (ends with `。`).
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **TOC detection boundaries**: The vertical TOC end is detected by finding the first line with 3+ CJK characters. If a page number like "2" sits between TOC and body, it lands in the TOC section — harmless but visible.
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# Bold fragments & conjoined paragraphs — fix patterns
|
||||
|
||||
From session fixing `静心学堂学员手册.md` (1575→1478 lines, ~100 fixes).
|
||||
|
||||
## Pattern A: Bold marker fragmentation
|
||||
|
||||
**Problem**: `**...text...**` block split across blank line with stray `**` markers:
|
||||
|
||||
```
|
||||
**第三条 特色——依据五大要素,构建次第修学。营造良好氛围,提供有效**
|
||||
|
||||
引导。
|
||||
```
|
||||
|
||||
**Detection**: Line ends with `**`, next non-blank line continues the sentence (does NOT start with `**`).
|
||||
|
||||
**Fix** (Python):
|
||||
```python
|
||||
# curr ends with **, nxt is continuation (no leading **)
|
||||
curr_fixed = curr.rstrip()[:-2].rstrip() # strip trailing **
|
||||
nxt_fixed = nxt.lstrip()
|
||||
if nxt_fixed.endswith('**'):
|
||||
nxt_fixed = nxt_fixed[:-2].rstrip()
|
||||
joined = curr_fixed + nxt_fixed + '**'
|
||||
else:
|
||||
joined = curr_fixed + nxt_fixed # lost closing ** — may need manual fix
|
||||
```
|
||||
|
||||
### Anti-pattern: Complete bold items
|
||||
|
||||
Do NOT join when BOTH lines are complete bold blocks (start+end with `**`):
|
||||
|
||||
```
|
||||
**第一条 ...之道。** ← DON'T JOIN
|
||||
← blank line
|
||||
**第二条 ...合一。** ← DON'T JOIN
|
||||
```
|
||||
|
||||
**Detection**: Both `curr` and `nxt` start with `**` and end with `**`.
|
||||
|
||||
## Pattern B: Conjoined paragraphs (Type 2)
|
||||
|
||||
Separate sections merged into one line. Common cases:
|
||||
|
||||
### Section headers merged with body
|
||||
```
|
||||
导言:这本指引怎么用这本指引是什么这是一本修学地图...
|
||||
```
|
||||
→ Split into:
|
||||
```
|
||||
导言:这本指引怎么用
|
||||
|
||||
这本指引是什么
|
||||
|
||||
这是一本修学地图...
|
||||
```
|
||||
|
||||
### Song titles merged mid-lyrics
|
||||
```
|
||||
...生生世世不再久违《菩提花开》如果你渴求一滴水...
|
||||
```
|
||||
→ Split into:
|
||||
```
|
||||
...生生世世不再久违
|
||||
|
||||
### 《菩提花开》
|
||||
|
||||
如果你渴求一滴水...
|
||||
```
|
||||
|
||||
### List items merged into one line
|
||||
```
|
||||
不在班级群发布...不从事违法活动不在班级平台拉拢...
|
||||
```
|
||||
→ Split into bullet list:
|
||||
```
|
||||
- 不在班级群发布...
|
||||
- 不从事违法活动
|
||||
- 不在班级平台拉拢...
|
||||
```
|
||||
|
||||
**Approach**: Manual string replacements for known patterns. Regex is unreliable for semantic splits.
|
||||
|
||||
## Pattern C: Stray page numbers
|
||||
|
||||
Standalone digits at line ends, often from PDF page number artifacts:
|
||||
- `42`, `43`, `46`, `47` at end of content lines
|
||||
|
||||
**Fix**: Strip trailing digits that aren't part of dates, durations, or course numbers.
|
||||
|
||||
## Pattern D: Encoding artifacts
|
||||
|
||||
`川` (U+5DDD) replacing curly quotes `"` (U+201C/U+201D):
|
||||
```
|
||||
把" 道理川变成" 自己的川 → 把"道理"变成"自己的"
|
||||
```
|
||||
|
||||
**Fix**: Replace `" 道理川` → `"道理"`, `" 自己的川` → `"自己的"`.
|
||||
|
||||
## Multi-pass workflow
|
||||
|
||||
1. **Pass 1 — Join word fragments**: Scan for lines split by blank line where first line doesn't end with `。!?` and neither line is structural (header/list/table). Skip complete bold items.
|
||||
2. **Pass 2 — Split conjoined**: Apply known string replacements for merged sections, song transitions, list items.
|
||||
3. **Pass 3 — Clean artifacts**: Fix stray `**` markers, encoding issues, stray page numbers.
|
||||
4. **Verify**: `git diff` after each pass; `git checkout` if over-aggressive.
|
||||
|
||||
## Rejected heuristics
|
||||
|
||||
- **Short-line join** (< 15 chars): Over-joins section headers (`中级和高级(以后的事)`) with body, and Q&A pairs (`正念是什么?\n\n就是...`). Only use for clear mid-word fragments.
|
||||
- **Blind `**` stripping**: Removes valid bold formatting from complete bold items.
|
||||
+1
-1
@@ -9,7 +9,7 @@ Add to `~/.hermes/config.yaml`:
|
||||
```yaml
|
||||
skills:
|
||||
external_dirs:
|
||||
- /home/user/documents/mpi/skills
|
||||
- $MPI_PROJECT_ROOT/skills
|
||||
```
|
||||
|
||||
{% Edit config.yaml directly — `hermes config set` stores list values as strings. %}
|
||||
|
||||
@@ -6,14 +6,14 @@ category: research
|
||||
|
||||
# Terms Search
|
||||
|
||||
Database: `/home/user/documents/mpi/terms-search/termlib.duckdb`
|
||||
CLI: `/home/user/documents/mpi/terms-search/search.py`
|
||||
Server: `/home/user/documents/mpi/terms-search/server.py`
|
||||
Database: `$MPI_PROJECT_ROOT/terms-search/termlib.duckdb`
|
||||
CLI: `$MPI_PROJECT_ROOT/terms-search/search.py`
|
||||
Server: `$MPI_PROJECT_ROOT/terms-search/server.py`
|
||||
|
||||
## CLI (preferred)
|
||||
|
||||
```
|
||||
/home/user/documents/mpi/terms-search/search.py <query> [limit]
|
||||
$MPI_PROJECT_ROOT/terms-search/search.py <query> [limit]
|
||||
```
|
||||
|
||||
Multi-word queries are ANDed. Searches both `zh` and `en` columns.
|
||||
@@ -22,7 +22,7 @@ Multi-word queries are ANDed. Searches both `zh` and `en` columns.
|
||||
|
||||
```python
|
||||
import sys
|
||||
sys.path.insert(0, '/home/user/documents/mpi/terms-search')
|
||||
sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
|
||||
from search import search
|
||||
results = search("空性", limit=5, src="DoT定稿")
|
||||
# → list of {zh, en, loc, source} dicts
|
||||
@@ -32,7 +32,7 @@ Use this inside `execute_code` scripts for batch lookups — no subprocess neede
|
||||
|
||||
## HTTP API (use only when CLI is insufficient)
|
||||
|
||||
Start: `python3 /home/user/documents/mpi/terms-search/server.py` (port 8910)
|
||||
Start: `python3 $MPI_PROJECT_ROOT/terms-search/server.py` (port 8910)
|
||||
|
||||
- `GET /` — plain HTML UI (form + results table, no CSS)
|
||||
- `GET /` — plain HTML UI (form + results table, no CSS)
|
||||
@@ -64,14 +64,14 @@ Errors return `{"error": "..."}` with HTTP 500 (API) or shown inline (UI).
|
||||
## Direct DuckDB
|
||||
|
||||
```
|
||||
duckdb /home/user/documents/mpi/terms-search/termlib.duckdb
|
||||
duckdb $MPI_PROJECT_ROOT/terms-search/termlib.duckdb
|
||||
```
|
||||
|
||||
Key tables: `unified_terms_flat` (zh, en, loc, source), individual source tables, `unified_terms` view.
|
||||
|
||||
## Rebuilding
|
||||
|
||||
Terms data comes from `/home/user/documents/mpi/guide/03 术语库/`. To rebuild:
|
||||
Terms data comes from `$MPI_PROJECT_ROOT/guide/03 术语库/`. To rebuild:
|
||||
1. Convert source xlsx/ods → CSV+YAML in `_output/`
|
||||
2. Rebuild DuckDB from CSVs
|
||||
3. Materialize `unified_terms_flat` view → table for performance
|
||||
|
||||
@@ -10,7 +10,7 @@ After producing a first-pass translation, or when the user asks to check termino
|
||||
|
||||
1. Read the full translated file. Extract all Chinese terms from `{% "TERM" (pinyin) = ENGLISH ... %}` blocks.
|
||||
|
||||
2. Start the search server: `python3 /home/user/documents/mpi/terms-search/server.py &` (port 8910). It may already be running — check with `curl -s http://localhost:8910/`.
|
||||
2. Start the search server: `python3 $MPI_PROJECT_ROOT/terms-search/server.py &` (port 8910). It may already be running — check with `curl -s http://localhost:8910/`.
|
||||
|
||||
3. Batch-search each term via the HTTP API:
|
||||
```
|
||||
|
||||
@@ -87,7 +87,7 @@ Use `terminal: cat` — `read_file` deduplicates within a session.
|
||||
|
||||
**Terms database drift** (systematic):
|
||||
- Cross-reference glossary terms against the MPI terms database
|
||||
- CLI preferred: `python3 /home/user/documents/mpi/terms-search/search.py <query>`. For a review, batch many queries in one `execute_code` script (subprocess loop) — one terminal call per term is slow and noisy.
|
||||
- CLI preferred: `python3 $MPI_PROJECT_ROOT/terms-search/search.py <query>`. For a review, batch many queries in one `execute_code` script (subprocess loop) — one terminal call per term is slow and noisy.
|
||||
- Source priority: DoT定稿 > 内部特色词 > 佛教术语 > 经论名
|
||||
- Fix both glossary comments AND body text
|
||||
- See `references/terms-db-alignment.md` for batch-lookup patterns
|
||||
|
||||
@@ -8,7 +8,7 @@ Import directly in `execute_code` scripts — no subprocess, no server, no text
|
||||
|
||||
```python
|
||||
import sys
|
||||
sys.path.insert(0, '/home/user/documents/mpi/terms-search')
|
||||
sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
|
||||
from search import search
|
||||
|
||||
results = search("三级修学", limit=5)
|
||||
@@ -20,7 +20,7 @@ results = search("空性", loc="心经", src="DoT定稿", limit=5)
|
||||
|
||||
```python
|
||||
import sys
|
||||
sys.path.insert(0, '/home/user/documents/mpi/terms-search')
|
||||
sys.path.insert(0, '$MPI_PROJECT_ROOT/terms-search')
|
||||
from search import search
|
||||
|
||||
terms = ["三无漏学", "八步三禅", "闻思修", ...]
|
||||
|
||||
@@ -73,5 +73,5 @@ initial/middle/great scope."
|
||||
### Translating before consulting terms DB
|
||||
|
||||
Always search key terms BEFORE translating. The dharma-translation skill says to do
|
||||
this, but it's easy to skip. Use the CLI: `/home/user/documents/mpi/terms-search/search.py <query>`.
|
||||
this, but it's easy to skip. Use the CLI: `$MPI_PROJECT_ROOT/terms-search/search.py <query>`.
|
||||
Prioritize DoT定稿 > 内部特色词 > 佛教术语 > 经论名.
|
||||
|
||||
@@ -56,4 +56,5 @@ When translating guided meditation scripts, exercise guides, or posture instruct
|
||||
|
||||
- `references/meditation-translation.md` — lighter workflow for meditation/mindfulness content
|
||||
- `references/markdown-to-djot.md` — converting .docx.md to .dj for translation prep
|
||||
- `references/bilingual-format.md` — bilingual.dj layout: source/target adjacent, blank separator between pairs
|
||||
- `references/diacritics-convention.md` — diacritics rules
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# Bilingual DJ Format
|
||||
|
||||
## Layout
|
||||
|
||||
Each pair: source line immediately followed by target line. Blank line separates pairs.
|
||||
|
||||
```
|
||||
source-line
|
||||
target-line
|
||||
|
||||
source-line
|
||||
target-line
|
||||
```
|
||||
|
||||
NOT:
|
||||
```
|
||||
source-line
|
||||
← WRONG: extra blank between source and target
|
||||
target-line
|
||||
```
|
||||
|
||||
## Creating initial bilingual from source only
|
||||
|
||||
Every source line gets an empty target placeholder + blank separator:
|
||||
|
||||
```
|
||||
source-A
|
||||
|
||||
source-B
|
||||
```
|
||||
|
||||
(2 blank lines between consecutive source lines: empty target + separator.)
|
||||
|
||||
## Verification
|
||||
|
||||
Source line count × 3 − 1 = bilingual line count (before trailing newline strip).
|
||||
@@ -11,6 +11,23 @@ sed -n '1,218p' combined.md > a1.md
|
||||
sed -n '220,282p' combined.md > a2.md
|
||||
```
|
||||
|
||||
## TOC stripping
|
||||
|
||||
Pandoc docx→md produces a markdown TOC with tab-separated page numbers:
|
||||
|
||||
```markdown
|
||||
[一、对佛教的感悟\t1](#一、对佛教的感悟)
|
||||
[二、佛教与人类文明\t5](#二、佛教与人类文明)
|
||||
```
|
||||
|
||||
Strip before conversion:
|
||||
|
||||
```bash
|
||||
sed -i '/^\[.*\t.*\](#.*)$/d' input.md
|
||||
```
|
||||
|
||||
Or in Python: skip lines matching `line.startswith("[") and "\t" in line and "](#" in line`.
|
||||
|
||||
## Heading anchor cleanup
|
||||
|
||||
Pandoc's docx→md conversion adds `{#heading-id}` anchors to every heading:
|
||||
|
||||
Reference in New Issue
Block a user