Initial toolkit: scripts, references, skills, and term database

2026-07-10 21:01:02 +08:00
commit ac9e6e3d1f
51 changed files with 5192 additions and 0 deletions
@@ -0,0 +1,85 @@
+---
+name: chinese-text-normalize
+description: Normalize Chinese markdown files — remove extraneous mid-sentence line breaks from fixed-width exports while preserving TOC structures, section headers, and intentional paragraph breaks.
+---
+
+When Chinese text has hard line breaks at a fixed width (~20-25 chars) — common in PDF exports, OCR output, or poorly-converted documents — use this skill to join them into flowing paragraphs.
+
+## Triggers
+
+- User asks to "fix line breaks" or "remove extraneous breaks" in Chinese text
+- Chinese markdown files with lines that break mid-sentence at a consistent short width
+- Files with vertical TOC (single-char-per-line 【】 sections) that need preservation
+
+## Approach
+
+Run `scripts/normalize_breaks.py <directory>` — it processes all .md files in the directory.
+
+The script handles three file patterns:
+
+1. **Vertical TOC + fixed-width body** — Preserves the decorative single-char TOC section, joins body paragraphs, strips inline page numbers (standalone digits like "3", "4")
+2. **Outline TOC with stray breaks** — Preserves numbered outline items (一、...、1、...、...... separators), joins body paragraphs
+3. **Already in paragraph format** — No change (safe to run idempotently)
+
+### What it preserves
+
+- Vertical TOC: single CJK/punctuation lines with 【】 brackets
+- Section headers: 【...】、## ...、# ...、一、二、三、...、1、2、3、...
+- Outline TOC entries: short numbered lines, lines with ...... separators
+- Blank lines as paragraph separators
+
+### What it removes
+
+- Mid-sentence hard line breaks (joins consecutive CJK body lines)
+- Inline page numbers (standalone 1-2 digit lines)
+- Trailing blank lines
+
+## Beyond the script: bold fragments, conjoined paragraphs, encoding
+
+The script handles simple fixed-width body text. Some PDF→markdown conversions produce more complex artifacts that need manual multi-pass Python scripts via `execute_code`:
+
+### Bold marker fragmentation
+
+`**...text...**` blocks split across blank lines with stray `**` at fragment boundaries:
+
+```
+**第三条 特色——依据五大要素，构建次第修学。营造良好氛围，提供有效**
+
+引导。
+```
+
+**Fix**: Join fragments, remove stray `**` from join point, add closing `**` to final result. See `references/bold-fragments.md` for full pattern catalog and multi-pass workflow.
+
+**Critical pitfall**: Do NOT join lines where BOTH the first and second line are complete bold blocks (start+end with `**`). These are separate entries, not fragments:
+```
+**第一条 ...之道。**    ← complete bold item
+                        ← blank line
+**第二条 ...合一。**    ← complete bold item (DON'T JOIN)
+```
+
+### Conjoined paragraphs
+
+Separate paragraphs/sections merged into one line — opposite problem to the script. Common in song lyrics, dense instructional sections. Requires semantic splitting. See `references/bold-fragments.md`.
+
+### Encoding artifacts
+
+`川` (U+5DDD) replacing `"` (curly quote) — search-and-replace: `" 道理川` → `"道理"`, `" 自己的川` → `"自己的"`.
+
+### Multi-pass approach
+
+1. **Pass 1**: Join word fragments split by blank lines (conservative — only when current line doesn't end with `。！？` or is NOT a complete bold block)
+2. **Pass 2**: Split obviously conjoined paragraphs (manual string replacements for known patterns)
+3. **Pass 3**: Fix stray bold markers, encoding artifacts, stray page numbers
+4. Verify after each pass; revert with `git checkout` if over-aggressive
+
+### Heuristic pitfalls
+
+- **Short-line join** (< 15 chars): Over-joins section headers with body, Q&A pairs (`正念是什么？\n\n就是...`). Only use for clear word-fragment continuations.
+- **Bold-end join**: Lines ending with `**` are ambiguous — either broken bold fragment or complete bold item. Check if the content before `**` forms a complete sentence (ends with `。`).
+
+## Pitfalls
+
+- **TOC detection boundaries**: The vertical TOC end is detected by finding the first line with 3+ CJK characters. If a page number like "2" sits between TOC and body, it lands in the TOC section — harmless but visible.
+- **Section headers without markers**: Plain-text section titles (e.g., "生命可以被设计的依据") without 【】 or number prefixes won't be detected as headers. They'll form standalone paragraphs separated by blank lines, which is fine as long as blank lines exist around them.
+- **Wiki-link TOC files**: Files like a course index with [[wiki links]] are NOT prose and should be excluded. The script has no special handling — skip those files manually or restore from git.
+- **Not for mixed CJK/English prose**: The script treats any line with CJK characters as body text. Mixed-language documents may need manual review.
@@ -0,0 +1,110 @@
+# Bold fragments & conjoined paragraphs — fix patterns
+
+From session fixing `静心学堂学员手册.md` (1575→1478 lines, ~100 fixes).
+
+## Pattern A: Bold marker fragmentation
+
+**Problem**: `**...text...**` block split across blank line with stray `**` markers:
+
+```
+**第三条 特色——依据五大要素，构建次第修学。营造良好氛围，提供有效**
+
+引导。
+```
+
+**Detection**: Line ends with `**`, next non-blank line continues the sentence (does NOT start with `**`).
+
+**Fix** (Python):
+```python
+# curr ends with **, nxt is continuation (no leading **)
+curr_fixed = curr.rstrip()[:-2].rstrip()  # strip trailing **
+nxt_fixed = nxt.lstrip()
+if nxt_fixed.endswith('**'):
+    nxt_fixed = nxt_fixed[:-2].rstrip()
+    joined = curr_fixed + nxt_fixed + '**'
+else:
+    joined = curr_fixed + nxt_fixed  # lost closing ** — may need manual fix
+```
+
+### Anti-pattern: Complete bold items
+
+Do NOT join when BOTH lines are complete bold blocks (start+end with `**`):
+
+```
+**第一条 ...之道。**    ← DON'T JOIN
+                        ← blank line
+**第二条 ...合一。**    ← DON'T JOIN
+```
+
+**Detection**: Both `curr` and `nxt` start with `**` and end with `**`.
+
+## Pattern B: Conjoined paragraphs (Type 2)
+
+Separate sections merged into one line. Common cases:
+
+### Section headers merged with body
+```
+导言：这本指引怎么用这本指引是什么这是一本修学地图...
+```
+→ Split into:
+```
+导言：这本指引怎么用
+
+这本指引是什么
+
+这是一本修学地图...
+```
+
+### Song titles merged mid-lyrics
+```
+...生生世世不再久违《菩提花开》如果你渴求一滴水...
+```
+→ Split into:
+```
+...生生世世不再久违
+
+### 《菩提花开》
+
+如果你渴求一滴水...
+```
+
+### List items merged into one line
+```
+不在班级群发布...不从事违法活动不在班级平台拉拢...
+```
+→ Split into bullet list:
+```
+- 不在班级群发布...
+- 不从事违法活动
+- 不在班级平台拉拢...
+```
+
+**Approach**: Manual string replacements for known patterns. Regex is unreliable for semantic splits.
+
+## Pattern C: Stray page numbers
+
+Standalone digits at line ends, often from PDF page number artifacts:
+- `42`, `43`, `46`, `47` at end of content lines
+
+**Fix**: Strip trailing digits that aren't part of dates, durations, or course numbers.
+
+## Pattern D: Encoding artifacts
+
+`川` (U+5DDD) replacing curly quotes `"` (U+201C/U+201D):
+```
+把" 道理川变成" 自己的川    →    把"道理"变成"自己的"
+```
+
+**Fix**: Replace `" 道理川` → `"道理"`, `" 自己的川` → `"自己的"`.
+
+## Multi-pass workflow
+
+1. **Pass 1 — Join word fragments**: Scan for lines split by blank line where first line doesn't end with `。！？` and neither line is structural (header/list/table). Skip complete bold items.
+2. **Pass 2 — Split conjoined**: Apply known string replacements for merged sections, song transitions, list items.
+3. **Pass 3 — Clean artifacts**: Fix stray `**` markers, encoding issues, stray page numbers.
+4. **Verify**: `git diff` after each pass; `git checkout` if over-aggressive.
+
+## Rejected heuristics
+
+- **Short-line join** (< 15 chars): Over-joins section headers (`中级和高级（以后的事）`) with body, and Q&A pairs (`正念是什么？\n\n就是...`). Only use for clear mid-word fragments.
+- **Blind `**` stripping**: Removes valid bold formatting from complete bold items.
@@ -0,0 +1,168 @@
+"""
+Fix extraneous line breaks in Chinese markdown files.
+
+Three file patterns:
+1. Fixed-width body text (20-25 chars/line) + vertical TOC -> join lines, remove page nums
+2. Mostly-paragraph with stray breaks + outline TOC -> join broken lines, preserve list items
+3. Already fine -> skip (idempotent)
+
+Usage: python3 normalize_breaks.py <directory>
+"""
+import re
+import sys
+from pathlib import Path
+
+CJK = re.compile(r'[\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff]')
+CN_PUNCT = '，。！？；：、""''（）《》【】…—～·'
+NUM_MARKER = re.compile(r'^[一二三四五六七八九十]+[、，,]')
+DIGIT_MARKER = re.compile(r'^\d+[、.,]')
+TOC_SEP = re.compile(r'\.{3,}')  # "......" separators in outline TOCs
+
+
+def has_cjk(s):
+    return bool(CJK.search(s))
+
+
+def is_page_num(line):
+    s = line.strip()
+    return s and s.isdigit() and len(s) <= 2
+
+
+def is_toc_line(line):
+    """Vertical TOC: single char, or 【, 】, ·, or solo digit"""
+    s = line.strip()
+    if not s:
+        return False
+    if len(s) == 1 and (has_cjk(s) or s in CN_PUNCT or s in '【】·' or s.isdigit()):
+        return True
+    return False
+
+
+def is_section_header(line):
+    """Section headers: 【...】, ## ..., # ..., 一、..., 1、..., or standalone title lines"""
+    s = line.strip()
+    if not s:
+        return False
+    if s.startswith('【') and s.endswith('】'):
+        return True
+    if s.startswith('#'):
+        return True
+    if NUM_MARKER.match(s):
+        return True
+    if DIGIT_MARKER.match(s):
+        return True
+    return False
+
+
+def is_outline_toc_line(line):
+    """Outline/list TOC: entries separated by ...... or short numbered items"""
+    s = line.strip()
+    if TOC_SEP.search(s):
+        return True
+    m = re.match(r'^(\d+[.、,]|[一二三四五六七八九十]+[、,])\s*\S', s)
+    if m and len(s) < 30:
+        return True
+    return False
+
+
+def find_toc_end(lines):
+    """Find where the vertical TOC section ends and body text begins."""
+    for i, line in enumerate(lines):
+        s = line.strip()
+        if has_cjk(s) and len([c for c in s if has_cjk(c)]) >= 3:
+            j = i
+            while j > 0 and not lines[j - 1].strip():
+                j -= 1
+            return j
+    return 0
+
+
+def process_body(lines):
+    """Join body text lines into paragraphs, preserving section headers and outline items."""
+    result = []
+    buf = []
+
+    def flush():
+        nonlocal buf
+        if buf:
+            joined = ''.join(buf)
+            result.append(joined)
+            buf = []
+
+    for line in lines:
+        s = line.strip()
+
+        if not s:
+            flush()
+            result.append('')
+            continue
+
+        if is_section_header(s):
+            flush()
+            result.append(s)
+            continue
+
+        if is_outline_toc_line(s):
+            flush()
+            result.append(s)
+            continue
+
+        if is_page_num(s):
+            continue
+
+        if has_cjk(s) or (buf and s):
+            buf.append(s)
+        else:
+            flush()
+            result.append(s)
+
+    flush()
+    return result
+
+
+def process_file(filepath):
+    content = filepath.read_text(encoding='utf-8')
+    lines = content.split('\n')
+
+    toc_end = find_toc_end(lines)
+
+    if toc_end > 10:
+        toc_part = lines[:toc_end]
+        body_part = lines[toc_end:]
+        body_processed = process_body(body_part)
+        new_lines = toc_part + body_processed
+    else:
+        new_lines = process_body(lines)
+
+    cleaned = []
+    prev_blank = False
+    for line in new_lines:
+        is_blank = line.strip() == ''
+        if is_blank and prev_blank:
+            continue
+        cleaned.append(line)
+        prev_blank = is_blank
+
+    while cleaned and cleaned[-1] == '':
+        cleaned.pop()
+
+    new_content = '\n'.join(cleaned) + '\n'
+
+    if new_content != content:
+        filepath.write_text(new_content, encoding='utf-8')
+        return True
+    return False
+
+
+def main():
+    workdir = Path(sys.argv[1])
+    files = sorted(workdir.glob('*.md'))
+
+    for f in files:
+        changed = process_file(f)
+        status = 'FIXED' if changed else 'OK'
+        print(f'{status}: {f.name}')
+
+
+if __name__ == '__main__':
+    main()