Migrate from duckdb to sqlite

skills/mpi-terms-search/SKILL.md

@@ -6,7 +6,7 @@ category: research
 
 # Terms Search
 
-Database: `toolkit/terms-database/termlib.duckdb`
+Database: `toolkit/terms-database/termlib.sqlite` (SQLite)
 CLI: `toolkit/terms-database/search.py`
 Server: `toolkit/terms-database/server.py`
 
@@ -61,20 +61,19 @@ Errors return `{"error": "..."}` with HTTP 500 (API) or shown inline (UI).
 | 禅意项目 | 14+11 | Zen program terms |
 | 公案 | 8 | Chan koans |
 
-## Direct DuckDB
+## Direct SQLite
 
 ```
-duckdb toolkit/terms-database/termlib.duckdb
+sqlite3 toolkit/terms-database/termlib.sqlite
 ```
 
-Key tables: `unified_terms_flat` (zh, en, loc, source), individual source tables, `unified_terms` view.
+Key table: `terms` (zh, en, loc, source).
 
 ## Rebuilding
 
 Terms data comes from `guide/03 术语库/`. To rebuild:
 1. Convert source xlsx/ods → CSV+YAML in `_output/`
-2. Rebuild DuckDB from CSVs
-3. Materialize `unified_terms_flat` view → table for performance
+2. Load CSVs into SQLite as the `terms` table (zh, en, loc, source)
 
 **Full rebuild pipeline:** See `references/termbase-rebuild.md` (absorbed from the `termbase-management` skill).
 

skills/mpi-terms-search/references/termbase-rebuild.md

@@ -1,11 +1,11 @@
 # Termbase Rebuild (from absorbed termbase-management)
 
-How to rebuild the terms DuckDB from source spreadsheets. This is the full pipeline from the now-archived `termbase-management` skill.
+How to rebuild the terms SQLite database from source spreadsheets. This is the full pipeline from the now-archived `termbase-management` skill.
 
 ## Prerequisites
 
 ```bash
-pip install openpyxl odfpy pyyaml duckdb
+pip install openpyxl odfpy pyyaml
 ```
 
 ## Step 1: Inspect spreadsheet structure
@@ -69,17 +69,23 @@ def dedup_headers(headers):
 - **CSV**: `csv.writer` — column-major, preserves all raw data
 - **YAML**: `yaml.dump(data, allow_unicode=True, default_flow_style=False, sort_keys=False, width=200)` — list of dicts
 
-## Step 3: Load into DuckDB
+## Step 3: Load into SQLite
 
 ```python
-import duckdb
-con = duckdb.connect('termlib.duckdb')
+import sqlite3
+import csv
 
-# Simple CSVs work with auto-detect:
-con.execute("""
-    CREATE TABLE table_name AS 
-    SELECT * FROM read_csv_auto('file.csv', header=true, all_varchar=true)
-""")
+con = sqlite3.connect('termlib.sqlite')
+
+# Simple CSVs work with csv.reader:
+with open('file.csv', 'r', encoding='utf-8') as f:
+    rows = list(csv.reader(f))
+headers = rows[0]
+data = rows[1:]
+col_defs = ', '.join(f'"{h}" TEXT' for h in headers)
+con.execute(f'CREATE TABLE "table_name" ({col_defs})')
+con.executemany(f'INSERT INTO "table_name" VALUES ({", ".join(["?"] * len(headers))})', data)
+con.commit()
 ```
 
 ### Pitfall: Multiline CSV fields
@@ -104,10 +110,10 @@ for i in range(0, len(data), batch_size):
     con.execute(f'INSERT INTO "{table}" VALUES {placeholders}', flat)
 ```
 
-### Pitfall: DuckDB CLI opens in-memory by default
-Running plain `duckdb` gives an empty database. Always pass the file path:
+### SQLite CLI
+Open the database file directly:
 ```
-duckdb path/to/termlib.duckdb
+sqlite3 path/to/termlib.sqlite
 ```
 
 ## Step 4: Create unified views
@@ -122,6 +128,6 @@ See `references/unified-view.sql` for the pattern. Key patterns:
 
 - **ODS reading**: Must traverse `odf.text.P` child elements, not direct text nodes
 - **Duplicate headers**: JSON/YAML dict silently overwrites duplicate keys — always deduplicate
-- **Multiline CSV + DuckDB**: `read_csv_auto` fails on CSVs with quoted newlines — use Python csv.reader
-- **DuckDB path**: Always explicit file path; `duckdb` alone is in-memory
+- **Multiline CSV**: Use Python `csv.reader` for CSVs with embedded newlines
+- **SQLite path**: Always pass the file path to `sqlite3`
 - **`execute_code` sandbox**: Does NOT share pip-installed packages — use `terminal` for Python scripts