134 lines
4.2 KiB
Markdown
134 lines
4.2 KiB
Markdown
# Termbase Rebuild (from absorbed termbase-management)
|
|
|
|
How to rebuild the terms SQLite database from source spreadsheets. This is the full pipeline from the now-archived `termbase-management` skill.
|
|
|
|
## Prerequisites
|
|
|
|
```bash
|
|
pip install openpyxl odfpy pyyaml
|
|
```
|
|
|
|
## Step 1: Inspect spreadsheet structure
|
|
|
|
```python
|
|
from openpyxl import load_workbook
|
|
wb = load_workbook(path, read_only=True, data_only=True)
|
|
for sn in wb.sheetnames:
|
|
ws = wb[sn]
|
|
rows = [list(r) for r in ws.iter_rows(min_row=1, max_row=6, values_only=True)]
|
|
print(f"[{sn}] {sum(1 for _ in ws.iter_rows())} rows, cols: {len(rows[0]) if rows else 0}")
|
|
for r in rows[:5]: print(f" {r}")
|
|
```
|
|
|
|
For ODS files, use odfpy:
|
|
```python
|
|
from odf.opendocument import load as odf_load
|
|
from odf.table import Table, TableRow, TableCell
|
|
from odf.text import P
|
|
|
|
doc = odf_load(path)
|
|
for table in doc.getElementsByType(Table):
|
|
for row in table.getElementsByType(TableRow):
|
|
cells = row.getElementsByType(TableCell)
|
|
vals = []
|
|
for cell in cells:
|
|
text = ''
|
|
for p in cell.getElementsByType(P):
|
|
for node in p.childNodes:
|
|
if node.nodeType == node.TEXT_NODE:
|
|
text += node.data
|
|
vals.append(text.strip() if text else None)
|
|
```
|
|
|
|
## Step 2: Convert to CSV + YAML
|
|
|
|
### Cleaning
|
|
- Strip trailing None/empty values from each row: `while row and not row[-1]: row.pop()`
|
|
- Skip entirely empty rows
|
|
- Pad all rows to the max column count
|
|
|
|
### Duplicate header handling
|
|
Some sheets have duplicate column names (e.g., two `英文` columns in paired layout). Deduplicate with suffixes:
|
|
```python
|
|
from collections import Counter
|
|
def dedup_headers(headers):
|
|
seen = Counter()
|
|
result = []
|
|
for h in headers:
|
|
s = str(h) if h else ''
|
|
if s in seen:
|
|
seen[s] += 1
|
|
result.append(f"{s}_{seen[s]}")
|
|
else:
|
|
seen[s] = 1
|
|
result.append(s)
|
|
return result
|
|
```
|
|
|
|
### Output formats
|
|
- **CSV**: `csv.writer` — column-major, preserves all raw data
|
|
- **YAML**: `yaml.dump(data, allow_unicode=True, default_flow_style=False, sort_keys=False, width=200)` — list of dicts
|
|
|
|
## Step 3: Load into SQLite
|
|
|
|
```python
|
|
import sqlite3
|
|
import csv
|
|
|
|
con = sqlite3.connect('termlib.sqlite')
|
|
|
|
# Simple CSVs work with csv.reader:
|
|
with open('file.csv', 'r', encoding='utf-8') as f:
|
|
rows = list(csv.reader(f))
|
|
headers = rows[0]
|
|
data = rows[1:]
|
|
col_defs = ', '.join(f'"{h}" TEXT' for h in headers)
|
|
con.execute(f'CREATE TABLE "table_name" ({col_defs})')
|
|
con.executemany(f'INSERT INTO "table_name" VALUES ({", ".join(["?"] * len(headers))})', data)
|
|
con.commit()
|
|
```
|
|
|
|
### Pitfall: Multiline CSV fields
|
|
CSV files with embedded newlines (common in glossary example-sentence columns) break DuckDB's auto-sniffer. Fall back to Python csv.reader:
|
|
|
|
```python
|
|
import csv
|
|
with open(path, 'r', encoding='utf-8') as f:
|
|
rows = list(csv.reader(f))
|
|
|
|
headers = rows[0]
|
|
data = rows[1:]
|
|
|
|
col_defs = ', '.join(f'"{h}" VARCHAR' for h in cleaned_headers)
|
|
con.execute(f'CREATE TABLE "{table}" ({col_defs})')
|
|
|
|
batch_size = 500
|
|
for i in range(0, len(data), batch_size):
|
|
batch = data[i:i+batch_size]
|
|
placeholders = ', '.join(['(' + ', '.join(['?' for _ in headers]) + ')' for _ in batch])
|
|
flat = [v for row in batch for v in row]
|
|
con.execute(f'INSERT INTO "{table}" VALUES {placeholders}', flat)
|
|
```
|
|
|
|
### SQLite CLI
|
|
Open the database file directly:
|
|
```
|
|
sqlite3 path/to/termlib.sqlite
|
|
```
|
|
|
|
## Step 4: Create unified views
|
|
|
|
See `references/unified-view.sql` for the pattern. Key patterns:
|
|
- `UNION ALL` across all source tables
|
|
- Normalize column names to `zh`, `en`, `loc` (出处), `source`
|
|
- For paired-column sheets (e.g., `中文/英文` + `补充内容/英文_1`), emit two UNION branches
|
|
- Filter out rows where zh or en is NULL/empty
|
|
|
|
## Pitfalls
|
|
|
|
- **ODS reading**: Must traverse `odf.text.P` child elements, not direct text nodes
|
|
- **Duplicate headers**: JSON/YAML dict silently overwrites duplicate keys — always deduplicate
|
|
- **Multiline CSV**: Use Python `csv.reader` for CSVs with embedded newlines
|
|
- **SQLite path**: Always pass the file path to `sqlite3`
|
|
- **`execute_code` sandbox**: Does NOT share pip-installed packages — use `terminal` for Python scripts
|