# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this project is

A Chinese-medicine (中医) knowledge-graph + LLM pipeline, built as a Heima/itheima training track. The numbered top-level directories are the **stages of one pipeline**, meant to be built and run in order. Each stage consumes the previous stage's output from disk — there is no orchestrator, you run the stages by hand.

| Directory | Stage | State |
|---|---|---|
| `_000_demo/` | Teaching demos, one per library the pipeline uses: `requests`, `BeautifulSoup`, `faiss`, `langgraph` | not part of the pipeline, but `04_langgraph_demo.py` is the reference for wiring `_004_` |
| `_001__clawler/` | Crawl 中药大全 / 方剂大全 from `zhongyibaike.com` → CSV lists → per-item `.txt` | **done**: 1472 herbs + 703 formulas on disk |
| `_002__extract_information/` | LLM entity/relation extraction → one json per item, plus Alpaca datasets for `_007_` | formulas **done** 703/703; herbs 1239/1472 |
| `_003_create_neo4j_database/` | Load triples into Neo4j, export `tcm_metadata.json` (schema layer), build the FAISS entity index | **done**: 18895 nodes + 48344 relations; metadata + index on disk |
| `_004_langgraph_more_nodes/` | LangGraph agent over the graph — `AgentState` + one file per node | **in progress**: 7 nodes written and individually tested, graph not wired yet |
| `_005_fastapi/` | HTTP API over the agent — `main.py` (`GET /`, `POST /process`) + `test_api.py` | **done** for those two endpoints; `/process_stream` and `/picture/{filename}` from the spec not written |
| `_006_streamlit/` | Streamlit UI | empty |
| `_007_fine_tune/` | Model fine-tuning | empty |

("clawler" is a typo for crawler, kept as-is. Directory prefixes are inconsistent — `_001__clawler` has a double underscore, others single. Read the real name, don't infer it.)

Requirements arrive as plain-text files named `需求`/`需求1`, written by the user in Chinese, placed inside the stage they describe (or at the repo root). **The user appends to these files as work proceeds** — `_004_langgraph_more_nodes/需求1` has grown one numbered block per node, and the newest block at the bottom is the live task. Check `git diff` on the 需求 file to see what was just added. Earlier stages' 需求 files (`_001_`, `_002_`, `_003_`) are complete and useful as a model for what "done" looks like.

## Architecture

### `common/` — the shared layer

Dependency order is `path_utils` → `config` → `llm`/`neo4j_manager`. Respect it; don't import upward.

- **`path_utils.py`** — `get_file_path(relative_path)` resolves against the repo root, computed from `__file__` rather than cwd. Every file path in this project goes through it; hardcoded absolute paths break the Windows-dev → Linux-deploy handoff this function exists for. It also `print`s the root at import time, so every script's first line of output is a stray path.
- **`config.py`** — `Config` reads all secrets/settings from env vars into instance attributes. Instantiate it (`conf = Config()`); don't call `os.getenv` elsewhere. New settings go here. Note `TCM_METADATA` is the **file contents as a string** (read eagerly in `__init__`, so constructing `Config` touches disk), while `ENTITY_INDEX_PATH` / `ENTITY_ID2TEXT_PATH` are paths.
- **`llm.py`** — module-level singleton `my_llm`, a `ChatOpenAI` pointed at an OpenAI-compatible endpoint via `MODEL_BASE_URL`. Import it rather than constructing new clients.
- **`neo4j_manager.py`** — `Neo4jClient` over the official driver, plus module-level singleton `neo4j_client` built from `Config()`. Pass values through `parameters`, not string interpolation. Five methods, each with a distinct job:
  - `run_cypher(query, parameters=None)` → `list[dict]`, one session per call. This is what stage `_004_` uses.
  - `run_multiple_cypher(queries_with_params)` → one **write** transaction for the whole batch with a tqdm bar; any failure rolls back everything. For bulk import (`_003_`), not for read queries.
  - `validate_cypher(query)` → bool, runs `EXPLAIN <query>` so Neo4j compiles without executing. Syntax + label/property existence checking lives here, not in the node.
  - `get_all_node_names(label=None)` → the entity-name list that feeds the FAISS index.
  - `export_tcm_metadata_to_json(output_path)` → dumps labels / relationship types / triples / property keys. Writes relative to **cwd**, unlike everything else here.
- **`embedding_model.py`** — `EmbeddingModel` wrapping `SentenceTransformer` (bge-large-zh-v1.5, 1024-dim), CUDA when available, plus module-level singleton `embedding_model`. `encode()` forwards `convert_to_numpy` / `normalize_embeddings`; the index was built with `normalize_embeddings=True`, so queries must pass it too. Two traps: the fallback path when `EMBEDDING_MODEL_PATH` is unset is a hardcoded `E:\...` that only exists on this box, and **import has side effects** — it prints the device, the repr, a sample vector and the dimension, so anything importing it emits four stray lines.
- **`ouput_graph_utils.py`** — still an empty placeholder (note the `ouput` typo; nothing references it).
- **`tcm_metadata.json`** — byte-identical copy of `_003_create_neo4j_database/tcm_metadata.json`. `Config` reads the `_003_` one; this copy is unreferenced. Edit the `_003_` file.

Every `common/` module ends with an `if __name__ == '__main__':` self-test. That's the established smoke-test convention here — keep it when adding modules.

### Stage `_001_` — the crawler pattern

Four scripts, two mirrored pairs: `01_herb_list.py` / `03_formula_list.py` write the CSV list, `02_herb_detail.py` / `04_formula_detail.py` consume it and write one `.txt` per item. The `*_detail` pair are near-identical (differing mainly in `方剂名` vs `中药名` column and output dir) — when you change parsing behaviour in one, check whether the other needs the same edit.

Conventions worth preserving, since later stages and re-crawls depend on them:

- **Resume is filesystem-based.** Existence of `中药/<name>.txt` means done; the todo list is built by filtering those out. Detail writes go to `<path>.<thread_id>.part` then `os.replace`, so an interrupted run never leaves a half file that looks complete. Stage `_002_`'s resume should follow the same shape.
- **Concurrency**: `ThreadPoolExecutor`, default 5 workers, `--workers` to change, `--limit N` to try a few first. `requests.Session` is per-thread via `threading.local()`; `print` goes through a lock (`log()`).
- **Parsing is order-driven, not field-driven.** The `h3` subsections differ per herb, so both detail parsers walk `h2/h3/p/li` in document order and open a new section on each heading, rather than looking up known field names. List parsers walk `content.descendants` (not `find_all("a")`) so each link can be attributed to the `h2` category above it. Both skip image `figure`s and `ol.p_alternative_list` as noise.
- CSVs are written `utf-8-sig` (Excel), detail `.txt` plain `utf-8`.

### Stage `_002_` — extraction contract

`__000__extract_graph_data_utils.py` defines the graph schema in Pydantic and exposes one function: `extract_tcm_knowledge(text) -> dict`, a `prompt | my_llm | JsonOutputParser` chain. The schema is the contract stage `_003_` will load and `_004_` will query, so treat it as the source of truth:

- Entities: `Symptom`, `Disease`, `Formula`, `Herb`, `Effect`, `Source`
- Relations: `TREATS_DISEASE`, `ALLEVIATES_SYMPTOM`, `HAS_EFFECT`, `HAS_INGREDIENT`, `HAS_SYMPTOM`, `FROM_SOURCE`
- Attributes hang off entities: `FormulaAttributes` (alias/effect/indication/taboo/usage), `HerbAttributes` (adds dosage/meridian/origin/place/processing/property_flavor/traits)

The prompt deliberately returns `{"entities": [], "relations": []}` when the text describes a single entity with no relation to another — so empty results are expected output, not failure. It also instructs the model not to mix formula and herb attribute sets in one extraction. `parser` is a `JsonOutputParser`, so the return value is a plain dict, not a `TCMKnowledgeGraph` instance — don't assume attribute access.

Four scripts, **two mirrored pairs** matching the crawler layout — `01_`/`02_` for herbs, `03_`/`04_` for formulas. Within a pair: `01_`/`03_` call the LLM and write graph json, `02_`/`04_` rebuild the Alpaca dataset from that json without any LLM calls. The two extractors are near-identical (differing only in input/output dirs and 味/方 wording), as are the two builders — **change one, check whether its mirror needs the same edit.**

The same prompt handles both: `FormulaAttributes` and `HAS_INGREDIENT`/`FROM_SOURCE` were already in the schema. Verified on formulas — every one got a `FROM_SOURCE`, ingredients came through as `HAS_INGREDIENT`, and attributes stayed within `FormulaAttributes` (no herb-only `dosage`/`meridian` leaking in), so the prompt's "don't mix the two attribute sets" instruction holds.

`01_extract_herb_graph.py` is the batch driver over `中药/`. Same resume/concurrency shape as the crawlers (json existence = done, `.part`-then-`os.replace`, `ThreadPoolExecutor`), plus what the LLM step needs on top:

- **Cost is the constraint, not politeness.** Measured 20s–4min per call, driven by *output* size (a herb with 87 entities is slow). Herbs: 1472 files ≈ 90h serially, ≈9–15h at the default 10 workers. Current progress: 1239 of 1472 herbs (233 left), formulas complete at 703/703. `中药图谱/_failed.log` has 2503 lines over 672 distinct names — it's append-only across runs, so most of those have since succeeded; the todo list is derived from json existence, not from this log. Formulas are much cheaper — 703 files averaging 377 chars vs the herbs' 1537, roughly 2–3h. There's no inter-call sleep; unlike the crawlers the LLM endpoint is the bottleneck, not an IP-ban risk.
- **`--limit` defaults to 0 (no limit), `--workers` to 10**, in both extractors and in the `run()` signatures. A bare invocation is a *full* run — use `--limit 5` to trial.
- **Failures don't write json**, so they stay in the todo list and a plain re-run retries them. They're also appended to `中药图谱/_failed.log` (`名字<TAB>原因`), which `--retry-failed` reads to narrow a run to just those.
- Empty extractions *are* written, otherwise every re-run would re-pay for the same no-relation herbs.
- Per-call retry is 2 attempts with 5s/10s backoff; `extract_one` returns `(name, ok, note, n_entity, n_relation)`.

### Alpaca fine-tuning dataset (feeds stage `_007_`)

Same extraction, second output. `ALPACA_INSTRUCTION` and `build_alpaca_record()` live in `__000__extract_graph_data_utils.py` — shared constants have to go in a normally-named module because `01_`/`02_` can't import each other. `instruction` is fixed, `input` is the herb `.txt` verbatim, `output` is `json.dumps({"entities", "relations"})` — a **string**, per the Alpaca convention, not a nested object.

Two write paths, deliberately:

- The extractors (`01_`/`03_`) append one JSONL line per item as they go (the "顺便" part). JSONL because concurrent appends of whole lines don't interleave destructively, and a crash costs at most the last line. The append happens *after* the graph json is renamed into place — reverse that order and a crash between the two would let resume re-extract and duplicate the row.
- The builders (`02_`/`04_`) rebuild the whole dataset from the graph json with no LLM calls. These exist because the append path only covers items extracted *in that run* — resume-skipped and pre-existing items would never get a row. They're idempotent (verified: identical md5 across two runs), so they're what to trust for the real dataset. `--json` for a json array instead of JSONL, `--skip-empty` to drop no-relation samples (kept by default — they teach the model not to invent relations).

`load_source_text()` prefers the graph json's `source_file` and falls back to `<dir>/<name>.txt`, since early extractions may predate that field.

### Stage `_003_` — Neo4j load

`01_import_graph_to_neo4j.py` reads both 图谱 dirs, builds Cypher in Python, writes through `Neo4jClient.run_multiple_cypher`. One script, not a mirrored pair — herbs and formulas share a schema and *have to* land in one graph, since a formula's `HAS_INGREDIENT` targets the same `Herb` nodes the herb files describe. Verified: 一贯煎's six ingredients resolve to `Herb` nodes carrying `property_flavor` from their own herb file.

- **Aggregate in memory first, then write.** 49226 entity records collapse to 18895 nodes and 50627 relation records to 48344 edges — 咳嗽 appears in hundreds of files. Per-record MERGE would redo the same work hundreds of times.
- **The node key is `(label, name)`, not `name`.** 356 names carry two types (热咳 is both `Disease` and `Symptom`); deduping on name alone would fuse them.
- **Labels, relation types and attribute keys are string-concatenated; values are always `$` parameters.** Cypher can't parameterize the former, so all three go through the `ENTITY_TYPES` / `RELATION_TYPES` / `ATTR_KEYS` whitelists (mirrors of the `_002_` schema) before hitting the f-string.
- Attributes use `MERGE (n:L {name: $name}) SET n += $attrs` — attrs inside the MERGE pattern would make a changed attribute miss and create a duplicate; `ON CREATE SET` would never backfill an existing node. So re-running after extracting more herbs upgrades the attribute-less nodes in place (1038 `Herb` nodes still lack `effect`, matching the 1240/1472 extraction progress).
- A `WITH` sits between the two relation MATCHes. `MATCH (s), (o)` is two disconnected patterns, so the driver logs a cartesian-product warning per statement — 48k of them buries the output. Both sides are unique-constraint equality lookups, so the product is 1×1; the rewrite only silences the warning.
- Unique constraints on `name` per label are created first (`IF NOT EXISTS`), otherwise every MERGE is a label scan.
- Batched at 500 per transaction. Batches aren't atomic with each other, which is fine — everything is MERGE, so an interrupted run is fixed by re-running.
- Relation endpoints missing from their file's own `entities` list (177 cases) get MERGEd anyway, or the MATCH would silently drop the edge.
- Two malformed relations in the corpus (`TREATS` in 枇杷叶.json, `TREATS_DISE` in 重楼.json, both with a null object) fail the whitelist and are counted as dropped.

```bash
python -X utf8 -m _003_create_neo4j_database.01_import_graph_to_neo4j --limit 5 --dry-run  # count + sample Cypher, no connection
python -X utf8 -m _003_create_neo4j_database.01_import_graph_to_neo4j --limit 5            # 5 per dir, real write
python -X utf8 -m _003_create_neo4j_database.01_import_graph_to_neo4j                      # full, ~1-2 min
python -X utf8 -m _003_create_neo4j_database.01_import_graph_to_neo4j --only formula
python -X utf8 -m _003_create_neo4j_database.01_import_graph_to_neo4j --wipe                # DETACH DELETE the six labels first
```

`--wipe` deletes only the six schema labels, leaving anything else in the database alone. The run ends by reading counts back per label and relation type rather than trusting the statement count.

### Stage `_003_` — the derived artifacts stage `_004_` reads

Both are generated *from the loaded graph*, so **re-importing data without regenerating them leaves them stale**. `__004__test_faiss_index.py`'s last check exists precisely to catch that.

- **`tcm_metadata.json`** (14.8 KB) — the schema layer, exported via `Neo4jClient.export_tcm_metadata_to_json()`. 6 labels, 6 relationship types, 14 triples, properties per label (`Herb` has 13, `Formula` 6, the other four only `name`). Every `description` field starts empty; the triples' descriptions have since been filled in by hand with row counts and usage notes, and that prose goes into the Cypher-generation prompt — **keep writing descriptions when the schema changes**, they're doing real work. The export method writes to cwd, so run it from `_003_/` or pass an absolute path.
- **`nero4j_embedding_faiss.index`** (76 MB) + **`nero4j_embedding_faiss_id2text.pkl`** — built by `__003__faiss_embedding.py` from `get_all_node_names()` (no label argument, so **all six labels are mixed into one flat index**; consumers must filter by label themselves). `IndexFlatL2` over unit-length vectors, so the returned squared distance converts as `cos = 1 - dist/2`. The pkl is the `id -> name` map, since FAISS stores no metadata — index and pkl must be regenerated as a pair or lookups silently return the wrong names. Note the `nero4j` typo in both filenames, and that FAISS is sensitive to non-ASCII paths, which is why these live in an ASCII-named directory. `__003__faiss_embedding.py` **runs its work at import time** with no `main` guard — importing it rebuilds the index.

`__004__test_faiss_index.py` is the health check, seven graded checks with a pass/fail summary and a nonzero exit on failure: files load, vector count == mapping count with contiguous ids, index dim == current model dim, self-retrieval (a known name must come back as top-1 at distance ≈0 — this is the check that actually proves vectors and mapping are aligned), normalized-vs-unnormalized query comparison, semantic recall on colloquial words not in the index, and reconciliation against Neo4j.

### Stage `_004_` — the LangGraph agent

Departs from the earlier stages' layout: **one file per node** under `nodes/`, no numeric prefixes, so nodes *can* import each other and be imported by tests. `agent_state.py` holds `AgentState` (a `TypedDict`) and is the contract every node reads and writes — **read it first**, it lists the whole pipeline in field order.

The shape every node follows:

- Signature is `def x_node(state: AgentState) -> AgentState`, mutating and returning the same dict. Nodes get called directly with hand-built dicts holding only their own input keys, so prefer `state.get(key) or []` over `state[key]` (most nodes do; `check_cypher_node` uses subscript access and will `KeyError` on a state without `cypher_query`).
- **Write every output key on every path, including early returns.** Empty input short-circuits before the LLM call, but still fills all six `user_input_*` keys, so downstream nodes never have to check for absence.
- **Skip the LLM when the answer is already determined.** Empty input, or no matched entity at all in the Cypher generator — both return early rather than paying for a call whose result can't be useful.
- A `if __name__ == "__main__":` block runs the node over a handful of hand-written cases including the empty/negative one. Same convention as `common/`. It's the only test mechanism in the repo.

Node by node (execution order, matching `agent_state.py`):

| Node | Reads | Writes |
|---|---|---|
| `zhongyi_intent_node` | `input` | `is_zhongyi_intent` |
| `extract_entity_from_user_input_node` | `input` | `user_input_*` ×6 |
| `match_entity_from_neo4j_node` | `user_input_*` | `matched_*` ×6 |
| `generate_neo4j_cypher_node` | `input`, `matched_*`, `Config().TCM_METADATA` | `cypher_query` |
| `check_cypher_node` | `cypher_query` | `is_all_validate_cypher` |
| `run_cypher_node` | `cypher_query` | `cypher_results` |
| `llm_direct_out_node` | `input` | `direct_out`, `output` |

`llm_direct_out_node` is the **other branch**, not a step in the chain: when `is_zhongyi_intent` is False the graph skips straight to it and ends. It writes the same answer to both `direct_out` and `output` — `output` is the graph's single exit, so `_005_` can read one key without caring whether the answer came from the graph or from the model directly.

Still to write: a node turning `cypher_results` into `neo4j_answer` (and on to `output`), and `build_langgraph()` wiring it all up. `_000_demo/04_langgraph_demo.py` is the reference for that last part — `StateGraph` + `add_conditional_edges` with a router function returning a node name, then `.compile()`. Chinese node names as graph keys, as in the demo.

The six entity types are the **same six** as the `_002_` schema and the Neo4j labels, and the same six recur as `user_input_*` / `matched_*` / prompt sections / `TYPE_FIELDS` / `MATCHED_FIELDS` dicts. Adding a seventh type means touching all of them.

Two things that are easy to get wrong and are load-bearing:

- **`matched_*`, never the user's own words, in `WHERE`.** The user says 拉肚子, the graph stores 腹泻. `match_entity_from_neo4j_node` exists solely to bridge that, and the Cypher prompt explicitly forbids using words from the question. Similarly the extractor is told not to rewrite or add suffixes (人参 must not become 人参片) — anything it invents can't match.
- **The FAISS index is unlabeled, so recall must be filtered by label.** `match_entity_from_neo4j_node` pulls each label's names from Neo4j (`get_all_node_names(label)`, cached per process) and drops hits outside the label being matched — otherwise 咳嗽 can return a herb name and land in `matched_symptoms`, generating meaningless Cypher. It over-fetches `top_k * 6` so filtering still leaves `top_k`. If Neo4j is unreachable it warns and degrades to unfiltered rather than failing. FAISS itself is lazy-loaded and cached at module level (the index is 76 MB — don't load it at import). `TOP_K = 5` matches the 需求; `SIMILARITY_THRESHOLD` is 0.80 while the 需求 and the module docstring both say 70% — tightened deliberately during testing, so don't "fix" it back without checking recall.

Generated Cypher goes straight to the live database, so it's gated in two independent places, and the prompt's "严禁 CREATE/MERGE/..." instruction is not one of them:

1. `generate_neo4j_cypher_node._is_readonly()` — a `FORBIDDEN_KEYWORDS` word-boundary scan plus "has RETURN and starts with MATCH/OPTIONAL MATCH/WITH". Statements failing it are dropped with a warning, capped at `MAX_CYPHER = 5`.
2. `check_cypher_node` — `Neo4jClient.validate_cypher()`, i.e. `EXPLAIN` per statement, which catches syntax errors and nonexistent labels/properties without executing. Sets `is_all_validate_cypher` False on the first failure. `check_cypher_node_bak.py` is a superseded alternative that reimplemented the checks in Python by regexing against `tcm_metadata.json`; the current version delegates to `validate_cypher` instead, letting Neo4j be the authority. Nothing imports the `_bak` file.

`generate_neo4j_cypher_node` hand-rolls its JSON parsing instead of using `JsonOutputParser`: it strips code fences, retries by regexing out the first `[...]`, unwraps a `{"cypher_query": [...]}` envelope, and finally falls back to scraping `MATCH`-prefixed lines — a parse failure must degrade, not break the chain. `run_cypher_node` runs statements **one at a time** via `run_cypher` (not `run_multiple_cypher`, whose all-or-nothing write transaction would let one bad read query discard the others) and appends an empty list for any that raise, keeping `cypher_results` index-aligned with `cypher_query`.

### Stage `_005_` — the HTTP API

`main.py` is deliberately thin: validate, run the graph, return `state["output"]`. Two endpoints from `API_INTERFACE.md` — `GET /` (health) and `POST /process` (`{input, user_id}` → `{input, output}`). Three things it does that are load-bearing:

- **The graph is a lazily-loaded module-level singleton** (`get_graph()`, with the import *inside* the function). Importing `langgraph_more_nodes` pulls in the 76 MB FAISS index and the embedding model — ~50s. `lifespan` warms it on startup so the first request doesn't pay for it; a warmup failure only logs, so the service still starts and health check still answers.
- **`graph.invoke` goes through `run_in_threadpool`.** It's synchronous and takes 5–30s per call; awaiting it on the event loop would serialize the whole service. `test_api.py`'s concurrency case exists to catch a regression here.
- **`user_id` is validated but not passed to the graph.** `AgentState` has no such field and there's no session store yet — the spec's "对话历史/记忆" is not implemented. Errors collapse to `500 {"input": "", "output": "系统出错了，请重试！"}` with the traceback server-side only.

`test_api.py` follows the repo's `__main__` convention (no pytest) with a `_check` counter and nonzero exit. Default mode swaps `main._graph` for a `FakeGraph`, so the 27 checks touch neither LLM, Neo4j, nor FAISS and run in under two seconds — that's the regression suite. `--live` (15 checks) needs a server already running and does spend money. Verified: 27/27 offline, 15/15 live including the graph path (四君子汤 → 人参、白术、茯苓、炙甘草) and the fallback path.

`nodes/元数据裁剪方案.md` is a **design document, not implemented behaviour** — a five-layer plan for trimming the schema fed to the Cypher prompt once it outgrows the current 14.8 KB. Its own recommendation for the present size is to do nothing. Read it before "optimizing" the prompt; its conclusion is that the first win is deriving a subgraph from `matched_*`, and its warning is that dropping triples while keeping labels makes the model reverse arrow directions, producing valid Cypher that always returns nothing.

Output files: `alpaca_herb_dataset.jsonl` and `alpaca_formula_dataset.jsonl`. **`instruction` is identical across both**, so `cat`-ing the two JSONL files is a valid combined training set — verified no duplicate `input` across them.

Verified end to end on both paths: graph json count matches dataset line count exactly, every `output` parses back to `{entities, relations}`, extracting N more items appends exactly N lines.

## Environment

- Python 3.10 (conda env `ctm_kg`, registered outside the repo tree). Windows / Git Bash; use forward slashes.
- Black is the configured formatter (88 chars, double quotes). Existing code is inconsistent about quote style — match Black for new code.

### Env vars

`common/.env` is the real file. `Config` expects:

```
MODEL_API_KEY  MODEL_BASE_URL  MODEL_NAME     # OpenAI-compatible LLM endpoint
NEO4J_URI  NEO4J_USER  NEO4J_PASSWORD          # graph database
JIMENG_AK  JIMENG_SK                           # 极梦 image generation
EMBEDDING_MODEL_PATH                           # local embedding model (stage _004_)
```

## Commands

Run everything with `python -m` **from the repo root**. Imports are absolute (`from common.config import Config`), so `python common/llm.py` fails with `ModuleNotFoundError` — the script's own directory lands on `sys.path` instead of the root.

```bash
python -m common.config           # print the loaded API key
python -m common.llm              # smoke-test the LLM connection
python -m common.neo4j_manager    # smoke-test Neo4j

# stage _001_ (already run; re-run is incremental thanks to resume)
python -X utf8 -m _001__clawler.01_herb_list                        # → herb_list.csv
python -X utf8 -m _001__clawler.02_herb_detail --limit 5            # try 5 first
python -X utf8 -m _001__clawler.02_herb_detail --workers 10         # full run
python -X utf8 -m _001__clawler.03_formula_list                     # → formula_list.csv
python -X utf8 -m _001__clawler.04_formula_detail

python -X utf8 -m _002__extract_information.__000__extract_graph_data_utils   # one-herb extraction demo

# stage _002_ batch driver (resumable; re-running only does what's missing)
python -X utf8 -m _002__extract_information.01_extract_herb_graph --limit 5   # try 5 first
python -X utf8 -m _002__extract_information.01_extract_herb_graph             # full run, 5 workers
python -X utf8 -m _002__extract_information.01_extract_herb_graph --workers 10
python -X utf8 -m _002__extract_information.01_extract_herb_graph --retry-failed

# formulas — same flags, mirror scripts (703 items, ~2-3h full run)
python -X utf8 -m _002__extract_information.03_extract_formula_graph --limit 5
python -X utf8 -m _002__extract_information.03_extract_formula_graph

# rebuild the Alpaca datasets from the extracted json (no LLM calls, idempotent)
python -X utf8 -m _002__extract_information.02_build_alpaca_dataset             # herbs
python -X utf8 -m _002__extract_information.04_build_formula_alpaca_dataset     # formulas
python -X utf8 -m _002__extract_information.02_build_alpaca_dataset --json --skip-empty

# stage _003_ derived artifacts — regenerate after re-importing the graph
python -X utf8 -c "from common.neo4j_manager import neo4j_client; \
  neo4j_client.export_tcm_metadata_to_json('_003_create_neo4j_database/tcm_metadata.json')"
python -X utf8 -m _003_create_neo4j_database.__003__faiss_embedding      # rebuilds index + pkl, ~76 MB
python -X utf8 -m _003_create_neo4j_database.__004__test_faiss_index     # 7 checks, exit 1 on failure
python -X utf8 -m _003_create_neo4j_database.__004__test_faiss_index --skip-neo4j
python -X utf8 -m _003_create_neo4j_database.__004__test_faiss_index --query 拉肚子 失眠 --top-k 10

# stage _004_ — each node's self-test, run standalone (needs LLM, Neo4j and the embedding model)
python -X utf8 -m _004_langgraph_more_nodes.nodes.zhongyi_intent_node
python -X utf8 -m _004_langgraph_more_nodes.nodes.extract_entity_from_user_input_node
python -X utf8 -m _004_langgraph_more_nodes.nodes.match_entity_from_neo4j_node
python -X utf8 -m _004_langgraph_more_nodes.nodes.generate_neo4j_cypher_node
python -X utf8 -m _004_langgraph_more_nodes.nodes.check_cypher_node
python -X utf8 -m _004_langgraph_more_nodes.nodes.run_cypher_node
python -X utf8 -m _004_langgraph_more_nodes.nodes.llm_direct_out_node

# stage _005_ — the HTTP API
python -X utf8 -m _005_fastapi.main                  # serve on 0.0.0.0:8000, ~50s warmup
python -X utf8 -m _005_fastapi.test_api              # 27 offline checks, no LLM/Neo4j/FAISS, <2s
python -X utf8 -m _005_fastapi.test_api --live       # 15 checks against a running server
python -X utf8 -m _005_fastapi.test_api --live --base-url http://127.0.0.1:8001
```

A full stage `_002_` run takes hours — start it in the background and poll, don't block on it.

`-X utf8` matters: this box's console encoding is GBK, and Chinese log output is mojibake without it. Anything that prints 中文 needs it.

Stage `_004_` node output is buried in noise from imports: `path_utils` prints the repo root, `embedding_model` prints the device and a sample vector, and `sentence_transformers` logs weight loading. Filtering with `grep -vE "Loading weights|^<common|^\[tensor|device=|^cuda$|^1024$|^D:"` is the established way to read it.

`requirements.txt` lists unpinned `>=` ranges, several of which are aspirational (`playwright`, `pandas`, `openpyxl`, `volcengine` aren't imported anywhere yet) and the tail has duplicate `python-dotenv` / `langchain_core` entries. Actually in use: `requests`, `beautifulsoup4`, `python-dotenv`, `langchain-core`, `langchain-openai`, `pydantic`, `neo4j`, `tqdm`, `langgraph`, `faiss` (installed as `faiss` 1.14.2, not `faiss-cpu`), `sentence-transformers` 5.5.1, `torch` 2.12+cu130. Pin real versions and drop the duplicates when you next touch dependency setup.

There is no test framework, linter config, or build step — the `__main__` self-test in each module is it, and it's a convention worth keeping when you add modules. If you add pytest, record the commands here.

## Known traps

- **`.env` path (fixed 2026-08-02).** `config.py` used to call `load_dotenv(get_file_path(".env"))`, resolving to a nonexistent `<root>/.env`, which left `MODEL_BASE_URL`, `MODEL_NAME` and all three `NEO4J_*` as `None` — `common.llm` couldn't even be imported from the repo root (`ChatOpenAI` rejects `model=None`). Now loads `common/.env` and all 8 settings populate. Note `MODEL_API_KEY` is still commented out *in the file* and comes from a machine-level shell env var on this dev box, so it will be missing on a fresh machine; the endpoint is DeepSeek.
- **Numeric-prefixed module names can be run but never imported.** `01_herb_list`, `02_herb_detail`, `01_extract_herb_graph` etc. aren't valid Python identifiers, so `python -m` works but `from ... import` is a `SyntaxError`. To reach into one from a test or REPL, use `importlib.import_module("_002__extract_information.01_extract_herb_graph")`. Anything meant to be reused across stages belongs in `common/` or a normally-named module.
- **Path drift in `_005_`'s spec.** `API_INTERFACE.md` says `cd __005_fastapi && python main.py`, and names `start_server.bat`, `quick_test.py`, `full_workflow.py`, `extract_entity_node.py`. The real directory is `_005_fastapi`, `main.py` must be run as `python -m _005_fastapi.main` from the repo root (its imports are absolute), and the `_004_` filenames are different (`langgraph_more_nodes.py`, `extract_entity_from_user_input_node.py`). Its FAISS-path snippets point at a nonexistent `E:\ctm_index\`. Treat the doc as a design target written ahead of the code — the request/response contracts in it are accurate, the paths and filenames aren't.
- **Importing some modules does work.** `common/embedding_model.py` loads the model and prints four lines of diagnostics at import; `_003_create_neo4j_database/__003__faiss_embedding.py` has no `main` guard at all, so importing it rebuilds the 76 MB index. Both are called from `__main__` blocks elsewhere as if they were inert.
- **Two `tcm_metadata.json` copies exist** (`common/` and `_003_create_neo4j_database/`), currently byte-identical. Only the `_003_` one is read.

## Repo hygiene

- **`common/.env` is currently staged and holds live credentials.** Root `.gitignore` is still empty, so a `git add .` picked it up. It is *not yet in any commit* (`git log -- common/.env` is empty), so `git rm --cached common/.env` still fixes it cleanly — once it's committed, only a history rewrite will. Do that, plus populate `.gitignore` (`__pycache__/`, `*.pyc`, `.venv/`, `.env`), before the next commit. Raise it with the user rather than silently unstaging.
- **22 `.pyc` files are staged** across six `__pycache__/` directories, a mix of `cpython-310` (the project interpreter) and stale `cpython-313`. Same cause, same fix.
- The 2175 crawled `.txt` files and both CSVs **are** tracked (they're the bulk of the 2205 tracked files). That's the established choice; don't "clean it up" without asking. The 76 MB FAISS index and the pkl are tracked too.
- `.idea/` is tracked deliberately (PyCharm's own `.idea/.gitignore` already excludes `workspace.xml` and `shelf/`).
- `README.md` and `cookie` are empty placeholders; `picture/` is an empty directory (`_005_`'s spec plans to serve it at `GET /picture/{filename}`); `main.py` is the untouched PyCharm sample.
- `.claude/settings.local.json` accumulates one allow-entry per command run. It's a useful record of what has actually been executed in this repo.

**Keep this file current.** As stages `_004_`–`_007_` get filled in, replace their table rows with the real entry points and add the commands above.