docs: results

This commit is contained in:
Andy Lee
2025-08-16 16:48:01 -07:00
parent baf2d76e0e
commit ed27ea6990
2 changed files with 32 additions and 60 deletions

View File

@@ -64,14 +64,27 @@ LLM-based answer evaluation using GPT-4o:
- Considers fractions, percentages, and decimal equivalents
- Evaluates semantic meaning rather than exact text match
## Expected Results
## Benchmark Results
Previous runs show:
- **Question Coverage**: ~65-75% (questions with relevant docs retrieved)
### LEANN-RAG Performance (sentence-transformers/all-mpnet-base-v2)
**Retrieval Metrics:**
- **Question Coverage**: 100.0% (all questions retrieve relevant docs)
- **Exact Match Rate**: 0.7% (substring overlap with evidence)
- **Number Match Rate**: 120.7% (key financial figures matched)*
- **Semantic Match Rate**: 4.7% (word overlap ≥20%)
- **Average Search Time**: 0.097s
**QA Metrics:**
- **Accuracy**: 42.7% (LLM-evaluated answer correctness)
- **Average QA Time**: 4.71s (end-to-end response time)
**System Performance:**
- **Index Size**: 53,985 chunks from 368 PDFs
- **Search Time**: ~0.1-0.2s per query
- **Build Time**: ~5-10 minutes with sentence-transformers/all-mpnet-base-v2
*Note: Number match rate >100% indicates multiple retrieved documents contain the same financial figures, which is expected behavior for financial data appearing across multiple document sections.
## Options
```bash