Commit Graph

46 Commits

Author SHA1 Message Date
Andy Lee 7add391b2c chore: build and package 2025-07-24 00:47:46 -07:00
Andy Lee 43155d2811 fix: supress resources leak logs 2025-07-22 19:53:45 -07:00
Andy Lee 8513471573 feat: make diskann runnable 2025-07-22 14:26:03 -07:00
Andy Lee c2f35c8e73 fix: logs 2025-07-21 23:02:13 -07:00
Andy Lee 573313f0b6 refactor: logs 2025-07-21 22:45:24 -07:00
yichuan520030910320 587ce65cf6 Merge branch 'main' of https://github.com/yichuan-w/LEANN 2025-07-21 21:54:27 -07:00
yichuan520030910320 ccf6c8bfd7 fix flush print 2025-07-21 21:54:20 -07:00
Andy Lee c112956d2d fix: mlx 2025-07-21 21:29:15 -07:00
Andy Lee b3970793cf fix: cache the loaded model 2025-07-21 21:20:53 -07:00
yichuan520030910320 530f6e4af5 add progress bar in build 2025-07-21 20:55:18 -07:00
Andy Lee 1b6272ce0e Building, CLI tool & Embedding Server Fixed (#5)
* chore: shorter build time

* chore: update faiss

* fix: no longger do embedding server reuse

* fix: do not reuse emb_server and close it properly

* feat: cli tool

* feat: cli more args

* fix: same embedding logic
2025-07-21 20:17:25 -07:00
Andy Lee 34c71c072d chore: parallel compile fix 2025-07-19 22:51:47 -07:00
Andy Lee 6d2149c503 chore: parallel compile fix 2025-07-19 22:46:24 -07:00
Andy Lee 043b0bf69d chore: parallel compile fix 2025-07-19 22:34:19 -07:00
Andy Lee 9b07e392c6 chore: parallel compile 2025-07-19 22:32:13 -07:00
yichuan520030910320 0796a52df1 change wecaht app split logic 2025-07-19 19:43:30 -07:00
Andy Lee 71c7de9c84 fix: build with direct embedding 2025-07-17 21:49:36 -07:00
Andy Lee 1c5fec5565 perf: make embedder loading faster by 6x, and embed queries through the server 2025-07-17 20:08:06 -07:00
Andy Lee a13c527e39 feat: openai embeddings 2025-07-17 17:02:47 -07:00
yichuan520030910320 51255bdffa update readme and add timer 2025-07-16 17:15:51 -07:00
Andy Lee 2a1a152073 refactor: nits 2025-07-16 15:39:58 -07:00
Andy Lee 7b9406a3ea feat: different search_args and docstrings 2025-07-16 15:25:58 -07:00
Andy Lee 6a1dc895fb feat: disable warmup by default 2025-07-15 22:16:02 -07:00
yichuan520030910320 e5a9ca8787 fix mem compare 2025-07-14 22:55:10 -07:00
Andy Lee baa60b40d1 fix: smaller warmup id 2025-07-14 15:20:45 -07:00
Andy Lee 3da5b44d7f fix: mlx when searching, added to embedding_server 2025-07-14 01:11:21 -07:00
Andy Lee 8b4654921b fix: run faiss in subprocess to prevent kmp 2025-07-14 00:29:21 -07:00
Andy Lee 3b5a185e60 refactor: check if current emb_server has correct passages/embedder 2025-07-13 22:43:51 -07:00
Andy Lee fddc61df5e chore: reset to latest version 2025-07-13 17:06:48 -07:00
yichuan520030910320 0fa8a9191f add wechat history extract app 2025-07-13 16:52:08 -07:00
Andy Lee 8e0ab4a28d chore: update deps 2025-07-12 22:48:13 +00:00
yichuan520030910320 9b8c5041dc update readme 2025-07-12 13:01:11 -07:00
Andy Lee eb6f504789 Datastore reproduce (#3)
* fix: diskann zmq port and passages

* feat: auto discovery of packages and fix passage gen for diskann

* docs: embedding pruning

* refactor: passage structure

* feat: reproducible research datas, rpj_wiki & dpr

* refactor: chat and base searcher

* feat: chat on mps
2025-07-11 23:37:23 -07:00
Andy Lee a6c400cd4f chroe: linux boost and protobuf 2025-07-08 21:25:43 +00:00
Andy Lee f25a1a3840 chore: macos compatible 2025-07-08 13:32:00 -07:00
yichuan520030910320 6497e17671 add gpu chunk embedd and add complexity in hnsw 2025-07-08 18:40:52 +00:00
yichuan520030910320 637dab379e add workaround code 2025-07-07 23:13:47 +00:00
Andy Lee cf17c85607 Make DiskANN and HNSW work on main example (#2)
* fix: diskann zmq port and passages

* feat: auto discovery of packages and fix passage gen for diskann
2025-07-05 22:18:12 -07:00
Andy Lee a38bc0a3fc refactor: embedding server manager 2025-07-06 01:54:46 +00:00
Andy Lee 910927a405 feat: support more embedders 2025-07-06 00:35:07 +00:00
Andy Lee 0aa84e147b feat: hnsw embedding server and csr format 2025-07-05 23:04:41 +00:00
yichuan520030910320 368474d036 fix larger file read and add faq 2025-07-03 23:25:36 +00:00
yichuan520030910320 a627abe794 fix file path bug still compatiable bug in hnsw search 2025-07-03 02:02:42 +00:00
yichuan520030910320 44815ee7fd add configuable funcname 2025-07-02 05:18:00 +00:00
yichuan520030910320 a075fd6f47 Add DiskANN and faiss as submodules 2025-06-30 10:11:39 +00:00
yichuan520030910320 46f6cc100b Initial commit 2025-06-30 09:05:05 +00:00