LEANN

You've already forked LEANN

Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: TBNilles:revert-161-feat/multi-vector-timing-and-dataset-improvements

TBNilles:main TBNilles:fix/drop-python39-support TBNilles:add-gh-pat-token-to-workflows TBNilles:feature/add-ty-type-checker TBNilles:feature/custom-folder-multi-vector TBNilles:feature/colqwen-integration TBNilles:feat/add-colqwen2.5-support TBNilles:revert-161-feat/multi-vector-timing-and-dataset-improvements TBNilles:feat/multi-vector-timing-and-dataset-improvements TBNilles:fix/pdf-duplicate-processing-175-clean TBNilles:fix/pdf-duplicate-processing-175 TBNilles:embed-launch TBNilles:issue-159-performance-analysis TBNilles:fix/security-eval-slack-mcp-reader-clean TBNilles:fix/security-eval-slack-mcp-reader TBNilles:fix/chunking-token-limit-behavior TBNilles:fix/chunking-token-limit-behavior-clean TBNilles:feature/optimize-ollama-batching TBNilles:gen-time TBNilles:fix/twitter-bookmarks-anchor-link TBNilles:fix-readme TBNilles:security/enhance-hf-model-loading-security TBNilles:feature/mcp-integration-clean TBNilles:feature/mcp-integration-support TBNilles:feature/imessage-rag-support TBNilles:feature/chatgpt-rag-support TBNilles:fix-update TBNilles:feature/claude-rag-support TBNilles:financebench TBNilles:fix/passage-server-signature TBNilles:colqwen TBNilles:fix/ask-cli-example TBNilles:feat/configurable-local-endpoints TBNilles:dynamic-add TBNilles:feat/add-github-templates TBNilles:fix/astchunk-uv-import TBNilles:ast-fork TBNilles:fix-arm64-linux TBNilles:cli_fix TBNilles:vldb-exp TBNilles:chore/ruff-imports-hnsw TBNilles:arch-eval TBNilles:mcp-fix TBNilles:fix-arch-ci TBNilles:fix/update-diskann-mkl-linking TBNilles:feature/auto-register-app-indexes TBNilles:feature/enhanced-cli-list-remove TBNilles:feature/add-gemini-api-support TBNilles:fix/52-include-hidden-flag TBNilles:fix/empty-chunks-cherry-pick TBNilles:docs/update-readmes-mcp-conclusion TBNilles:feature/skypilot-docs TBNilles:debug/clean-state-investigation TBNilles:clean-state-investigation TBNilles:add-macos15-support TBNilles:feat/multiple-input-formats TBNilles:fix-ollama-embedding-dimension-bug TBNilles:fix-mac-intel-build TBNilles:feature/graph-partition-support TBNilles:fix/claude-code-gitignore-support TBNilles:fix/clean-hang-solution TBNilles:feat/claude-code-refine TBNilles:feature/claude-code-research TBNilles:docs/update-contributors TBNilles:config TBNilles:refactor-app TBNilles:feat/diskann-smart-memory TBNilles:fix-macos-abi TBNilles:fix/openai-embeddings-cosine-distance TBNilles:fix/manylinux-compatibility TBNilles:apps TBNilles:perf-build TBNilles:readme-polish TBNilles:datastore-reproduce TBNilles:fix-all TBNilles:debug_diskann_disable_pipe TBNilles:master

compare: TBNilles:fix/pdf-duplicate-processing-175-clean

Branches Tags

1 Commits

revert-161 ... fix/pdf-du

Author	SHA1	Message	Date
aakash	877fbe81f4	Fix: Prevent duplicate PDF processing when using --file-types .pdf Fixes #175 Problem: When --file-types .pdf is specified, PDFs were being processed twice: 1. Separately with PyMuPDF/pdfplumber extractors 2. Again in the 'other file types' section via SimpleDirectoryReader This caused duplicate processing and potential conflicts. Solution: - Exclude .pdf from other_file_extensions when PDFs are already processed separately - Only load other file types if there are extensions to process - Prevents duplicate PDF processing Changes: - Added logic to filter out .pdf from code_extensions when loading other file types if PDFs were processed separately - Updated SimpleDirectoryReader to use filtered extensions - Added check to skip loading if no other extensions to process	2025-11-30 11:18:57 -08:00

Author

SHA1

Message

Date

aakash

877fbe81f4

Fix: Prevent duplicate PDF processing when using --file-types .pdf

Fixes #175

Problem:
When --file-types .pdf is specified, PDFs were being processed twice:
1. Separately with PyMuPDF/pdfplumber extractors
2. Again in the 'other file types' section via SimpleDirectoryReader

This caused duplicate processing and potential conflicts.

Solution:
- Exclude .pdf from other_file_extensions when PDFs are already
  processed separately
- Only load other file types if there are extensions to process
- Prevents duplicate PDF processing

Changes:
- Added logic to filter out .pdf from code_extensions when loading
  other file types if PDFs were processed separately
- Updated SimpleDirectoryReader to use filtered extensions
- Added check to skip loading if no other extensions to process

2025-11-30 11:18:57 -08:00

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Expand all files Collapse all files

Compare commits

1 Commits revert-161 ... fix/pdf-du

Diff Content Not Available

1 Commits

revert-161 ... fix/pdf-du