* feat: Add MCP integration support for Slack and Twitter - Implement SlackMCPReader for connecting to Slack MCP servers - Implement TwitterMCPReader for connecting to Twitter MCP servers - Add SlackRAG and TwitterRAG applications with full CLI support - Support live data fetching via Model Context Protocol (MCP) - Add comprehensive documentation and usage examples - Include connection testing capabilities with --test-connection flag - Add standalone tests for core functionality - Update README with detailed MCP integration guide - Add Aakash Suresh to Active Contributors Resolves #36 * fix: Resolve linting issues in MCP integration - Replace deprecated typing.Dict/List with built-in dict/list - Fix boolean comparisons (== True/False) to direct checks - Remove unused variables in demo script - Update type annotations to use modern Python syntax All pre-commit hooks should now pass. * fix: Apply final formatting fixes for pre-commit hooks - Remove unused imports (asyncio, pathlib.Path) - Remove unused class imports in demo script - Ensure all files pass ruff format and pre-commit checks This should resolve all remaining CI linting issues. * fix: Apply pre-commit formatting changes - Fix trailing whitespace in all files - Apply ruff formatting to match project standards - Ensure consistent code style across all MCP integration files This commit applies the exact changes that pre-commit hooks expect. * fix: Apply pre-commit hooks formatting fixes - Remove trailing whitespace from all files - Fix ruff formatting issues (2 errors resolved) - Apply consistent code formatting across 3 files - Ensure all files pass pre-commit validation This resolves all CI formatting failures. * fix: Update MCP RAG classes to match BaseRAGExample signature - Fix SlackMCPRAG and TwitterMCPRAG __init__ methods to provide required parameters - Add name, description, and default_index_name to super().__init__ calls - Resolves test failures: test_slack_rag_initialization and test_twitter_rag_initialization This fixes the TypeError caused by BaseRAGExample requiring additional parameters. * style: Apply ruff formatting - add trailing commas - Add trailing commas to super().__init__ calls in SlackMCPRAG and TwitterMCPRAG - Fixes ruff format pre-commit hook requirements * fix: Resolve SentenceTransformer model_kwargs parameter conflict - Fix local_files_only parameter conflict in embedding_compute.py - Create separate copies of model_kwargs and tokenizer_kwargs for local vs network loading - Prevents parameter conflicts when falling back from local to network loading - Resolves TypeError in test_readme_examples.py tests This addresses the SentenceTransformer initialization issues in CI tests. * fix: Add comprehensive SentenceTransformer version compatibility - Handle both old and new sentence-transformers versions - Gracefully fallback from advanced parameters to basic initialization - Catch TypeError for model_kwargs/tokenizer_kwargs and use basic SentenceTransformer init - Ensures compatibility across different CI environments and local setups - Maintains optimization benefits where supported while ensuring broad compatibility This resolves test failures in CI environments with older sentence-transformers versions. * style: Apply ruff formatting to embedding_compute.py - Break long logger.warning lines for better readability - Fixes pre-commit hook formatting requirements * docs: Comprehensive documentation improvements for better user experience - Add clear step-by-step Getting Started Guide for new users - Add comprehensive CLI Reference with all commands and options - Improve installation instructions with clear steps and verification - Add detailed troubleshooting section for common issues (Ollama, OpenAI, etc.) - Clarify difference between CLI commands and specialized apps - Add environment variables documentation - Improve MCP integration documentation with CLI integration examples - Address user feedback about confusing installation and setup process This resolves documentation gaps that made LEANN difficult for non-specialists to use. * style: Remove trailing whitespace from README.md - Fix trailing whitespace issues found by pre-commit hooks - Ensures consistent formatting across documentation * docs: Simplify README by removing excessive documentation - Remove overly complex CLI reference and getting started sections (lines 61-334) - Remove emojis from section headers for cleaner appearance - Keep README simple and focused as requested - Maintain essential MCP integration documentation This addresses feedback to keep documentation minimal and avoid auto-generated content. * docs: Address maintainer feedback on README improvements - Restore emojis in section headers (Prerequisites and Quick Install) - Add MCP live data feature mention in line 23 with links to Slack and Twitter - Add detailed API credential setup instructions for Slack: - Step-by-step Slack App creation process - Required OAuth scopes and permissions - Clear token identification (xoxb- vs xapp-) - Add detailed API credential setup instructions for Twitter: - Twitter Developer Account application process - API v2 requirements for bookmarks access - Required permissions and scopes This addresses maintainer feedback to make API setup more user-friendly.
209 lines
6.6 KiB
Python
209 lines
6.6 KiB
Python
#!/usr/bin/env python3
|
|
"""
|
|
Test script for MCP integration implementations.
|
|
|
|
This script tests the basic functionality of the MCP readers and RAG applications
|
|
without requiring actual MCP servers to be running.
|
|
"""
|
|
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
# Add the parent directory to the path so we can import from apps
|
|
sys.path.append(str(Path(__file__).parent.parent))
|
|
|
|
from apps.slack_data.slack_mcp_reader import SlackMCPReader
|
|
from apps.slack_rag import SlackMCPRAG
|
|
from apps.twitter_data.twitter_mcp_reader import TwitterMCPReader
|
|
from apps.twitter_rag import TwitterMCPRAG
|
|
|
|
|
|
def test_slack_reader_initialization():
|
|
"""Test that SlackMCPReader can be initialized with various parameters."""
|
|
print("Testing SlackMCPReader initialization...")
|
|
|
|
# Test basic initialization
|
|
reader = SlackMCPReader("slack-mcp-server")
|
|
assert reader.mcp_server_command == "slack-mcp-server"
|
|
assert reader.concatenate_conversations
|
|
assert reader.max_messages_per_conversation == 100
|
|
|
|
# Test with custom parameters
|
|
reader = SlackMCPReader(
|
|
"custom-slack-server",
|
|
workspace_name="test-workspace",
|
|
concatenate_conversations=False,
|
|
max_messages_per_conversation=50,
|
|
)
|
|
assert reader.workspace_name == "test-workspace"
|
|
assert not reader.concatenate_conversations
|
|
assert reader.max_messages_per_conversation == 50
|
|
|
|
print("✅ SlackMCPReader initialization tests passed")
|
|
|
|
|
|
def test_twitter_reader_initialization():
|
|
"""Test that TwitterMCPReader can be initialized with various parameters."""
|
|
print("Testing TwitterMCPReader initialization...")
|
|
|
|
# Test basic initialization
|
|
reader = TwitterMCPReader("twitter-mcp-server")
|
|
assert reader.mcp_server_command == "twitter-mcp-server"
|
|
assert reader.include_tweet_content
|
|
assert reader.include_metadata
|
|
assert reader.max_bookmarks == 1000
|
|
|
|
# Test with custom parameters
|
|
reader = TwitterMCPReader(
|
|
"custom-twitter-server",
|
|
username="testuser",
|
|
include_tweet_content=False,
|
|
include_metadata=False,
|
|
max_bookmarks=500,
|
|
)
|
|
assert reader.username == "testuser"
|
|
assert not reader.include_tweet_content
|
|
assert not reader.include_metadata
|
|
assert reader.max_bookmarks == 500
|
|
|
|
print("✅ TwitterMCPReader initialization tests passed")
|
|
|
|
|
|
def test_slack_message_formatting():
|
|
"""Test Slack message formatting functionality."""
|
|
print("Testing Slack message formatting...")
|
|
|
|
reader = SlackMCPReader("slack-mcp-server")
|
|
|
|
# Test basic message formatting
|
|
message = {
|
|
"text": "Hello, world!",
|
|
"user": "john_doe",
|
|
"channel": "general",
|
|
"ts": "1234567890.123456",
|
|
}
|
|
|
|
formatted = reader._format_message(message)
|
|
assert "Channel: #general" in formatted
|
|
assert "User: john_doe" in formatted
|
|
assert "Message: Hello, world!" in formatted
|
|
assert "Time:" in formatted
|
|
|
|
# Test with missing fields
|
|
message = {"text": "Simple message"}
|
|
formatted = reader._format_message(message)
|
|
assert "Message: Simple message" in formatted
|
|
|
|
print("✅ Slack message formatting tests passed")
|
|
|
|
|
|
def test_twitter_bookmark_formatting():
|
|
"""Test Twitter bookmark formatting functionality."""
|
|
print("Testing Twitter bookmark formatting...")
|
|
|
|
reader = TwitterMCPReader("twitter-mcp-server")
|
|
|
|
# Test basic bookmark formatting
|
|
bookmark = {
|
|
"text": "This is a great article about AI!",
|
|
"author": "ai_researcher",
|
|
"created_at": "2024-01-01T12:00:00Z",
|
|
"url": "https://twitter.com/ai_researcher/status/123456789",
|
|
"likes": 42,
|
|
"retweets": 15,
|
|
}
|
|
|
|
formatted = reader._format_bookmark(bookmark)
|
|
assert "=== Twitter Bookmark ===" in formatted
|
|
assert "Author: @ai_researcher" in formatted
|
|
assert "Content:" in formatted
|
|
assert "This is a great article about AI!" in formatted
|
|
assert "URL: https://twitter.com" in formatted
|
|
assert "Likes: 42" in formatted
|
|
assert "Retweets: 15" in formatted
|
|
|
|
# Test with minimal data
|
|
bookmark = {"text": "Simple tweet"}
|
|
formatted = reader._format_bookmark(bookmark)
|
|
assert "=== Twitter Bookmark ===" in formatted
|
|
assert "Simple tweet" in formatted
|
|
|
|
print("✅ Twitter bookmark formatting tests passed")
|
|
|
|
|
|
def test_slack_rag_initialization():
|
|
"""Test that SlackMCPRAG can be initialized."""
|
|
print("Testing SlackMCPRAG initialization...")
|
|
|
|
app = SlackMCPRAG()
|
|
assert app.default_index_name == "slack_messages"
|
|
assert hasattr(app, "parser")
|
|
|
|
print("✅ SlackMCPRAG initialization tests passed")
|
|
|
|
|
|
def test_twitter_rag_initialization():
|
|
"""Test that TwitterMCPRAG can be initialized."""
|
|
print("Testing TwitterMCPRAG initialization...")
|
|
|
|
app = TwitterMCPRAG()
|
|
assert app.default_index_name == "twitter_bookmarks"
|
|
assert hasattr(app, "parser")
|
|
|
|
print("✅ TwitterMCPRAG initialization tests passed")
|
|
|
|
|
|
def test_concatenated_content_creation():
|
|
"""Test creation of concatenated content from multiple messages."""
|
|
print("Testing concatenated content creation...")
|
|
|
|
reader = SlackMCPReader("slack-mcp-server", workspace_name="test-workspace")
|
|
|
|
messages = [
|
|
{"text": "First message", "user": "alice", "ts": "1000"},
|
|
{"text": "Second message", "user": "bob", "ts": "2000"},
|
|
{"text": "Third message", "user": "charlie", "ts": "3000"},
|
|
]
|
|
|
|
content = reader._create_concatenated_content(messages, "general")
|
|
|
|
assert "Slack Channel: #general" in content
|
|
assert "Message Count: 3" in content
|
|
assert "Workspace: test-workspace" in content
|
|
assert "First message" in content
|
|
assert "Second message" in content
|
|
assert "Third message" in content
|
|
|
|
print("✅ Concatenated content creation tests passed")
|
|
|
|
|
|
def main():
|
|
"""Run all tests."""
|
|
print("🧪 Running MCP Integration Tests")
|
|
print("=" * 50)
|
|
|
|
try:
|
|
test_slack_reader_initialization()
|
|
test_twitter_reader_initialization()
|
|
test_slack_message_formatting()
|
|
test_twitter_bookmark_formatting()
|
|
test_slack_rag_initialization()
|
|
test_twitter_rag_initialization()
|
|
test_concatenated_content_creation()
|
|
|
|
print("\n" + "=" * 50)
|
|
print("🎉 All tests passed! MCP integration is working correctly.")
|
|
print("\nNext steps:")
|
|
print("1. Install actual MCP servers for Slack and Twitter")
|
|
print("2. Configure API credentials")
|
|
print("3. Test with --test-connection flag")
|
|
print("4. Start indexing your live data!")
|
|
|
|
except Exception as e:
|
|
print(f"\n❌ Test failed: {e}")
|
|
sys.exit(1)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|