Compare commits

...

22 Commits

Author SHA1 Message Date
aakash
bc00b29d7e Remove Key Features section from Slack RAG examples
- Simplified documentation by removing the bullet point list
- Keeps the focus on the actual examples and screenshots
2025-10-18 23:55:36 -07:00
aakash
22664dace8 Update Slack RAG documentation with Ollama integration and new screenshots
- Updated slack-setup-guide.md with comprehensive Ollama setup instructions
- Added 6 new screenshots showing complete RAG workflow:
  - Command setup, search results, and LLM responses for both queries
- Removed simulated LLM references, now uses real Ollama with llama3.2:1b
- Enhanced documentation with step-by-step Ollama installation
- Updated troubleshooting checklist to include Ollama-specific checks
- Fixed command syntax and added proper Ollama configuration
- Demonstrates working Slack RAG with real AI-generated responses
2025-10-18 23:46:57 -07:00
aakash
fb9405e99a Update Slack RAG integration with improved CSV parsing and new screenshots
- Fixed CSV message parsing in slack_mcp_reader.py to properly handle individual messages
- Updated slack_rag.py to filter empty channel strings
- Enhanced slack-setup-guide.md with two new query examples:
  - Advisor Models query: 'train black-box models to adopt to your personal data'
  - Barbarians at the Gate query: 'AI-driven research systems ADRS'
- Replaced old screenshots with four new ones showing both query examples
- Updated documentation to use User OAuth Token (xoxp-) instead of Bot Token (xoxb-)
- Added proper command examples with --no-concatenate-conversations and --force-rebuild flags
2025-10-18 22:25:16 -07:00
aakash
73ffc3cc37 Update Slack RAG example to show LEANN announcement retrieval
- Change query from 'PUBPOL 290' to 'What is LEANN about?' for more challenging retrieval
- Update command to use python -m apps.slack_rag instead of test script
- Add expected response showing Yichuan Wang's LEANN announcement message
- Emphasize this demonstrates ability to find specific announcements in conversation history
- Update description to highlight challenging query capabilities
2025-10-18 16:28:09 -07:00
aakash
d411c94f21 Remove test_channel_by_id_or_name.py
- Clean up temporary test file that was used for debugging
- Keep only the main slack_rag.py application for production use
2025-10-18 01:38:16 -07:00
aakash
3937e1b143 Update Slack integration screenshot with latest changes 2025-10-18 01:25:08 -07:00
aakash
8221b37156 Add Slack integration screenshots to docs/videos
- Add slack_integration.png showing RAG query results
- Add slack_integration_2.png showing additional demo functionality
- Fixes lychee link checker errors for missing image files
2025-10-18 01:14:03 -07:00
aakash
c05650103b Fix Slack MCP integration and update documentation
- Fix SlackMCPReader to use conversations_history instead of channels_list
- Add fallback imports for leann.interactive_utils and leann.settings
- Update slack-setup-guide.md with real screenshots and improved text
- Remove old screenshot files
2025-10-18 01:08:34 -07:00
aakash
8bdd5a17ba Docs: finalize Slack setup guide with Sky random RAG example and image path fixes\n\n- Redact example tokens from docs 2025-10-15 12:53:46 -07:00
aakash
8b537f6246 Docs: fix image path for lychee (use videos/ relative under docs/) 2025-10-15 04:18:45 -07:00
aakash
0a7a283dda Docs/CI: fix broken image paths and ruff lint\n\n- Move screenshot to docs/videos and update references\n- Remove obsolete rag-query-results image\n- Rename variable to satisfy ruff 2025-10-15 04:15:44 -07:00
aakash
151b24a456 Docs: add real RAG example for Sky Lab #random
- Embed screenshot videos/rag-sky-random.png
- Add step-by-step commands and notes
- Include helper test script tests/test_channel_by_id_or_name.py
- Redact example tokens from docs
2025-10-15 04:09:26 -07:00
aakash
06505c069e Update Slack setup guide with bot invitation requirements
- Add important section about inviting bot to channels before RAG queries
- Explain the 'not_in_channel' errors and their meaning
- Provide clear steps for bot invitation process
- Document realistic scenario where bot needs explicit channel access
- Update documentation to be more professional and less cursor-style
2025-10-12 16:17:47 -07:00
aakash
c76a1e2c71 Add real RAG example showing intelligent Slack query functionality
- Add detailed example of asking 'What is LEANN about?'
- Show retrieved messages from Slack channels
- Demonstrate intelligent answer generation based on context
- Add command example for running real RAG queries
- Explain the 4-step process: retrieve, index, generate, cite
2025-10-12 15:49:42 -07:00
aakash
1b80bcf1a0 Fix formatting issues in Slack setup guide
- Remove trailing whitespace
- Fix end of file formatting
- Pre-commit hooks formatting fixes
2025-10-12 14:17:56 -07:00
aakash
f28c3ecc7a Add comprehensive Slack setup guide with success screenshot
- Create detailed setup guide with step-by-step instructions
- Add troubleshooting section for common issues like cache sync errors
- Include real terminal output example from successful integration
- Add screenshot showing VS Code interface with Slack channel data
- Remove excessive emojis for more professional documentation
- Document retry logic improvements and CLI arguments
2025-10-12 14:09:52 -07:00
aakash
9e067e7fb3 Fix trailing whitespace in slack setup guide
Pre-commit hooks formatting fixes
2025-10-09 20:04:32 -07:00
Aakash Suresh
1f98681d68 Merge branch 'main' into fix/twitter-bookmarks-anchor-link 2025-10-09 20:02:41 -07:00
aakash
3b94b7b8af Improve Slack MCP integration with retry logic and comprehensive setup guide
- Add retry mechanism with exponential backoff for cache sync issues
- Handle 'users cache is not ready yet' errors gracefully
- Add max-retries and retry-delay CLI arguments for better control
- Create comprehensive Slack setup guide with troubleshooting
- Update README with link to detailed setup guide
- Improve error messages and user experience
2025-10-09 19:54:38 -07:00
aakash
df168634c8 fix: Point Slack and Twitter links to main MCP section
- Both Slack and Twitter are subsections under MCP Integration
- Links should point to #mcp-integration-rag-on-live-data-from-any-platform
- Users will land on the MCP section and can find both Slack and Twitter subsections there

This matches the actual document structure where Slack and Twitter are under the MCP Integration section.
2025-10-08 02:24:04 -07:00
aakash
9798e3cbe6 fix: Fix Slack messages anchor link as well
- Convert Slack Messages from collapsible details to proper header
- Update internal link to match new anchor format
- Ensures external links to #slack-messages-search-your-team-conversations work correctly

Both Twitter and Slack MCP sections now have reliable anchor links.
2025-10-08 02:16:59 -07:00
aakash
ab8bcba2c6 fix: Fix Twitter bookmarks anchor link
- Convert Twitter Bookmarks from collapsible details to proper header
- Update internal link to match new anchor format
- Ensures external links to #twitter-bookmarks-your-personal-tweet-library work correctly

Fixes broken link: https://github.com/yichuan-w/LEANN?tab=readme-ov-file#twitter-bookmarks-your-personal-tweet-library
2025-10-08 01:52:14 -07:00
11 changed files with 684 additions and 61 deletions

View File

@@ -781,7 +781,7 @@ Once your iMessage conversations are indexed, you can search with queries like:
### MCP Integration: RAG on Live Data from Any Platform ### MCP Integration: RAG on Live Data from Any Platform
**NEW!** Connect to live data sources through the Model Context Protocol (MCP). LEANN now supports real-time RAG on platforms like Slack, Twitter, and more through standardized MCP servers. Connect to live data sources through the Model Context Protocol (MCP). LEANN now supports real-time RAG on platforms like Slack, Twitter, and more through standardized MCP servers.
**Key Benefits:** **Key Benefits:**
- **Live Data Access**: Fetch real-time data without manual exports - **Live Data Access**: Fetch real-time data without manual exports
@@ -805,18 +805,17 @@ python -m apps.slack_rag \
--query "What did we decide about the product launch?" --query "What did we decide about the product launch?"
``` ```
**Setup Requirements:** **📖 Comprehensive Setup Guide**: For detailed setup instructions, troubleshooting common issues (like "users cache is not ready yet"), and advanced configuration options, see our [**Slack Setup Guide**](docs/slack-setup-guide.md).
**Quick Setup:**
1. Install a Slack MCP server (e.g., `npm install -g slack-mcp-server`) 1. Install a Slack MCP server (e.g., `npm install -g slack-mcp-server`)
2. Create a Slack App and get API credentials: 2. Create a Slack App and get API credentials (see detailed guide above)
- Go to [api.slack.com/apps](https://api.slack.com/apps) and create a new app 3. Set environment variables:
- Under "OAuth & Permissions", add these Bot Token Scopes: `channels:read`, `channels:history`, `groups:read`, `groups:history`, `im:read`, `im:history`, `mpim:read`, `mpim:history`
- Install the app to your workspace and copy the "Bot User OAuth Token" (starts with `xoxb-`)
- Under "App-Level Tokens", create a token with `connections:write` scope (starts with `xapp-`)
```bash ```bash
export SLACK_BOT_TOKEN="xoxb-your-bot-token" export SLACK_BOT_TOKEN="xoxb-your-bot-token"
export SLACK_APP_TOKEN="xapp-your-app-token" export SLACK_APP_TOKEN="xapp-your-app-token" # Optional
``` ```
3. Test connection with `--test-connection` flag 4. Test connection with `--test-connection` flag
**Arguments:** **Arguments:**
- `--mcp-server`: Command to start the Slack MCP server - `--mcp-server`: Command to start the Slack MCP server
@@ -824,6 +823,8 @@ python -m apps.slack_rag \
- `--channels`: Specific channels to index (optional) - `--channels`: Specific channels to index (optional)
- `--concatenate-conversations`: Group messages by channel (default: true) - `--concatenate-conversations`: Group messages by channel (default: true)
- `--max-messages-per-channel`: Limit messages per channel (default: 100) - `--max-messages-per-channel`: Limit messages per channel (default: 100)
- `--max-retries`: Maximum retries for cache sync issues (default: 5)
- `--retry-delay`: Initial delay between retries in seconds (default: 2.0)
#### 🐦 Twitter Bookmarks: Your Personal Tweet Library #### 🐦 Twitter Bookmarks: Your Personal Tweet Library
@@ -925,7 +926,7 @@ Want to add support for other platforms? LEANN's MCP integration is designed for
### 🚀 Claude Code Integration: Transform Your Development Workflow! ### 🚀 Claude Code Integration: Transform Your Development Workflow!
<details> <details>
<summary><strong>NEW!! ASTAware Code Chunking</strong></summary> <summary><strong>ASTAware Code Chunking</strong></summary>
LEANN features intelligent code chunking that preserves semantic boundaries (functions, classes, methods) for Python, Java, C#, and TypeScript, improving code understanding compared to text-based chunking. LEANN features intelligent code chunking that preserves semantic boundaries (functions, classes, methods) for Python, Java, C#, and TypeScript, improving code understanding compared to text-based chunking.

View File

@@ -10,9 +10,39 @@ from typing import Any
import dotenv import dotenv
from leann.api import LeannBuilder, LeannChat from leann.api import LeannBuilder, LeannChat
from leann.interactive_utils import create_rag_session
# Optional import: older PyPI builds may not include interactive_utils
try:
from leann.interactive_utils import create_rag_session
except ImportError:
def create_rag_session(app_name: str, data_description: str):
class _SimpleSession:
def run_interactive_loop(self, handler):
print(f"Interactive session for {app_name}: {data_description}")
print("Interactive mode not available in this build")
return _SimpleSession()
from leann.registry import register_project_directory from leann.registry import register_project_directory
from leann.settings import resolve_ollama_host, resolve_openai_api_key, resolve_openai_base_url
# Optional import: older PyPI builds may not include settings
try:
from leann.settings import resolve_ollama_host, resolve_openai_api_key, resolve_openai_base_url
except ImportError:
# Minimal fallbacks if settings helpers are unavailable
import os
def resolve_ollama_host(value: str | None) -> str | None:
return value or os.getenv("LEANN_OLLAMA_HOST") or os.getenv("OLLAMA_HOST")
def resolve_openai_api_key(value: str | None) -> str | None:
return value or os.getenv("OPENAI_API_KEY")
def resolve_openai_base_url(value: str | None) -> str | None:
return value or os.getenv("OPENAI_BASE_URL")
dotenv.load_dotenv() dotenv.load_dotenv()

View File

@@ -29,6 +29,8 @@ class SlackMCPReader:
workspace_name: Optional[str] = None, workspace_name: Optional[str] = None,
concatenate_conversations: bool = True, concatenate_conversations: bool = True,
max_messages_per_conversation: int = 100, max_messages_per_conversation: int = 100,
max_retries: int = 5,
retry_delay: float = 2.0,
): ):
""" """
Initialize the Slack MCP Reader. Initialize the Slack MCP Reader.
@@ -38,11 +40,15 @@ class SlackMCPReader:
workspace_name: Optional workspace name to filter messages workspace_name: Optional workspace name to filter messages
concatenate_conversations: Whether to group messages by channel/thread concatenate_conversations: Whether to group messages by channel/thread
max_messages_per_conversation: Maximum messages to include per conversation max_messages_per_conversation: Maximum messages to include per conversation
max_retries: Maximum number of retries for failed operations
retry_delay: Initial delay between retries in seconds
""" """
self.mcp_server_command = mcp_server_command self.mcp_server_command = mcp_server_command
self.workspace_name = workspace_name self.workspace_name = workspace_name
self.concatenate_conversations = concatenate_conversations self.concatenate_conversations = concatenate_conversations
self.max_messages_per_conversation = max_messages_per_conversation self.max_messages_per_conversation = max_messages_per_conversation
self.max_retries = max_retries
self.retry_delay = retry_delay
self.mcp_process = None self.mcp_process = None
async def start_mcp_server(self): async def start_mcp_server(self):
@@ -110,11 +116,73 @@ class SlackMCPReader:
return response.get("result", {}).get("tools", []) return response.get("result", {}).get("tools", [])
def _is_cache_sync_error(self, error: dict) -> bool:
"""Check if the error is related to users cache not being ready."""
if isinstance(error, dict):
message = error.get("message", "").lower()
return (
"users cache is not ready" in message or "sync process is still running" in message
)
return False
async def _retry_with_backoff(self, func, *args, **kwargs):
"""Retry a function with exponential backoff, especially for cache sync issues."""
last_exception = None
for attempt in range(self.max_retries + 1):
try:
return await func(*args, **kwargs)
except Exception as e:
last_exception = e
# Check if this is a cache sync error
error_dict = {}
if hasattr(e, "args") and e.args and isinstance(e.args[0], dict):
error_dict = e.args[0]
elif "Failed to fetch messages" in str(e):
# Try to extract error from the exception message
import re
match = re.search(r"'error':\s*(\{[^}]+\})", str(e))
if match:
try:
error_dict = eval(match.group(1))
except (ValueError, SyntaxError, NameError):
pass
else:
# Try alternative format
match = re.search(r"Failed to fetch messages:\s*(\{[^}]+\})", str(e))
if match:
try:
error_dict = eval(match.group(1))
except (ValueError, SyntaxError, NameError):
pass
if self._is_cache_sync_error(error_dict):
if attempt < self.max_retries:
delay = self.retry_delay * (2**attempt) # Exponential backoff
logger.info(
f"Cache sync not ready, waiting {delay:.1f}s before retry {attempt + 1}/{self.max_retries}"
)
await asyncio.sleep(delay)
continue
else:
logger.warning(
f"Cache sync still not ready after {self.max_retries} retries, giving up"
)
break
else:
# Not a cache sync error, don't retry
break
# If we get here, all retries failed or it's not a retryable error
raise last_exception
async def fetch_slack_messages( async def fetch_slack_messages(
self, channel: Optional[str] = None, limit: int = 100 self, channel: Optional[str] = None, limit: int = 100
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
""" """
Fetch Slack messages using MCP tools. Fetch Slack messages using MCP tools with retry logic for cache sync issues.
Args: Args:
channel: Optional channel name to filter messages channel: Optional channel name to filter messages
@@ -123,32 +191,59 @@ class SlackMCPReader:
Returns: Returns:
List of message dictionaries List of message dictionaries
""" """
return await self._retry_with_backoff(self._fetch_slack_messages_impl, channel, limit)
async def _fetch_slack_messages_impl(
self, channel: Optional[str] = None, limit: int = 100
) -> list[dict[str, Any]]:
"""
Internal implementation of fetch_slack_messages without retry logic.
"""
# This is a generic implementation - specific MCP servers may have different tool names # This is a generic implementation - specific MCP servers may have different tool names
# Common tool names might be: 'get_messages', 'list_messages', 'fetch_channel_history' # Common tool names might be: 'get_messages', 'list_messages', 'fetch_channel_history'
tools = await self.list_available_tools() tools = await self.list_available_tools()
logger.info(f"Available tools: {[tool.get('name') for tool in tools]}")
message_tool = None message_tool = None
# Look for a tool that can fetch messages # Look for a tool that can fetch messages - prioritize conversations_history
message_tool = None
# First, try to find conversations_history specifically
for tool in tools: for tool in tools:
tool_name = tool.get("name", "").lower() tool_name = tool.get("name", "").lower()
if any( if "conversations_history" in tool_name:
keyword in tool_name
for keyword in ["message", "history", "channel", "conversation"]
):
message_tool = tool message_tool = tool
logger.info(f"Found conversations_history tool: {tool}")
break break
# If not found, look for other message-fetching tools
if not message_tool:
for tool in tools:
tool_name = tool.get("name", "").lower()
if any(
keyword in tool_name
for keyword in ["conversations_search", "message", "history"]
):
message_tool = tool
break
if not message_tool: if not message_tool:
raise RuntimeError("No message fetching tool found in MCP server") raise RuntimeError("No message fetching tool found in MCP server")
# Prepare tool call parameters # Prepare tool call parameters
tool_params = {"limit": limit} tool_params = {"limit": "180d"} # Use 180 days to get older messages
if channel: if channel:
# Try common parameter names for channel specification # For conversations_history, use channel_id parameter
for param_name in ["channel", "channel_id", "channel_name"]: if message_tool["name"] == "conversations_history":
tool_params[param_name] = channel tool_params["channel_id"] = channel
break else:
# Try common parameter names for channel specification
for param_name in ["channel", "channel_id", "channel_name"]:
tool_params[param_name] = channel
break
logger.info(f"Tool parameters: {tool_params}")
fetch_request = { fetch_request = {
"jsonrpc": "2.0", "jsonrpc": "2.0",
@@ -170,8 +265,8 @@ class SlackMCPReader:
try: try:
messages = json.loads(content["text"]) messages = json.loads(content["text"])
except json.JSONDecodeError: except json.JSONDecodeError:
# If not JSON, treat as plain text # If not JSON, try to parse as CSV format (Slack MCP server format)
messages = [{"text": content["text"], "channel": channel or "unknown"}] messages = self._parse_csv_messages(content["text"], channel)
else: else:
messages = result["content"] messages = result["content"]
else: else:
@@ -180,6 +275,56 @@ class SlackMCPReader:
return messages if isinstance(messages, list) else [messages] return messages if isinstance(messages, list) else [messages]
def _parse_csv_messages(self, csv_text: str, channel: str) -> list[dict[str, Any]]:
"""Parse CSV format messages from Slack MCP server."""
import csv
import io
messages = []
try:
# Split by lines and process each line as a CSV row
lines = csv_text.strip().split("\n")
if not lines:
return messages
# Skip header line if it exists
start_idx = 0
if lines[0].startswith("MsgID,UserID,UserName"):
start_idx = 1
for line in lines[start_idx:]:
if not line.strip():
continue
# Parse CSV line
reader = csv.reader(io.StringIO(line))
try:
row = next(reader)
if len(row) >= 7: # Ensure we have enough columns
message = {
"ts": row[0],
"user": row[1],
"username": row[2],
"real_name": row[3],
"channel": row[4],
"thread_ts": row[5],
"text": row[6],
"time": row[7] if len(row) > 7 else "",
"reactions": row[8] if len(row) > 8 else "",
"cursor": row[9] if len(row) > 9 else "",
}
messages.append(message)
except Exception as e:
logger.warning(f"Failed to parse CSV line: {line[:100]}... Error: {e}")
continue
except Exception as e:
logger.warning(f"Failed to parse CSV messages: {e}")
# Fallback: treat entire text as one message
messages = [{"text": csv_text, "channel": channel or "unknown"}]
return messages
def _format_message(self, message: dict[str, Any]) -> str: def _format_message(self, message: dict[str, Any]) -> str:
"""Format a single message for indexing.""" """Format a single message for indexing."""
text = message.get("text", "") text = message.get("text", "")
@@ -251,6 +396,40 @@ class SlackMCPReader:
return "\n".join(content_parts) return "\n".join(content_parts)
async def get_all_channels(self) -> list[str]:
"""Get list of all available channels."""
try:
channels_list_request = {
"jsonrpc": "2.0",
"id": 4,
"method": "tools/call",
"params": {"name": "channels_list", "arguments": {}},
}
channels_response = await self.send_mcp_request(channels_list_request)
if "result" in channels_response:
result = channels_response["result"]
if "content" in result and isinstance(result["content"], list):
content = result["content"][0] if result["content"] else {}
if "text" in content:
# Parse the channels from the response
channels = []
lines = content["text"].split("\n")
for line in lines:
if line.strip() and ("#" in line or "C" in line[:10]):
# Extract channel ID or name
parts = line.split()
for part in parts:
if part.startswith("C") and len(part) > 5:
channels.append(part)
elif part.startswith("#"):
channels.append(part[1:]) # Remove #
logger.info(f"Found {len(channels)} channels: {channels}")
return channels
return []
except Exception as e:
logger.warning(f"Failed to get channels list: {e}")
return []
async def read_slack_data(self, channels: Optional[list[str]] = None) -> list[str]: async def read_slack_data(self, channels: Optional[list[str]] = None) -> list[str]:
""" """
Read Slack data and return formatted text chunks. Read Slack data and return formatted text chunks.
@@ -287,36 +466,33 @@ class SlackMCPReader:
logger.warning(f"Failed to fetch messages from channel {channel}: {e}") logger.warning(f"Failed to fetch messages from channel {channel}: {e}")
continue continue
else: else:
# Fetch from all available channels/conversations # Fetch from all available channels
# This is a simplified approach - real implementation would need to logger.info("Fetching from all available channels...")
# discover available channels first all_channels = await self.get_all_channels()
try:
messages = await self.fetch_slack_messages(limit=1000)
if messages:
# Group messages by channel if concatenating
if self.concatenate_conversations:
channel_messages = {}
for message in messages:
channel = message.get(
"channel", message.get("channel_name", "general")
)
if channel not in channel_messages:
channel_messages[channel] = []
channel_messages[channel].append(message)
# Create concatenated content for each channel if not all_channels:
for channel, msgs in channel_messages.items(): # Fallback to common channel names if we can't get the list
text_content = self._create_concatenated_content(msgs, channel) all_channels = ["general", "random", "announcements", "C0GN5BX0F"]
logger.info(f"Using fallback channels: {all_channels}")
for channel in all_channels:
try:
logger.info(f"Searching channel: {channel}")
messages = await self.fetch_slack_messages(channel=channel, limit=1000)
if messages:
if self.concatenate_conversations:
text_content = self._create_concatenated_content(messages, channel)
if text_content.strip(): if text_content.strip():
all_texts.append(text_content) all_texts.append(text_content)
else: else:
# Process individual messages # Process individual messages
for message in messages: for message in messages:
formatted_msg = self._format_message(message) formatted_msg = self._format_message(message)
if formatted_msg.strip(): if formatted_msg.strip():
all_texts.append(formatted_msg) all_texts.append(formatted_msg)
except Exception as e: except Exception as e:
logger.error(f"Failed to fetch messages: {e}") logger.warning(f"Failed to fetch messages from channel {channel}: {e}")
continue
return all_texts return all_texts

View File

@@ -78,6 +78,20 @@ class SlackMCPRAG(BaseRAGExample):
help="Test MCP server connection and list available tools without indexing", help="Test MCP server connection and list available tools without indexing",
) )
parser.add_argument(
"--max-retries",
type=int,
default=5,
help="Maximum number of retries for failed operations (default: 5)",
)
parser.add_argument(
"--retry-delay",
type=float,
default=2.0,
help="Initial delay between retries in seconds (default: 2.0)",
)
async def test_mcp_connection(self, args) -> bool: async def test_mcp_connection(self, args) -> bool:
"""Test the MCP server connection and display available tools.""" """Test the MCP server connection and display available tools."""
print(f"Testing connection to MCP server: {args.mcp_server}") print(f"Testing connection to MCP server: {args.mcp_server}")
@@ -88,12 +102,14 @@ class SlackMCPRAG(BaseRAGExample):
workspace_name=args.workspace_name, workspace_name=args.workspace_name,
concatenate_conversations=not args.no_concatenate_conversations, concatenate_conversations=not args.no_concatenate_conversations,
max_messages_per_conversation=args.max_messages_per_channel, max_messages_per_conversation=args.max_messages_per_channel,
max_retries=args.max_retries,
retry_delay=args.retry_delay,
) )
async with reader: async with reader:
tools = await reader.list_available_tools() tools = await reader.list_available_tools()
print("\nSuccessfully connected to MCP server!") print("Successfully connected to MCP server!")
print(f"Available tools ({len(tools)}):") print(f"Available tools ({len(tools)}):")
for i, tool in enumerate(tools, 1): for i, tool in enumerate(tools, 1):
@@ -115,7 +131,7 @@ class SlackMCPRAG(BaseRAGExample):
return True return True
except Exception as e: except Exception as e:
print(f"\nFailed to connect to MCP server: {e}") print(f"Failed to connect to MCP server: {e}")
print("\nTroubleshooting tips:") print("\nTroubleshooting tips:")
print("1. Make sure the MCP server is installed and accessible") print("1. Make sure the MCP server is installed and accessible")
print("2. Check if the server command is correct") print("2. Check if the server command is correct")
@@ -130,8 +146,11 @@ class SlackMCPRAG(BaseRAGExample):
if args.workspace_name: if args.workspace_name:
print(f"Workspace: {args.workspace_name}") print(f"Workspace: {args.workspace_name}")
if args.channels: # Filter out empty strings from channels
print(f"Channels: {', '.join(args.channels)}") channels = [ch for ch in args.channels if ch.strip()] if args.channels else None
if channels:
print(f"Channels: {', '.join(channels)}")
else: else:
print("Fetching from all available channels") print("Fetching from all available channels")
@@ -146,18 +165,20 @@ class SlackMCPRAG(BaseRAGExample):
workspace_name=args.workspace_name, workspace_name=args.workspace_name,
concatenate_conversations=concatenate, concatenate_conversations=concatenate,
max_messages_per_conversation=args.max_messages_per_channel, max_messages_per_conversation=args.max_messages_per_channel,
max_retries=args.max_retries,
retry_delay=args.retry_delay,
) )
texts = await reader.read_slack_data(channels=args.channels) texts = await reader.read_slack_data(channels=channels)
if not texts: if not texts:
print("No messages found! This could mean:") print("No messages found! This could mean:")
print("- The MCP server couldn't fetch messages") print("- The MCP server couldn't fetch messages")
print("- The specified channels don't exist or are empty") print("- The specified channels don't exist or are empty")
print("- Authentication issues with the Slack workspace") print("- Authentication issues with the Slack workspace")
return [] return []
print(f"Successfully loaded {len(texts)} text chunks from Slack") print(f"Successfully loaded {len(texts)} text chunks from Slack")
# Show sample of what was loaded # Show sample of what was loaded
if texts: if texts:
@@ -170,7 +191,7 @@ class SlackMCPRAG(BaseRAGExample):
return texts return texts
except Exception as e: except Exception as e:
print(f"Error loading Slack data: {e}") print(f"Error loading Slack data: {e}")
print("\nThis might be due to:") print("\nThis might be due to:")
print("- MCP server connection issues") print("- MCP server connection issues")
print("- Authentication problems") print("- Authentication problems")
@@ -188,7 +209,7 @@ class SlackMCPRAG(BaseRAGExample):
if not success: if not success:
return return
print( print(
"\n🎉 MCP server is working! You can now run without --test-connection to start indexing." "MCP server is working! You can now run without --test-connection to start indexing."
) )
return return

395
docs/slack-setup-guide.md Normal file
View File

@@ -0,0 +1,395 @@
# Slack Integration Setup Guide
This guide provides step-by-step instructions for setting up Slack integration with LEANN.
## Overview
LEANN's Slack integration uses MCP (Model Context Protocol) servers to fetch and index your Slack messages for RAG (Retrieval-Augmented Generation). This allows you to search through your Slack conversations using natural language queries.
## Prerequisites
1. **Slack Workspace Access**: You need admin or owner permissions in your Slack workspace to create apps and configure OAuth tokens.
2. **Slack MCP Server**: Install a Slack MCP server (e.g., `slack-mcp-server` via npm)
3. **LEANN**: Ensure you have LEANN installed and working
## Step 1: Create a Slack App
### 1.1 Go to Slack API Dashboard
1. Visit [https://api.slack.com/apps](https://api.slack.com/apps)
2. Click **"Create New App"**
3. Choose **"From scratch"**
4. Enter your app name (e.g., "LEANN Slack Integration")
5. Select your workspace
6. Click **"Create App"**
### 1.2 Configure App Permissions
#### Token Scopes
1. In your app dashboard, go to **"OAuth & Permissions"** in the left sidebar
2. Scroll down to **"Scopes"** section
3. Under **"Bot Token Scopes & OAuth Scope"**, click **"Add an OAuth Scope"**
4. Add the following scopes:
- `channels:read` - Read public channel information
- `channels:history` - Read messages in public channels
- `groups:read` - Read private channel information
- `groups:history` - Read messages in private channels
- `im:read` - Read direct message information
- `im:history` - Read direct messages
- `mpim:read` - Read group direct message information
- `mpim:history` - Read group direct messages
- `users:read` - Read user information
- `team:read` - Read workspace information
#### App-Level Tokens (Optional)
Some MCP servers may require app-level tokens:
1. Go to **"Basic Information"** in the left sidebar
2. Scroll down to **"App-Level Tokens"**
3. Click **"Generate Token and Scopes"**
4. Enter a name (e.g., "LEANN Integration")
5. Add the `connections:write` scope
6. Click **"Generate"**
7. Copy the token (starts with `xapp-`)
### 1.3 Install App to Workspace
1. Go to **"OAuth & Permissions"** in the left sidebar
2. Click **"Install to Workspace"**
3. Review the permissions and click **"Allow"**
4. Copy the **"Bot User OAuth Token"** (starts with `xoxb-`)
5. Copy the **"User OAuth Token"** (starts with `xoxp-`)
## Step 2: Install Slack MCP Server
### Option A: Using npm (Recommended)
```bash
# Install globally
npm install -g slack-mcp-server
# Or install locally
npm install slack-mcp-server
```
### Option B: Using npx (No installation required)
```bash
# Use directly without installation
npx slack-mcp-server
```
## Step 3: Install and Configure Ollama (for Real LLM Responses)
### 3.1 Install Ollama
```bash
# Install Ollama using Homebrew (macOS)
brew install ollama
# Or download from https://ollama.ai/
```
### 3.2 Start Ollama Service
```bash
# Start Ollama as a service
brew services start ollama
# Or start manually
ollama serve
```
### 3.3 Pull a Model
```bash
# Pull a lightweight model for testing
ollama pull llama3.2:1b
# Verify the model is available
ollama list
```
## Step 4: Configure Environment Variables
Create a `.env` file or set environment variables:
```bash
# Required: User OAuth Token
SLACK_OAUTH_TOKEN=xoxp-your-user-oauth-token-here
# Optional: App-Level Token (if your MCP server requires it)
SLACK_APP_TOKEN=xapp-your-app-token-here
# Optional: Workspace-specific settings
SLACK_WORKSPACE_ID=T1234567890 # Your workspace ID (optional)
```
## Step 5: Test the Setup
### 5.1 Test MCP Server Connection
```bash
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--test-connection \
--workspace-name "Your Workspace Name"
```
This will test the connection and list available tools without indexing any data.
### 5.2 Index a Specific Channel
```bash
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--workspace-name "Your Workspace Name" \
--channels general \
--query "What did we discuss about the project?"
```
### 5.3 Real RAG Query Examples
This section demonstrates successful Slack RAG integration queries against the Sky Lab Computing workspace's "random" channel. The system successfully retrieves actual conversation messages and performs semantic search with high relevance scores, including finding specific research paper announcements and technical discussions.
### Example 1: Advisor Models Query
**Query:** "train black-box models to adopt to your personal data"
This query demonstrates the system's ability to find specific research announcements about training black-box models for personal data adaptation.
![Advisor Models Query - Command Setup](videos/slack_integration_1.1.png)
![Advisor Models Query - Search Results](videos/slack_integration_1.2.png)
![Advisor Models Query - LLM Response](videos/slack_integration_1.3.png)
### Example 2: Barbarians at the Gate Query
**Query:** "AI-driven research systems ADRS"
This query demonstrates the system's ability to find specific research announcements about AI-driven research systems and algorithm discovery.
![Barbarians Query - Command Setup](videos/slack_integration_2.1.png)
![Barbarians Query - Search Results](videos/slack_integration_2.2.png)
![Barbarians Query - LLM Response](videos/slack_integration_2.3.png)
### Prerequisites
- Bot is installed in the Sky Lab Computing workspace and invited to the target channel (run `/invite @YourBotName` in the channel if needed)
- Bot token available and exported in the same terminal session
### Commands
1) Set the workspace token for this shell
```bash
export SLACK_MCP_XOXP_TOKEN="xoxp-***-redacted-***"
```
2) Run queries against the "random" channel by channel ID (C0GN5BX0F)
**Advisor Models Query:**
```bash
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--workspace-name "Sky Lab Computing" \
--channels C0GN5BX0F \
--max-messages-per-channel 100000 \
--query "train black-box models to adopt to your personal data" \
--llm ollama \
--llm-model "llama3.2:1b" \
--llm-host "http://localhost:11434" \
--no-concatenate-conversations
```
**Barbarians at the Gate Query:**
```bash
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--workspace-name "Sky Lab Computing" \
--channels C0GN5BX0F \
--max-messages-per-channel 100000 \
--query "AI-driven research systems ADRS" \
--llm ollama \
--llm-model "llama3.2:1b" \
--llm-host "http://localhost:11434" \
--no-concatenate-conversations
```
These examples demonstrate the system's ability to find and retrieve specific research announcements and technical discussions from the conversation history, showcasing the power of semantic search in Slack data.
3) Optional: Ask a broader question
```bash
python test_channel_by_id_or_name.py \
--channel-id C0GN5BX0F \
--workspace-name "Sky Lab Computing" \
--query "What is LEANN about?"
```
Notes:
- If you see `not_in_channel`, invite the bot to the channel and re-run.
- If you see `channel_not_found`, confirm the channel ID and workspace.
- Deep search via server-side “search” tools may require additional Slack scopes; the example above performs client-side filtering over retrieved history.
## Common Issues and Solutions
### Issue 1: "users cache is not ready yet" Error
**Problem**: You see this warning:
```
WARNING - Failed to fetch messages from channel random: Failed to fetch messages: {'code': -32603, 'message': 'users cache is not ready yet, sync process is still running... please wait'}
```
**Solution**: This is a common timing issue. The LEANN integration now includes automatic retry logic:
1. **Wait and Retry**: The system will automatically retry with exponential backoff (2s, 4s, 8s, etc.)
2. **Increase Retry Parameters**: If needed, you can customize retry behavior:
```bash
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--max-retries 10 \
--retry-delay 3.0 \
--channels general \
--query "Your query here"
```
3. **Keep MCP Server Running**: Start the MCP server separately and keep it running:
```bash
# Terminal 1: Start MCP server
slack-mcp-server
# Terminal 2: Run LEANN (it will connect to the running server)
python -m apps.slack_rag --mcp-server "slack-mcp-server" --channels general --query "test"
```
### Issue 2: "No message fetching tool found"
**Problem**: The MCP server doesn't have the expected tools.
**Solution**:
1. Check if your MCP server is properly installed and configured
2. Verify your Slack tokens are correct
3. Try a different MCP server implementation
4. Check the MCP server documentation for required configuration
### Issue 3: Permission Denied Errors
**Problem**: You get permission errors when trying to access channels.
**Solutions**:
1. **Check Bot Permissions**: Ensure your bot has been added to the channels you want to access
2. **Verify Token Scopes**: Make sure you have all required scopes configured
3. **Channel Access**: For private channels, the bot needs to be explicitly invited
4. **Workspace Permissions**: Ensure your Slack app has the necessary workspace permissions
### Issue 4: Empty Results
**Problem**: No messages are returned even though the channel has messages.
**Solutions**:
1. **Check Channel Names**: Ensure channel names are correct (without the # symbol)
2. **Verify Bot Access**: Make sure the bot can access the channels
3. **Check Date Ranges**: Some MCP servers have limitations on message history
4. **Increase Message Limits**: Try increasing the message limit:
```bash
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--channels general \
--max-messages-per-channel 1000 \
--query "test"
```
## Advanced Configuration
### Custom MCP Server Commands
If you need to pass additional parameters to your MCP server:
```bash
python -m apps.slack_rag \
--mcp-server "slack-mcp-server --token-file /path/to/tokens.json" \
--workspace-name "Your Workspace" \
--channels general \
--query "Your query"
```
### Multiple Workspaces
To work with multiple Slack workspaces, you can:
1. Create separate apps for each workspace
2. Use different environment variables
3. Run separate instances with different configurations
### Performance Optimization
For better performance with large workspaces:
```bash
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--workspace-name "Your Workspace" \
--max-messages-per-channel 500 \
--no-concatenate-conversations \
--query "Your query"
```
---
## Troubleshooting Checklist
- [ ] Slack app created with proper permissions
- [ ] Bot token (xoxb-) copied correctly
- [ ] App-level token (xapp-) created if needed
- [ ] MCP server installed and accessible
- [ ] Ollama installed and running (`brew services start ollama`)
- [ ] Ollama model pulled (`ollama pull llama3.2:1b`)
- [ ] Environment variables set correctly
- [ ] Bot invited to relevant channels
- [ ] Channel names specified without # symbol
- [ ] Sufficient retry attempts configured
- [ ] Network connectivity to Slack APIs
## Getting Help
If you continue to have issues:
1. **Check Logs**: Look for detailed error messages in the console output
2. **Test MCP Server**: Use `--test-connection` to verify the MCP server is working
3. **Verify Tokens**: Double-check that your Slack tokens are valid and have the right scopes
4. **Check Ollama**: Ensure Ollama is running (`ollama serve`) and the model is available (`ollama list`)
5. **Community Support**: Reach out to the LEANN community for help
## Example Commands
### Basic Usage
```bash
# Test connection
python -m apps.slack_rag --mcp-server "slack-mcp-server" --test-connection
# Index specific channels
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--workspace-name "My Company" \
--channels general random \
--query "What did we decide about the project timeline?"
```
### Advanced Usage
```bash
# With custom retry settings
python -m apps.slack_rag \
--mcp-server "slack-mcp-server" \
--workspace-name "My Company" \
--channels general \
--max-retries 10 \
--retry-delay 5.0 \
--max-messages-per-channel 2000 \
--query "Show me all decisions made in the last month"
```

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 445 KiB

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 508 KiB

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 437 KiB

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 474 KiB

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 501 KiB

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 454 KiB