diff --git a/.roo/rules/01-general.md b/.roo/rules/01-general.md index a2a3fdb..5260f85 100644 --- a/.roo/rules/01-general.md +++ b/.roo/rules/01-general.md @@ -14,6 +14,7 @@ Here are special rules you must follow: 12. Before completing work, ensure that no design docs are left out of sync 13. Plans go into docs/plans/*.md. These may not be kept in sync, as they are just for brainstorming. 14. Database migrations are automatically created via `flask db migrate -m 'message'`. NEVER create migrations by hand. +15. In the technical documentation, code should be used sparingly. Also, when needed, focus on the APIs, and only use snippets. # Conventions diff --git a/README.md b/README.md index 827fac6..495cae8 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,25 @@ # Email Organizer -A self-hosted AI-powered email organization system that automates folder sorting, prioritization, and rule recommendations through natural language configuration. +A self-hosted email organization system that automates folder sorting and management through natural language configuration. ## Core Value Proposition -- **Natural Language Rules**: Define email organization logic in plain English instead of writing sieve/filter rules -- **Smart Separation**: Automatically categorize marketing emails, receipts, and transactional messages -- **Adaptive Intelligence**: Learns your preferences to star important emails and suggest new organizational patterns +- **Natural Language Rules**: Define email organization logic in plain English instead of writing complex filter rules +- **Smart Separation**: Automatically categorize emails into custom folders based on your rules +- **IMAP Integration**: Sync and manage folders directly from your email server +- **User Authentication**: Secure multi-user system with individual configurations ## Technical Overview **Stack**: - Backend: Flask (Python 3.10+) - Frontend: HTMX + AlpineJS + DaisyUI (Tailwind CSS) - Database: PostgreSQL -- AI: OpenAI-compatible API endpoints +- IMAP: Direct server integration **Key Components**: -1. **Rule Engine**: Converts natural language rules → executable classification logic -2. **Email Processor**: Polls IMAP server, applies AI classifications, moves messages -3. **Recommendation System**: Analyzes usage patterns to suggest new folders/rules +1. **Authentication System**: User registration, login, and session management +2. **Folder Management**: Create, edit, and organize email folders with natural language rules +3. **IMAP Service**: Direct integration with email servers for folder synchronization +4. **User Interface**: Modern, responsive UI with dynamic updates ## Getting Started ### Prerequisites @@ -44,41 +46,97 @@ flask db upgrade flask run ``` -## Roadmap -### Milestone 1: Prototype (Current Focus) -- [ ] Core infrastructure setup -- [ ] Basic UI for rule configuration -- [ ] Mock email processing pipeline -- [ ] Database schema implementation +## Features -### Future Milestones -- [ ] MVP - enter your imap server. You list out folders and rules, and once a day emails in the "Pending" folder will be reorganized. -- [ ] "Import" your list of folders from your imap server -- [ ] Polling every 5 minutes, working off recent emails since last poll -- [ ] Label support for services like gmail -- [ ] Generate config to automate integration into imap servers like dovecot or gmail or proton -- [ ] Supporting auth / multi user. Admin controls the list of users, users control their folders. -- [ ] Making a paid, hosted version -- [ ] Auth via google / facebook / etc. -- [ ] Automatically star based off of your habits +### Current Implementation +- **User Authentication**: Complete user registration and login system with password validation +- **Folder Management**: Create, edit, delete, and toggle email organization folders +- **IMAP Integration**: Configure and test connections to email servers +- **Folder Synchronization**: Import existing folders from IMAP servers +- **Natural Language Rules**: Define organization rules in plain English +- **Priority System**: Organize folders with priority levels +- **Email Metrics**: Track email counts and recent email information + +### User Interface +- **Modern Design**: Clean, responsive interface using DaisyUI +- **Dynamic Updates**: HTMX for seamless UI interactions without page reloads +- **Modal System**: Forms and configuration in intuitive modal dialogs +- **Real-time Feedback**: Immediate visual feedback for all operations + +## Architecture +The application follows a modular architecture with clear separation of concerns: + +- **Frontend Layer**: HTMX + AlpineJS + DaisyUI for dynamic user interface +- **Application Layer**: Flask blueprints for modular feature organization +- **Data Layer**: PostgreSQL with SQLAlchemy ORM for data persistence +- **Service Layer**: IMAP service for email server communication + +For detailed architecture information, see [System Architecture](docs/design/system-architecture.md). ## Data Model +The system uses two main entities with a one-to-many relationship: + **Users** | Column | Type | Description | |--------|------|-------------| -| id | UUID | Primary key | -| email | VARCHAR | Unique identifier | -| password_hash | BYTEA | Argon2-hashed | -| imap_config | JSONB | Encrypted server settings | +| id | Integer | Primary key | +| first_name | String | User's first name | +| last_name | String | User's last name | +| email | String | Unique identifier | +| password_hash | String | Hashed password | +| imap_config | JSON | Server settings | +| created_at | DateTime | Account creation timestamp | +| updated_at | DateTime | Last update timestamp | **Folders** | Column | Type | Description | |--------|------|-------------| -| id | UUID | Primary key | -| user_id | UUID | Foreign key | -| name | VARCHAR | Display name | -| rule_text | TEXT | Natural language rule | -| priority | INT | Processing order | +| id | Integer | Primary key | +| user_id | Integer | Foreign key to user | +| name | String | Display name | +| rule_text | Text | Natural language rule | +| priority | Integer | Processing order (0=normal, 1=high) | +| organize_enabled | Boolean | Rule active status | +| total_count | Integer | Total emails in folder | +| pending_count | Integer | Emails to process | +| recent_emails | JSON | Recent email metadata | +| created_at | DateTime | Folder creation timestamp | +| updated_at | DateTime | Last update timestamp | + +For detailed data model information, see [Data Model](docs/design/data-model.md). + +## Development Environment +The development environment includes: +- **PostgreSQL Database**: For data persistence +- **Dovecot IMAP Server**: For testing email integration +- **Docker Compose**: For easy service orchestration + +### Development Services +- **Database**: localhost:5432 (email_organizer_dev) +- **IMAP Server**: localhost:1143 +- **Test Users**: + - user1@example.com / password1 + - user2@example.com / password2 + +## Roadmap +### Current Status +- ✅ User authentication system +- ✅ Folder CRUD operations +- ✅ IMAP configuration and testing +- ✅ Folder synchronization from IMAP servers +- ✅ Natural language rule configuration +- ✅ Priority-based folder organization +- ✅ Email metrics tracking + +### Future Enhancements +- **AI-Powered Rules**: Advanced rule processing and recommendations +- **Email Processing**: Automated email classification and movement +- **Advanced IMAP Features**: Two-factor authentication, OAuth support +- **Label Support**: Integration with Gmail and other providers +- **Admin Interface**: Multi-user management and monitoring +- **Hosted Service**: Cloud-based offering +- **Social Authentication**: Google, Facebook, etc. login options +- **Usage Analytics**: Email processing statistics and insights ## Contributing 1. Fork the repository @@ -87,4 +145,10 @@ flask run 4. Push to the branch (`git push origin feature/AmazingFeature`) 5. Open a Pull Request -> **Note**: All new features require corresponding unit tests and documentation updates. \ No newline at end of file +> **Note**: All new features require corresponding unit tests and documentation updates. + +## Documentation +- [System Architecture](docs/design/system-architecture.md) +- [Data Model](docs/design/data-model.md) +- [IMAP Connectivity](docs/design/imap-connectivity.md) +- [Development Setup](docs/design/development-setup.md) \ No newline at end of file diff --git a/docs/design/imap-connectivity.md b/docs/design/imap-connectivity.md index 7e763a6..8934a63 100644 --- a/docs/design/imap-connectivity.md +++ b/docs/design/imap-connectivity.md @@ -2,7 +2,7 @@ ## Overview -This document outlines the design for implementing IMAP connectivity in the Email Organizer application. The feature will allow users to configure their IMAP server connection details, test the connection, and synchronize their email folders from the IMAP server to the application. +This document outlines the design for implementing IMAP connectivity in the Email Organizer application. The feature allows users to configure their IMAP server connection details, test the connection, and synchronize their email folders from the IMAP server to the application. ## Current State Analysis @@ -67,678 +67,9 @@ sequenceDiagram M->>U: Show sync result ``` -## Database Schema -### User Model Updates -The existing `User` model already has the `imap_config` field: -```python -class User(Base, UserMixin): - # ... existing fields ... - imap_config = db.Column(db.JSON) # Store IMAP connection details - # ... existing fields ... -``` - -### Folder Model Updates - -The `Folder` model now includes a new field for storing recent email information: - -```python -class Folder(Base): - # ... existing fields ... - recent_emails = db.Column(db.JSON, default=list) # Store recent email subjects with dates - # ... existing fields ... -``` - -### IMAP Configuration Structure - -```json -{ - "server": "imap.gmail.com", - "port": 993, - "username": "user@gmail.com", - "password": "app_password", - "use_ssl": true, - "use_tls": false, - "connection_timeout": 30, - "folder_prefix": "" -} -``` - -## IMAP Service Module - -### File: `app/imap_service.py` - -```python -import imaplib -import ssl -import logging -from typing import List, Dict, Optional, Tuple -from email.header import decode_header -from app.models import db, Folder, User -from app import create_app - -class IMAPService: - def __init__(self, user: User): - self.user = user - self.config = user.imap_config or {} - self.connection = None - - def test_connection(self) -> Tuple[bool, str]: - """Test IMAP connection with current configuration.""" - try: - if not self.config: - return False, "No IMAP configuration found" - - # Create SSL context - context = ssl.create_default_context() - - # Connect to IMAP server - self.connection = imaplib.IMAP4_SSL( - self.config.get('server', 'imap.gmail.com'), - self.config.get('port', 993) - ) - - # Login - self.connection.login( - self.config.get('username', ''), - self.config.get('password', '') - ) - - # Select inbox to verify connection - self.connection.select('INBOX') - - # Close connection - self.connection.close() - self.connection.logout() - - return True, "Connection successful" - - except imaplib.IMAP4.error as e: - return False, f"IMAP connection error: {str(e)}" - except Exception as e: - return False, f"Connection error: {str(e)}" - finally: - if self.connection: - try: - self.connection.logout() - except: - pass - - def get_folders(self) -> List[Dict[str, str]]: - """Get list of folders from IMAP server.""" - try: - if not self.config: - return [] - - # Create SSL context - context = ssl.create_default_context() - - # Connect to IMAP server - self.connection = imaplib.IMAP4_SSL( - self.config.get('server', 'imap.gmail.com'), - self.config.get('port', 993) - ) - - # Login - self.connection.login( - self.config.get('username', ''), - self.config.get('password', '') - ) - - # List folders - status, folder_data = self.connection.list() - - if status != 'OK': - return [] - - folders = [] - for folder_item in folder_data: - if isinstance(folder_item, bytes): - folder_item = folder_item.decode('utf-8') - - # Parse folder name (handle different IMAP server formats) - parts = folder_item.split('"') - if len(parts) >= 3: - folder_name = parts[-1] if parts[-1] else parts[-2] - else: - folder_name = folder_item.split()[-1] - - # Handle nested folders (convert to slash notation) - if folder_name.startswith('"') and folder_name.endswith('"'): - folder_name = folder_name[1:-1] - - folders.append({ - 'name': folder_name, - 'full_path': folder_name - }) - - # Close connection - self.connection.close() - self.connection.logout() - - return folders - - except Exception as e: - logging.error(f"Error fetching IMAP folders: {str(e)}") - return [] - finally: - if self.connection: - try: - self.connection.logout() - except: - pass - - def sync_folders(self) -> Tuple[bool, str]: - """Sync IMAP folders with local database.""" - try: - if not self.config: - return False, "No IMAP configuration found" - - # Get folders from IMAP server - imap_folders = self.get_folders() - - if not imap_folders: - return False, "No folders found on IMAP server" - - # Process each folder - synced_count = 0 - for imap_folder in imap_folders: - folder_name = imap_folder['name'] - - # Skip special folders that might not be needed - if folder_name.lower() in ['inbox', 'sent', 'drafts', 'spam', 'trash']: - continue - - # Handle nested folder names (convert slashes to underscores or keep as-is) - # According to requirements, nested folders should be created with slashes in the name - display_name = folder_name - - # Check if folder already exists - existing_folder = Folder.query.filter_by( - user_id=self.user.id, - name=display_name - ).first() - - if not existing_folder: - # Create new folder - new_folder = Folder( - user_id=self.user.id, - name=display_name, - rule_text=f"Auto-synced from IMAP folder: {folder_name}", - priority=0 # Default priority - ) - db.session.add(new_folder) - synced_count += 1 - - db.session.commit() - return True, f"Successfully synced {synced_count} folders" - - except Exception as e: - db.session.rollback() - return False, f"Sync error: {str(e)}" - - def get_recent_emails(self, folder_name: str, limit: int = 3) -> List[Dict[str, any]]: - """Get the most recent email subjects and dates from a specific folder.""" - try: - # Connect to IMAP server - self._connect() - - # Login - self.connection.login( - self.config.get('username', ''), - self.config.get('password', '') - ) - - # Select the folder - resp_code, content = self.connection.select(folder_name) - if resp_code != 'OK': - return [] - - # Get email IDs (most recent first) - resp_code, content = self.connection.search(None, 'ALL') - if resp_code != 'OK': - return [] - - # Get the most recent emails (limit to 3) - email_ids = content[0].split() - recent_email_ids = email_ids[:limit] if len(email_ids) >= limit else email_ids - - recent_emails = [] - for email_id in reversed(recent_email_ids): # Process from newest to oldest - # Fetch the email headers - resp_code, content = self.connection.fetch(email_id, '(RFC822.HEADER)') - if resp_code != 'OK': - continue - - # Parse the email headers - raw_email = content[0][1] - import email - msg = email.message_from_bytes(raw_email) - - # Extract subject and date - subject = msg.get('Subject', 'No Subject') - date_str = msg.get('Date', '') - - # Decode the subject if needed - try: - decoded_parts = decode_header(subject) - subject = ''.join([str(part, encoding or 'utf-8') if isinstance(part, bytes) else part for part, encoding in decoded_parts]) - except Exception: - pass # If decoding fails, use the original subject - - # Parse date if available - try: - email_date = parsedate_to_datetime(date_str) if date_str else None - except Exception: - email_date = None - - recent_emails.append({ - 'subject': subject, - 'date': email_date.isoformat() if email_date else None - }) - - # Close folder and logout - self.connection.close() - self.connection.logout() - self.connection = None - - return recent_emails - - except Exception as e: - logging.error(f"Error getting recent emails for folder {folder_name}: {str(e)}") - if self.connection: - try: - self.connection.logout() - except: - pass - self.connection = None - return [] -``` - -## Routes Implementation - -### File: `app/routes.py` (additions) - -```python -# Add to existing imports -from app.imap_service import IMAPService -import json - -@main.route('/api/imap/config', methods=['GET']) -@login_required -def imap_config_modal(): - """Return the IMAP configuration modal.""" - response = make_response(render_template('partials/imap_config_modal.html')) - response.headers['HX-Trigger'] = 'open-modal' - return response - -@main.route('/api/imap/test', methods=['POST']) -@login_required -def test_imap_connection(): - """Test IMAP connection with provided configuration.""" - try: - # Get form data - server = request.form.get('server') - port = request.form.get('port') - username = request.form.get('username') - password = request.form.get('password') - use_ssl = request.form.get('use_ssl') == 'on' - - # Validate required fields - errors = {} - if not server: - errors['server'] = 'Server is required' - if not port: - errors['port'] = 'Port is required' - elif not port.isdigit(): - errors['port'] = 'Port must be a number' - if not username: - errors['username'] = 'Username is required' - if not password: - errors['password'] = 'Password is required' - - if errors: - response = make_response(render_template('partials/imap_config_modal.html', errors=errors)) - response.headers['HX-Retarget'] = '#imap-modal' - response.headers['HX-Reswap'] = 'outerHTML' - return response - - # Store configuration temporarily for testing - test_config = { - 'server': server, - 'port': int(port), - 'username': username, - 'password': password, - 'use_ssl': use_ssl, - 'use_tls': False, - 'connection_timeout': 30 - } - - # Test connection - temp_user = type('User', (), {'imap_config': test_config})() - imap_service = IMAPService(temp_user) - success, message = imap_service.test_connection() - - if success: - # Save configuration to user's profile - current_user.imap_config = test_config - db.session.commit() - - response = make_response(render_template('partials/imap_config_modal.html', - success=True, message=message)) - response.headers['HX-Retarget'] = '#imap-modal' - response.headers['HX-Reswap'] = 'outerHTML' - else: - response = make_response(render_template('partials/imap_config_modal.html', - errors={'general': message})) - response.headers['HX-Retarget'] = '#imap-modal' - response.headers['HX-Reswap'] = 'outerHTML' - - return response - - except Exception as e: - logging.exception("Error testing IMAP connection: %s", e) - errors = {'general': 'An unexpected error occurred. Please try again.'} - response = make_response(render_template('partials/imap_config_modal.html', errors=errors)) - response.headers['HX-Retarget'] = '#imap-modal' - response.headers['HX-Reswap'] = 'outerHTML' - return response - -@main.route('/api/imap/sync', methods=['POST']) -@login_required -def sync_imap_folders(): - """Sync folders from IMAP server.""" - try: - if not current_user.imap_config: - return jsonify({'error': 'No IMAP configuration found. Please configure IMAP first.'}), 400 - - # Test connection first - imap_service = IMAPService(current_user) - success, message = imap_service.sync_folders() - - if success: - # Get updated list of folders - folders = Folder.query.filter_by(user_id=current_user.id).all() - return render_template('partials/folders_list.html', folders=folders) - else: - return jsonify({'error': message}), 400 - - except Exception as e: - logging.exception("Error syncing IMAP folders: %s", e) - return jsonify({'error': 'An unexpected error occurred. Please try again.'}), 500 -``` - -## UI Implementation - -### Template: `app/templates/partials/imap_config_modal.html` - -```html - -``` - -### Template Update: `app/templates/partials/sidebar.html` - -Add IMAP configuration option to the sidebar: - -```html -
- - - -
-
- - - Logout - -
-
-
-``` - -## Testing Strategy - -### Unit Tests - -```python -# tests/test_imap_service.py -import pytest -from app.imap_service import IMAPService -from app.models import User, db -from unittest.mock import Mock, patch - -class TestIMAPService: - def test_init_with_user(self, app): - with app.app_context(): - user = User(email='test@example.com', first_name='Test', last_name='User') - user.imap_config = {'server': 'test.com', 'port': 993} - imap_service = IMAPService(user) - assert imap_service.user == user - assert imap_service.config == {'server': 'test.com', 'port': 993} - - @patch('app.imap_service.imaplib.IMAP4_SSL') - def test_test_connection_success(self, mock_imap, app): - with app.app_context(): - # Mock successful connection - mock_connection = Mock() - mock_imap.return_value = mock_connection - mock_connection.login.return_value = None - mock_connection.select.return_value = ('OK', [b'']) - mock_connection.close.return_value = None - mock_connection.logout.return_value = None - - user = User(email='test@example.com', first_name='Test', last_name='User') - user.imap_config = {'server': 'test.com', 'port': 993, 'username': 'user', 'password': 'pass'} - imap_service = IMAPService(user) - - success, message = imap_service.test_connection() - - assert success is True - assert message == "Connection successful" - mock_imap.assert_called_once_with('test.com', 993) - mock_connection.login.assert_called_once_with('user', 'pass') - - @patch('app.imap_service.imaplib.IMAP4_SSL') - def test_test_connection_failure(self, mock_imap, app): - with app.app_context(): - # Mock failed connection - mock_imap.side_effect = Exception("Connection failed") - - user = User(email='test@example.com', first_name='Test', last_name='User') - user.imap_config = {'server': 'test.com', 'port': 993, 'username': 'user', 'password': 'pass'} - imap_service = IMAPService(user) - - success, message = imap_service.test_connection() - - assert success is False - assert "Connection error" in message -``` - -### Integration Tests - -```python -# tests/test_imap_routes.py -import pytest -from app import create_app -from app.models import User, db, Folder - -class TestIMAPRoutes: - def test_imap_config_modal(self, client): - response = client.get('/api/imap/config') - assert response.status_code == 200 - assert b'Configure IMAP Connection' in response.data - - def test_imap_connection_test_success(self, client, app): - with app.app_context(): - user = User(email='test@example.com', first_name='Test', last_name='User') - user.set_password('password') - db.session.add(user) - db.session.commit() - - client.post('/auth/login', data={ - 'email': 'test@example.com', - 'password': 'password' - }) - - response = client.post('/api/imap/test', data={ - 'server': 'test.com', - 'port': '993', - 'username': 'test@test.com', - 'password': 'testpass', - 'use_ssl': 'on' - }) - - assert response.status_code == 200 - assert b'Test Connection' in response.data - - def test_imap_sync_folders(self, client, app): - with app.app_context(): - user = User(email='test@example.com', first_name='Test', last_name='User') - user.set_password('password') - user.imap_config = {'server': 'test.com', 'port': 993, 'username': 'test', 'password': 'pass'} - db.session.add(user) - db.session.commit() - - client.post('/auth/login', data={ - 'email': 'test@example.com', - 'password': 'password' - }) - - response = client.post('/api/imap/sync') - assert response.status_code == 400 # Should fail without real IMAP server -``` - -## Implementation Dependencies - -### New Dependencies - -Add to `requirements.txt`: -``` -imapclient==3.0.1 -``` - -### Database Migration - -No database migration is required as the existing `imap_config` field in the `User` model is sufficient. ## Security Considerations @@ -771,58 +102,7 @@ No database migration is required as the existing `imap_config` field in the `Us ## Success Metrics 1. **Functional**: Users can configure IMAP connections, test them, and sync folders successfully. - 2. **Usability**: The UI is intuitive and provides clear feedback for all operations. - 3. **Reliability**: IMAP connections are stable and handle various server configurations. - 4. **Performance**: Folder synchronization completes in reasonable time. - -5. **Security**: User credentials are handled securely. - -## Acceptance Criteria - -Based on the original requirements: - -1. ✅ **Verify that a user can enter their imap server's connection details, and that they can be saved** - - Users can access the IMAP configuration modal - - Form validation ensures all required fields are filled - - Configuration is saved to the user's `imap_config` field - -2. ✅ **Verify that, once the connection is made, the list of folders are created for that user, copying the name and structure from IMAP** - - Connection testing validates the IMAP server access - - Folder synchronization fetches all available folders - - Folders are created in the local database with appropriate names - -3. ✅ **Verify that nested folders are just created with slashes in the name** - - Nested folder names from IMAP servers are preserved with slash notation - - Display names maintain the hierarchical structure - -4. ✅ **Verify that a failed connection attempt gives the user a validation error message and allows them to retry** - - Connection failures show clear error messages - - Users can modify their configuration and retry the connection - - Form validation prevents submission with invalid data - -## Implementation Plan - -### Phase 1: Core IMAP Service -1. Create `app/imap_service.py` with basic IMAP connection functionality -2. Implement connection testing logic -3. Add folder listing and synchronization - -### Phase 2: UI Integration -1. Create IMAP configuration modal template -2. Add routes for IMAP configuration and testing -3. Update sidebar to include IMAP settings option - -### Phase 3: Testing and Validation -1. Write unit tests for IMAP service -2. Write integration tests for IMAP routes -3. Test with various IMAP server configurations - -### Phase 4: Error Handling and Polish -1. Add comprehensive error handling -2. Improve user feedback and validation messages -3. Optimize performance and reliability - -This design provides a comprehensive approach to implementing IMAP connectivity while maintaining the existing application architecture and following the established patterns. \ No newline at end of file +5. **Security**: User credentials are handled securely. \ No newline at end of file