diff --git a/.roo/rules/01-general.md b/.roo/rules/01-general.md index a2a3fdb..5260f85 100644 --- a/.roo/rules/01-general.md +++ b/.roo/rules/01-general.md @@ -14,6 +14,7 @@ Here are special rules you must follow: 12. Before completing work, ensure that no design docs are left out of sync 13. Plans go into docs/plans/*.md. These may not be kept in sync, as they are just for brainstorming. 14. Database migrations are automatically created via `flask db migrate -m 'message'`. NEVER create migrations by hand. +15. In the technical documentation, code should be used sparingly. Also, when needed, focus on the APIs, and only use snippets. # Conventions diff --git a/README.md b/README.md index 827fac6..495cae8 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,25 @@ # Email Organizer -A self-hosted AI-powered email organization system that automates folder sorting, prioritization, and rule recommendations through natural language configuration. +A self-hosted email organization system that automates folder sorting and management through natural language configuration. ## Core Value Proposition -- **Natural Language Rules**: Define email organization logic in plain English instead of writing sieve/filter rules -- **Smart Separation**: Automatically categorize marketing emails, receipts, and transactional messages -- **Adaptive Intelligence**: Learns your preferences to star important emails and suggest new organizational patterns +- **Natural Language Rules**: Define email organization logic in plain English instead of writing complex filter rules +- **Smart Separation**: Automatically categorize emails into custom folders based on your rules +- **IMAP Integration**: Sync and manage folders directly from your email server +- **User Authentication**: Secure multi-user system with individual configurations ## Technical Overview **Stack**: - Backend: Flask (Python 3.10+) - Frontend: HTMX + AlpineJS + DaisyUI (Tailwind CSS) - Database: PostgreSQL -- AI: OpenAI-compatible API endpoints +- IMAP: Direct server integration **Key Components**: -1. **Rule Engine**: Converts natural language rules → executable classification logic -2. **Email Processor**: Polls IMAP server, applies AI classifications, moves messages -3. **Recommendation System**: Analyzes usage patterns to suggest new folders/rules +1. **Authentication System**: User registration, login, and session management +2. **Folder Management**: Create, edit, and organize email folders with natural language rules +3. **IMAP Service**: Direct integration with email servers for folder synchronization +4. **User Interface**: Modern, responsive UI with dynamic updates ## Getting Started ### Prerequisites @@ -44,41 +46,97 @@ flask db upgrade flask run ``` -## Roadmap -### Milestone 1: Prototype (Current Focus) -- [ ] Core infrastructure setup -- [ ] Basic UI for rule configuration -- [ ] Mock email processing pipeline -- [ ] Database schema implementation +## Features -### Future Milestones -- [ ] MVP - enter your imap server. You list out folders and rules, and once a day emails in the "Pending" folder will be reorganized. -- [ ] "Import" your list of folders from your imap server -- [ ] Polling every 5 minutes, working off recent emails since last poll -- [ ] Label support for services like gmail -- [ ] Generate config to automate integration into imap servers like dovecot or gmail or proton -- [ ] Supporting auth / multi user. Admin controls the list of users, users control their folders. -- [ ] Making a paid, hosted version -- [ ] Auth via google / facebook / etc. -- [ ] Automatically star based off of your habits +### Current Implementation +- **User Authentication**: Complete user registration and login system with password validation +- **Folder Management**: Create, edit, delete, and toggle email organization folders +- **IMAP Integration**: Configure and test connections to email servers +- **Folder Synchronization**: Import existing folders from IMAP servers +- **Natural Language Rules**: Define organization rules in plain English +- **Priority System**: Organize folders with priority levels +- **Email Metrics**: Track email counts and recent email information + +### User Interface +- **Modern Design**: Clean, responsive interface using DaisyUI +- **Dynamic Updates**: HTMX for seamless UI interactions without page reloads +- **Modal System**: Forms and configuration in intuitive modal dialogs +- **Real-time Feedback**: Immediate visual feedback for all operations + +## Architecture +The application follows a modular architecture with clear separation of concerns: + +- **Frontend Layer**: HTMX + AlpineJS + DaisyUI for dynamic user interface +- **Application Layer**: Flask blueprints for modular feature organization +- **Data Layer**: PostgreSQL with SQLAlchemy ORM for data persistence +- **Service Layer**: IMAP service for email server communication + +For detailed architecture information, see [System Architecture](docs/design/system-architecture.md). ## Data Model +The system uses two main entities with a one-to-many relationship: + **Users** | Column | Type | Description | |--------|------|-------------| -| id | UUID | Primary key | -| email | VARCHAR | Unique identifier | -| password_hash | BYTEA | Argon2-hashed | -| imap_config | JSONB | Encrypted server settings | +| id | Integer | Primary key | +| first_name | String | User's first name | +| last_name | String | User's last name | +| email | String | Unique identifier | +| password_hash | String | Hashed password | +| imap_config | JSON | Server settings | +| created_at | DateTime | Account creation timestamp | +| updated_at | DateTime | Last update timestamp | **Folders** | Column | Type | Description | |--------|------|-------------| -| id | UUID | Primary key | -| user_id | UUID | Foreign key | -| name | VARCHAR | Display name | -| rule_text | TEXT | Natural language rule | -| priority | INT | Processing order | +| id | Integer | Primary key | +| user_id | Integer | Foreign key to user | +| name | String | Display name | +| rule_text | Text | Natural language rule | +| priority | Integer | Processing order (0=normal, 1=high) | +| organize_enabled | Boolean | Rule active status | +| total_count | Integer | Total emails in folder | +| pending_count | Integer | Emails to process | +| recent_emails | JSON | Recent email metadata | +| created_at | DateTime | Folder creation timestamp | +| updated_at | DateTime | Last update timestamp | + +For detailed data model information, see [Data Model](docs/design/data-model.md). + +## Development Environment +The development environment includes: +- **PostgreSQL Database**: For data persistence +- **Dovecot IMAP Server**: For testing email integration +- **Docker Compose**: For easy service orchestration + +### Development Services +- **Database**: localhost:5432 (email_organizer_dev) +- **IMAP Server**: localhost:1143 +- **Test Users**: + - user1@example.com / password1 + - user2@example.com / password2 + +## Roadmap +### Current Status +- ✅ User authentication system +- ✅ Folder CRUD operations +- ✅ IMAP configuration and testing +- ✅ Folder synchronization from IMAP servers +- ✅ Natural language rule configuration +- ✅ Priority-based folder organization +- ✅ Email metrics tracking + +### Future Enhancements +- **AI-Powered Rules**: Advanced rule processing and recommendations +- **Email Processing**: Automated email classification and movement +- **Advanced IMAP Features**: Two-factor authentication, OAuth support +- **Label Support**: Integration with Gmail and other providers +- **Admin Interface**: Multi-user management and monitoring +- **Hosted Service**: Cloud-based offering +- **Social Authentication**: Google, Facebook, etc. login options +- **Usage Analytics**: Email processing statistics and insights ## Contributing 1. Fork the repository @@ -87,4 +145,10 @@ flask run 4. Push to the branch (`git push origin feature/AmazingFeature`) 5. Open a Pull Request -> **Note**: All new features require corresponding unit tests and documentation updates. \ No newline at end of file +> **Note**: All new features require corresponding unit tests and documentation updates. + +## Documentation +- [System Architecture](docs/design/system-architecture.md) +- [Data Model](docs/design/data-model.md) +- [IMAP Connectivity](docs/design/imap-connectivity.md) +- [Development Setup](docs/design/development-setup.md) \ No newline at end of file diff --git a/docs/design/authentication-system.md b/docs/design/authentication-system.md new file mode 100644 index 0000000..04ff394 --- /dev/null +++ b/docs/design/authentication-system.md @@ -0,0 +1,353 @@ +# Authentication System Design + +## Overview + +The Email Organizer implements a complete user authentication system using Flask-Login for session management and Werkzeug for password security. This document provides a detailed overview of the authentication architecture, components, and implementation details. + +## System Architecture + +```mermaid +graph TB + subgraph "Authentication Flow" + A[User Browser] --> B[Login Page] + B --> C[Submit Credentials] + C --> D[Validate Input] + D --> E[Check Password Hash] + E --> F[Create Session] + F --> G[Redirect to Dashboard] + end + + subgraph "Backend Components" + H[Auth Blueprint] --> I[Validation Logic] + H --> J[Password Hashing] + H --> K[Session Management] + I --> L[User Model] + J --> L + K --> M[Flask-Login] + end + + subgraph "Security Measures" + N[Password Validation] --> O[Complexity Requirements] + N --> P[Hashing Algorithm] + Q[Session Security] --> R[Cookie Protection] + Q --> S[Timeout Handling] + end + + C --> D + D --> I + I --> J + J --> K + K --> F +``` + +## Components + +### 1. Auth Blueprint ([`app/auth.py`](app/auth.py:1)) + +The authentication blueprint handles all authentication-related routes and logic: + +#### Routes +- `/login`: GET/POST - User login page and authentication +- `/signup`: GET/POST - User registration page and account creation +- `/logout`: GET - User logout and session termination + +#### Key Functions +- `login()`: Handles user authentication +- `signup()`: Manages user registration +- `logout()`: Terminates user sessions +- `validate_password()`: Enforces password complexity requirements + +### 2. User Model ([`app/models.py`](app/models.py:13)) + +The User model extends Flask-Login's UserMixin to provide authentication functionality: + +#### Authentication Methods +- `set_password()`: Hashes and stores passwords +- `check_password()`: Verifies password against hash + +#### Security Attributes +- Password hashing using Werkzeug's secure hashing algorithm +- Email uniqueness constraint +- Timestamp tracking for account management + +### 3. Flask-Login Integration + +The application uses Flask-Login for session management: + +#### Configuration +```python +login_manager = LoginManager() +login_manager.login_view = 'auth.login' +login_manager.login_message = 'Please log in to access this page.' +login_manager.login_message_category = 'warning' +``` + +#### User Loader +```python +@login_manager.user_loader +def load_user(user_id): + return User.query.get(int(user_id)) +``` + +## Authentication Flow + +### User Registration + +```mermaid +sequenceDiagram + participant U as User + participant B as Browser + participant A as Auth Blueprint + participant DB as Database + + U->>B: Navigate to /auth/signup + B->>A: GET /auth/signup + A->>B: Return signup form + U->>B: Fill and submit form + B->>A: POST /auth/signup + + loop Validation + A->>A: Validate input fields + A->>A: Check email uniqueness + A->>A: Validate password complexity + end + + A->>DB: Create new user record + DB-->>A: User created + A->>A: Hash and store password + A->>B: Login user + B->>U: Redirect to dashboard +``` + +### User Login + +```mermaid +sequenceDiagram + participant U as User + participant B as Browser + participant A as Auth Blueprint + participant DB as Database + + U->>B: Navigate to /auth/login + B->>A: GET /auth/login + A->>B: Return login form + U->>B: Submit credentials + B->>A: POST /auth/login + + A->>DB: Query user by email + DB-->>A: User data + + alt User exists + A->>A: Verify password hash + alt Password correct + A->>A: Create session + A->>B: Redirect to dashboard + else Password incorrect + A->>B: Show error message + end + else User not found + A->>B: Show error message + end +``` + +### Password Validation + +The system enforces strong password requirements: + +```python +def validate_password(password): + """Validate password strength.""" + if len(password) < 8: + return False, "Password must be at least 8 characters long" + if not re.search(r'[A-Z]', password): + return False, "Password must contain at least one uppercase letter" + if not re.search(r'[a-z]', password): + return False, "Password must contain at least one lowercase letter" + if not re.search(r'\d', password): + return False, "Password must contain at least one digit" + return True, "Password is valid" +``` + +### Session Management + +Flask-Login handles session security: + +- **Session Cookies**: Secure, HttpOnly cookies for session storage +- **CSRF Protection**: Built-in CSRF protection for form submissions +- **Session Timeout**: Automatic session expiration +- **Remember Me**: Optional persistent login functionality + +## Security Measures + +### 1. Password Security + +#### Hashing Algorithm +- Uses Werkzeug's `generate_password_hash()` and `check_password_hash()` +- Implements PBKDF2 with SHA256 for secure password storage +- Random salt generation for each password + +#### Password Complexity +- Minimum 8 characters +- At least one uppercase letter +- At least one lowercase letter +- At least one digit +- No maximum length limit + +### 2. Input Validation + +#### Client-Side Validation +- HTML5 form validation +- JavaScript feedback for user experience + +#### Server-Side Validation +- Comprehensive input sanitization +- Email format validation +- Length restrictions for all fields +- SQL injection prevention through SQLAlchemy ORM + +### 3. Session Security + +#### Cookie Protection +- Secure flag for HTTPS environments +- HttpOnly flag to prevent JavaScript access +- SameSite policy for cross-site request protection + +#### Session Management +- Automatic session regeneration on login +- Session timeout handling +- Logout cleanup + +### 4. Error Handling + +#### User-Friendly Messages +- Clear validation error messages +- General error messages for security-sensitive operations +- No exposure of internal system details + +#### Logging +- Authentication attempt logging +- Security event tracking +- Error debugging information + +## API Endpoints + +### Authentication Endpoints + +| Endpoint | Method | Description | Authentication Required | +|----------|--------|-------------|---------------------------| +| `/auth/login` | GET/POST | User login | No | +| `/auth/signup` | GET/POST | User registration | No | +| `/auth/logout` | GET | User logout | Yes | + +### Response Formats + +#### Login Success +```http +HTTP/1.1 302 Found +Location: / +Set-Cookie: session=; HttpOnly; Secure; Path=/ +``` + +#### Login Failure +```http +HTTP/1.1 200 OK +Content-Type: text/html + + +``` + +#### Registration Success +```http +HTTP/1.1 302 Found +Location: / +Set-Cookie: session=; HttpOnly; Secure; Path=/ +``` + +#### Registration Failure +```http +HTTP/1.1 200 OK +Content-Type: text/html + + +``` + +## Configuration + +### Environment Variables + +| Variable | Description | Default Value | +|----------|-------------|--------------| +| `SECRET_KEY` | Flask secret key for session encryption | `dev-secret-key` | + +### Flask-Login Configuration + +```python +login_manager = LoginManager() +login_manager.login_view = 'auth.login' +login_manager.login_message = 'Please log in to access this page.' +login_manager.login_message_category = 'warning' +``` + +## Testing Strategy + +### Unit Tests + +#### Authentication Tests +- Test password validation logic +- Test password hashing and verification +- Test user creation and validation +- Test session creation and management + +#### Integration Tests +- Test login flow with valid credentials +- Test login flow with invalid credentials +- Test registration flow with valid data +- Test registration flow with invalid data +- Test session persistence and timeout + +### Security Tests + +- Test SQL injection attempts +- Test XSS vulnerabilities +- Test session hijacking prevention +- Test CSRF protection + +## Performance Considerations + +### Database Optimization +- Index on email column for fast login lookups +- Efficient password hashing with proper salting +- Session data stored in server-side session store + +### Security vs. Usability Balance +- Reasonable password complexity requirements +- Clear error messages for failed login attempts +- Session timeout balanced with user convenience + +## Future Enhancements + +### 1. Multi-Factor Authentication +- SMS-based 2FA +- TOTP (Time-based One-Time Password) support +- Hardware token integration + +### 2. OAuth Integration +- Google OAuth +- Facebook OAuth +- GitHub OAuth + +### 3. Password Reset +- Email-based password reset +- Secure token generation +- Expiration handling + +### 4. Account Management +- User profile management +- Email address changes +- Password change functionality + +### 5. Security Enhancements +- Rate limiting for login attempts +- Account lockout after failed attempts +- Suspicious activity monitoring +- IP-based security checks \ No newline at end of file diff --git a/docs/design/data-model.md b/docs/design/data-model.md new file mode 100644 index 0000000..7a2c319 --- /dev/null +++ b/docs/design/data-model.md @@ -0,0 +1,240 @@ +# Email Organizer Data Model + +## Overview + +This document describes the data model for the Email Organizer application, including entities, attributes, relationships, and constraints. The system uses PostgreSQL with SQLAlchemy ORM for data persistence. + +## Entity Relationship Diagram + +```mermaid +erDiagram + USER { + int id PK "Primary Key" + string first_name "Not Null" + string last_name "Not Null" + string email "Unique, Not Null" + string password_hash "Not Null" + json imap_config "JSON Configuration" + datetime created_at "Default: UTC Now" + datetime updated_at "Default: UTC Now, On Update" + } + + FOLDER { + int id PK "Primary Key" + int user_id FK "Foreign Key to User" + string name "Not Null" + text rule_text "Natural Language Rule" + int priority "Processing Order" + boolean organize_enabled "Default: True" + int total_count "Default: 0" + int pending_count "Default: 0" + json recent_emails "JSON Array" + datetime created_at "Default: UTC Now" + datetime updated_at "Default: UTC Now, On Update" + } + + USER ||--o{ FOLDER : "has" + + note "User-Folder Relationship" + note "One-to-Many: Each user can have multiple folders" +``` + +## Entities + +### User Entity + +The `User` entity stores account information and authentication data for each user. + +#### Attributes + +| Column Name | Data Type | Constraints | Description | +|-------------|------------|--------------|-------------| +| id | Integer | Primary Key, Autoincrement | Unique identifier for each user | +| first_name | String(255) | Not Null | User's first name | +| last_name | String(255) | Not Null | User's last name | +| email | String(255) | Unique, Not Null | User's email address (login identifier) | +| password_hash | String(2048) | Not Null | Hashed password for authentication | +| imap_config | JSON | Nullable | IMAP server configuration settings | +| created_at | DateTime | Default: datetime.utcnow | Timestamp of account creation | +| updated_at | DateTime | Default: datetime.utcnow, On Update | Timestamp of last update | + +#### Relationships + +- **One-to-Many**: Each `User` can have multiple `Folder` instances +- **Self-referencing**: No direct relationships to other User instances + +#### Business Rules + +- Email must be unique across all users +- Password is stored as a hash, never in plain text +- IMAP configuration is stored as JSON for flexibility + +### Folder Entity + +The `Folder` entity stores email organization rules and metadata for each user's email folders. + +#### Attributes + +| Column Name | Data Type | Constraints | Description | +|-------------|------------|--------------|-------------| +| id | Integer | Primary Key, Autoincrement | Unique identifier for each folder | +| user_id | Integer | Foreign Key to User, Not Null | Reference to the owning user | +| name | String(255) | Not Null | Display name of the folder | +| rule_text | Text | Nullable | Natural language description of the folder rule | +| priority | Integer | Nullable | Processing order (0=normal, 1=high) | +| organize_enabled | Boolean | Default: True | Whether the organization rule is active | +| total_count | Integer | Default: 0 | Total number of emails in the folder | +| pending_count | Integer | Default: 0 | Number of emails waiting to be processed | +| recent_emails | JSON | Default: [] | Array of recent email metadata | +| created_at | DateTime | Default: datetime.utcnow | Timestamp of folder creation | +| updated_at | DateTime | Default: datetime.utcnow, On Update | Timestamp of last update | + +#### Relationships + +- **Many-to-One**: Each `Folder` belongs to one `User` +- **Self-referencing**: No direct relationships to other Folder instances + +#### Business Rules + +- Each folder must belong to a user +- Folder name must be unique per user +- Rule text can be null (for manually created folders) +- Priority values: 0 (normal), 1 (high priority) +- Recent emails array stores JSON objects with subject and date information + +## Data Constraints + +### Primary Keys + +- `User.id`: Integer, auto-incrementing +- `Folder.id`: Integer, auto-incrementing + +### Foreign Keys + +- `Folder.user_id`: References `User.id` with ON DELETE CASCADE + +### Unique Constraints + +- `User.email`: Ensures no duplicate email addresses +- Composite unique constraint on `(User.id, Folder.name)` to prevent duplicate folder names per user + +### Not Null Constraints + +- `User.first_name`, `User.last_name`, `User.email`, `User.password_hash` +- `Folder.user_id`, `Folder.name` + +### Default Values + +- `User.created_at`, `User.updated_at`: Current UTC timestamp +- `Folder.created_at`, `Folder.updated_at`: Current UTC timestamp +- `Folder.organize_enabled`: True +- `Folder.total_count`, `Folder.pending_count`: 0 +- `Folder.recent_emails`: Empty array + +## JSON Data Structures + +### IMAP Configuration + +The `imap_config` field stores JSON with the following structure: + +```json +{ + "server": "imap.gmail.com", + "port": 993, + "username": "user@example.com", + "password": "app-specific-password", + "use_ssl": true, + "use_tls": false, + "connection_timeout": 30 +} +``` + +### Recent Emails + +The `recent_emails` field stores an array of JSON objects: + +```json +[ + { + "subject": "Order Confirmation", + "date": "2023-11-15T10:30:00Z" + }, + { + "subject": "Meeting Reminder", + "date": "2023-11-14T14:45:00Z" + } +] +``` + +## Database Indexes + +### Current Indexes + +- Primary key indexes on `User.id` and `Folder.id` +- Foreign key index on `Folder.user_id` +- Index on `User.email` for faster login lookups + +### Recommended Indexes + +- Composite index on `(user_id, name)` for folder uniqueness checks +- Index on `Folder.priority` for filtering by priority +- Index on `Folder.organize_enabled` for active/inactive filtering + +## Data Migration History + +### Migration Files + +1. **Initial Migration** ([`migrations/versions/02a7c13515a4_initial.py`](migrations/versions/02a7c13515a4_initial.py:1)) + - Created basic User and Folder tables + - Established primary keys and foreign keys + +2. **Add Name Fields** ([`migrations/versions/28e8e0be0355_add_first_name_last_name_and_timestamp_.py`](migrations/versions/28e8e0be0355_add_first_name_last_name_and_timestamp_.py:1)) + - Added first_name and last_name columns to User table + - Added created_at and updated_at timestamps + +3. **Add Email Count Fields** ([`migrations/versions/a3ad1b9a0e5f_add_email_count_fields_to_folders.py`](migrations/versions/a3ad1b9a0e5f_add_email_count_fields_to_folders.py:1)) + - Added total_count and pending_count columns to Folder table + - Added organize_enabled boolean flag + +4. **Add Recent Emails Field** ([`migrations/versions/9a88c7e94083_add_recent_emails_field_to_folders_table.py`](migrations/versions/9a88c7e94083_add_recent_emails_field_to_folders_table.py:1)) + - Added recent_emails JSON column to Folder table + - Default value set to empty array + +5. **Add Toggle Feature** ([`migrations/versions/f8ba65458ba2_adding_toggle.py`](migrations/versions/f8ba65458ba2_adding_toggle.py:1)) + - Added organize_enabled toggle functionality + - Enhanced folder management features + + +### Performance Considerations + +1. **User Authentication** + - Index on email column for fast login lookups + - Password hash comparison is done in application code + +2. **Folder Operations** + - Foreign key index on user_id for efficient filtering + - Consider pagination for users with many folders + +3. **IMAP Sync Operations** + - Batch updates for email counts + - JSON operations for recent emails metadata + +## Future Data Model Considerations + +### Potential Enhancements + +1. **Email Entity** + - Store email metadata for better analytics + - Track email movement between folders + +2. **Rule Engine** + - Store parsed rule structures for better processing + - Version control for rule changes + +3. **User Preferences** + - Additional customization options + - UI preference storage + +4. **Audit Log** + - Track changes to user data + - Monitor folder operations \ No newline at end of file diff --git a/docs/design/imap-connectivity.md b/docs/design/imap-connectivity.md index 7e763a6..8934a63 100644 --- a/docs/design/imap-connectivity.md +++ b/docs/design/imap-connectivity.md @@ -2,7 +2,7 @@ ## Overview -This document outlines the design for implementing IMAP connectivity in the Email Organizer application. The feature will allow users to configure their IMAP server connection details, test the connection, and synchronize their email folders from the IMAP server to the application. +This document outlines the design for implementing IMAP connectivity in the Email Organizer application. The feature allows users to configure their IMAP server connection details, test the connection, and synchronize their email folders from the IMAP server to the application. ## Current State Analysis @@ -67,678 +67,9 @@ sequenceDiagram M->>U: Show sync result ``` -## Database Schema -### User Model Updates -The existing `User` model already has the `imap_config` field: -```python -class User(Base, UserMixin): - # ... existing fields ... - imap_config = db.Column(db.JSON) # Store IMAP connection details - # ... existing fields ... -``` - -### Folder Model Updates - -The `Folder` model now includes a new field for storing recent email information: - -```python -class Folder(Base): - # ... existing fields ... - recent_emails = db.Column(db.JSON, default=list) # Store recent email subjects with dates - # ... existing fields ... -``` - -### IMAP Configuration Structure - -```json -{ - "server": "imap.gmail.com", - "port": 993, - "username": "user@gmail.com", - "password": "app_password", - "use_ssl": true, - "use_tls": false, - "connection_timeout": 30, - "folder_prefix": "" -} -``` - -## IMAP Service Module - -### File: `app/imap_service.py` - -```python -import imaplib -import ssl -import logging -from typing import List, Dict, Optional, Tuple -from email.header import decode_header -from app.models import db, Folder, User -from app import create_app - -class IMAPService: - def __init__(self, user: User): - self.user = user - self.config = user.imap_config or {} - self.connection = None - - def test_connection(self) -> Tuple[bool, str]: - """Test IMAP connection with current configuration.""" - try: - if not self.config: - return False, "No IMAP configuration found" - - # Create SSL context - context = ssl.create_default_context() - - # Connect to IMAP server - self.connection = imaplib.IMAP4_SSL( - self.config.get('server', 'imap.gmail.com'), - self.config.get('port', 993) - ) - - # Login - self.connection.login( - self.config.get('username', ''), - self.config.get('password', '') - ) - - # Select inbox to verify connection - self.connection.select('INBOX') - - # Close connection - self.connection.close() - self.connection.logout() - - return True, "Connection successful" - - except imaplib.IMAP4.error as e: - return False, f"IMAP connection error: {str(e)}" - except Exception as e: - return False, f"Connection error: {str(e)}" - finally: - if self.connection: - try: - self.connection.logout() - except: - pass - - def get_folders(self) -> List[Dict[str, str]]: - """Get list of folders from IMAP server.""" - try: - if not self.config: - return [] - - # Create SSL context - context = ssl.create_default_context() - - # Connect to IMAP server - self.connection = imaplib.IMAP4_SSL( - self.config.get('server', 'imap.gmail.com'), - self.config.get('port', 993) - ) - - # Login - self.connection.login( - self.config.get('username', ''), - self.config.get('password', '') - ) - - # List folders - status, folder_data = self.connection.list() - - if status != 'OK': - return [] - - folders = [] - for folder_item in folder_data: - if isinstance(folder_item, bytes): - folder_item = folder_item.decode('utf-8') - - # Parse folder name (handle different IMAP server formats) - parts = folder_item.split('"') - if len(parts) >= 3: - folder_name = parts[-1] if parts[-1] else parts[-2] - else: - folder_name = folder_item.split()[-1] - - # Handle nested folders (convert to slash notation) - if folder_name.startswith('"') and folder_name.endswith('"'): - folder_name = folder_name[1:-1] - - folders.append({ - 'name': folder_name, - 'full_path': folder_name - }) - - # Close connection - self.connection.close() - self.connection.logout() - - return folders - - except Exception as e: - logging.error(f"Error fetching IMAP folders: {str(e)}") - return [] - finally: - if self.connection: - try: - self.connection.logout() - except: - pass - - def sync_folders(self) -> Tuple[bool, str]: - """Sync IMAP folders with local database.""" - try: - if not self.config: - return False, "No IMAP configuration found" - - # Get folders from IMAP server - imap_folders = self.get_folders() - - if not imap_folders: - return False, "No folders found on IMAP server" - - # Process each folder - synced_count = 0 - for imap_folder in imap_folders: - folder_name = imap_folder['name'] - - # Skip special folders that might not be needed - if folder_name.lower() in ['inbox', 'sent', 'drafts', 'spam', 'trash']: - continue - - # Handle nested folder names (convert slashes to underscores or keep as-is) - # According to requirements, nested folders should be created with slashes in the name - display_name = folder_name - - # Check if folder already exists - existing_folder = Folder.query.filter_by( - user_id=self.user.id, - name=display_name - ).first() - - if not existing_folder: - # Create new folder - new_folder = Folder( - user_id=self.user.id, - name=display_name, - rule_text=f"Auto-synced from IMAP folder: {folder_name}", - priority=0 # Default priority - ) - db.session.add(new_folder) - synced_count += 1 - - db.session.commit() - return True, f"Successfully synced {synced_count} folders" - - except Exception as e: - db.session.rollback() - return False, f"Sync error: {str(e)}" - - def get_recent_emails(self, folder_name: str, limit: int = 3) -> List[Dict[str, any]]: - """Get the most recent email subjects and dates from a specific folder.""" - try: - # Connect to IMAP server - self._connect() - - # Login - self.connection.login( - self.config.get('username', ''), - self.config.get('password', '') - ) - - # Select the folder - resp_code, content = self.connection.select(folder_name) - if resp_code != 'OK': - return [] - - # Get email IDs (most recent first) - resp_code, content = self.connection.search(None, 'ALL') - if resp_code != 'OK': - return [] - - # Get the most recent emails (limit to 3) - email_ids = content[0].split() - recent_email_ids = email_ids[:limit] if len(email_ids) >= limit else email_ids - - recent_emails = [] - for email_id in reversed(recent_email_ids): # Process from newest to oldest - # Fetch the email headers - resp_code, content = self.connection.fetch(email_id, '(RFC822.HEADER)') - if resp_code != 'OK': - continue - - # Parse the email headers - raw_email = content[0][1] - import email - msg = email.message_from_bytes(raw_email) - - # Extract subject and date - subject = msg.get('Subject', 'No Subject') - date_str = msg.get('Date', '') - - # Decode the subject if needed - try: - decoded_parts = decode_header(subject) - subject = ''.join([str(part, encoding or 'utf-8') if isinstance(part, bytes) else part for part, encoding in decoded_parts]) - except Exception: - pass # If decoding fails, use the original subject - - # Parse date if available - try: - email_date = parsedate_to_datetime(date_str) if date_str else None - except Exception: - email_date = None - - recent_emails.append({ - 'subject': subject, - 'date': email_date.isoformat() if email_date else None - }) - - # Close folder and logout - self.connection.close() - self.connection.logout() - self.connection = None - - return recent_emails - - except Exception as e: - logging.error(f"Error getting recent emails for folder {folder_name}: {str(e)}") - if self.connection: - try: - self.connection.logout() - except: - pass - self.connection = None - return [] -``` - -## Routes Implementation - -### File: `app/routes.py` (additions) - -```python -# Add to existing imports -from app.imap_service import IMAPService -import json - -@main.route('/api/imap/config', methods=['GET']) -@login_required -def imap_config_modal(): - """Return the IMAP configuration modal.""" - response = make_response(render_template('partials/imap_config_modal.html')) - response.headers['HX-Trigger'] = 'open-modal' - return response - -@main.route('/api/imap/test', methods=['POST']) -@login_required -def test_imap_connection(): - """Test IMAP connection with provided configuration.""" - try: - # Get form data - server = request.form.get('server') - port = request.form.get('port') - username = request.form.get('username') - password = request.form.get('password') - use_ssl = request.form.get('use_ssl') == 'on' - - # Validate required fields - errors = {} - if not server: - errors['server'] = 'Server is required' - if not port: - errors['port'] = 'Port is required' - elif not port.isdigit(): - errors['port'] = 'Port must be a number' - if not username: - errors['username'] = 'Username is required' - if not password: - errors['password'] = 'Password is required' - - if errors: - response = make_response(render_template('partials/imap_config_modal.html', errors=errors)) - response.headers['HX-Retarget'] = '#imap-modal' - response.headers['HX-Reswap'] = 'outerHTML' - return response - - # Store configuration temporarily for testing - test_config = { - 'server': server, - 'port': int(port), - 'username': username, - 'password': password, - 'use_ssl': use_ssl, - 'use_tls': False, - 'connection_timeout': 30 - } - - # Test connection - temp_user = type('User', (), {'imap_config': test_config})() - imap_service = IMAPService(temp_user) - success, message = imap_service.test_connection() - - if success: - # Save configuration to user's profile - current_user.imap_config = test_config - db.session.commit() - - response = make_response(render_template('partials/imap_config_modal.html', - success=True, message=message)) - response.headers['HX-Retarget'] = '#imap-modal' - response.headers['HX-Reswap'] = 'outerHTML' - else: - response = make_response(render_template('partials/imap_config_modal.html', - errors={'general': message})) - response.headers['HX-Retarget'] = '#imap-modal' - response.headers['HX-Reswap'] = 'outerHTML' - - return response - - except Exception as e: - logging.exception("Error testing IMAP connection: %s", e) - errors = {'general': 'An unexpected error occurred. Please try again.'} - response = make_response(render_template('partials/imap_config_modal.html', errors=errors)) - response.headers['HX-Retarget'] = '#imap-modal' - response.headers['HX-Reswap'] = 'outerHTML' - return response - -@main.route('/api/imap/sync', methods=['POST']) -@login_required -def sync_imap_folders(): - """Sync folders from IMAP server.""" - try: - if not current_user.imap_config: - return jsonify({'error': 'No IMAP configuration found. Please configure IMAP first.'}), 400 - - # Test connection first - imap_service = IMAPService(current_user) - success, message = imap_service.sync_folders() - - if success: - # Get updated list of folders - folders = Folder.query.filter_by(user_id=current_user.id).all() - return render_template('partials/folders_list.html', folders=folders) - else: - return jsonify({'error': message}), 400 - - except Exception as e: - logging.exception("Error syncing IMAP folders: %s", e) - return jsonify({'error': 'An unexpected error occurred. Please try again.'}), 500 -``` - -## UI Implementation - -### Template: `app/templates/partials/imap_config_modal.html` - -```html - -``` - -### Template Update: `app/templates/partials/sidebar.html` - -Add IMAP configuration option to the sidebar: - -```html -
- - - -
-
- - - Logout - -
-
-
-``` - -## Testing Strategy - -### Unit Tests - -```python -# tests/test_imap_service.py -import pytest -from app.imap_service import IMAPService -from app.models import User, db -from unittest.mock import Mock, patch - -class TestIMAPService: - def test_init_with_user(self, app): - with app.app_context(): - user = User(email='test@example.com', first_name='Test', last_name='User') - user.imap_config = {'server': 'test.com', 'port': 993} - imap_service = IMAPService(user) - assert imap_service.user == user - assert imap_service.config == {'server': 'test.com', 'port': 993} - - @patch('app.imap_service.imaplib.IMAP4_SSL') - def test_test_connection_success(self, mock_imap, app): - with app.app_context(): - # Mock successful connection - mock_connection = Mock() - mock_imap.return_value = mock_connection - mock_connection.login.return_value = None - mock_connection.select.return_value = ('OK', [b'']) - mock_connection.close.return_value = None - mock_connection.logout.return_value = None - - user = User(email='test@example.com', first_name='Test', last_name='User') - user.imap_config = {'server': 'test.com', 'port': 993, 'username': 'user', 'password': 'pass'} - imap_service = IMAPService(user) - - success, message = imap_service.test_connection() - - assert success is True - assert message == "Connection successful" - mock_imap.assert_called_once_with('test.com', 993) - mock_connection.login.assert_called_once_with('user', 'pass') - - @patch('app.imap_service.imaplib.IMAP4_SSL') - def test_test_connection_failure(self, mock_imap, app): - with app.app_context(): - # Mock failed connection - mock_imap.side_effect = Exception("Connection failed") - - user = User(email='test@example.com', first_name='Test', last_name='User') - user.imap_config = {'server': 'test.com', 'port': 993, 'username': 'user', 'password': 'pass'} - imap_service = IMAPService(user) - - success, message = imap_service.test_connection() - - assert success is False - assert "Connection error" in message -``` - -### Integration Tests - -```python -# tests/test_imap_routes.py -import pytest -from app import create_app -from app.models import User, db, Folder - -class TestIMAPRoutes: - def test_imap_config_modal(self, client): - response = client.get('/api/imap/config') - assert response.status_code == 200 - assert b'Configure IMAP Connection' in response.data - - def test_imap_connection_test_success(self, client, app): - with app.app_context(): - user = User(email='test@example.com', first_name='Test', last_name='User') - user.set_password('password') - db.session.add(user) - db.session.commit() - - client.post('/auth/login', data={ - 'email': 'test@example.com', - 'password': 'password' - }) - - response = client.post('/api/imap/test', data={ - 'server': 'test.com', - 'port': '993', - 'username': 'test@test.com', - 'password': 'testpass', - 'use_ssl': 'on' - }) - - assert response.status_code == 200 - assert b'Test Connection' in response.data - - def test_imap_sync_folders(self, client, app): - with app.app_context(): - user = User(email='test@example.com', first_name='Test', last_name='User') - user.set_password('password') - user.imap_config = {'server': 'test.com', 'port': 993, 'username': 'test', 'password': 'pass'} - db.session.add(user) - db.session.commit() - - client.post('/auth/login', data={ - 'email': 'test@example.com', - 'password': 'password' - }) - - response = client.post('/api/imap/sync') - assert response.status_code == 400 # Should fail without real IMAP server -``` - -## Implementation Dependencies - -### New Dependencies - -Add to `requirements.txt`: -``` -imapclient==3.0.1 -``` - -### Database Migration - -No database migration is required as the existing `imap_config` field in the `User` model is sufficient. ## Security Considerations @@ -771,58 +102,7 @@ No database migration is required as the existing `imap_config` field in the `Us ## Success Metrics 1. **Functional**: Users can configure IMAP connections, test them, and sync folders successfully. - 2. **Usability**: The UI is intuitive and provides clear feedback for all operations. - 3. **Reliability**: IMAP connections are stable and handle various server configurations. - 4. **Performance**: Folder synchronization completes in reasonable time. - -5. **Security**: User credentials are handled securely. - -## Acceptance Criteria - -Based on the original requirements: - -1. ✅ **Verify that a user can enter their imap server's connection details, and that they can be saved** - - Users can access the IMAP configuration modal - - Form validation ensures all required fields are filled - - Configuration is saved to the user's `imap_config` field - -2. ✅ **Verify that, once the connection is made, the list of folders are created for that user, copying the name and structure from IMAP** - - Connection testing validates the IMAP server access - - Folder synchronization fetches all available folders - - Folders are created in the local database with appropriate names - -3. ✅ **Verify that nested folders are just created with slashes in the name** - - Nested folder names from IMAP servers are preserved with slash notation - - Display names maintain the hierarchical structure - -4. ✅ **Verify that a failed connection attempt gives the user a validation error message and allows them to retry** - - Connection failures show clear error messages - - Users can modify their configuration and retry the connection - - Form validation prevents submission with invalid data - -## Implementation Plan - -### Phase 1: Core IMAP Service -1. Create `app/imap_service.py` with basic IMAP connection functionality -2. Implement connection testing logic -3. Add folder listing and synchronization - -### Phase 2: UI Integration -1. Create IMAP configuration modal template -2. Add routes for IMAP configuration and testing -3. Update sidebar to include IMAP settings option - -### Phase 3: Testing and Validation -1. Write unit tests for IMAP service -2. Write integration tests for IMAP routes -3. Test with various IMAP server configurations - -### Phase 4: Error Handling and Polish -1. Add comprehensive error handling -2. Improve user feedback and validation messages -3. Optimize performance and reliability - -This design provides a comprehensive approach to implementing IMAP connectivity while maintaining the existing application architecture and following the established patterns. \ No newline at end of file +5. **Security**: User credentials are handled securely. \ No newline at end of file diff --git a/docs/design/system-architecture.md b/docs/design/system-architecture.md new file mode 100644 index 0000000..8455628 --- /dev/null +++ b/docs/design/system-architecture.md @@ -0,0 +1,256 @@ +# Email Organizer System Architecture + +## Overview + +The Email Organizer is a self-hosted AI-powered email organization system that automates folder sorting, prioritization, and rule recommendations through natural language configuration. This document provides a comprehensive overview of the system architecture, components, and data flow. + +## High-Level Architecture + +```mermaid +graph TB + subgraph "Frontend Layer" + A[Browser] --> B[Base Template] + B --> C[HTMX/AlpineJS/DaisyUI] + C --> D[Dynamic UI Components] + C --> E[Modal System] + end + + subgraph "Application Layer" + F[Flask App] --> G[Main Blueprint] + F --> H[Auth Blueprint] + G --> I[IMAP Service] + G --> J[Folder Management] + H --> K[User Authentication] + end + + subgraph "Data Layer" + L[PostgreSQL] --> M[User Model] + L --> N[Folder Model] + M --> O[IMAP Configuration] + N --> P[Folder Rules] + N --> Q[Email Metadata] + end + + subgraph "External Services" + R[IMAP Server] --> I + S[Future AI Service] --> I + end + + D --> F + F --> L + I --> R +``` + +## System Components + +### 1. Frontend Layer + +The frontend is built using a modern, lightweight stack that provides a responsive and interactive user experience: + +- **Base Template**: Foundation with DaisyUI theme, HTMX, and AlpineJS +- **Dynamic UI Components**: + - HTMX for server-side rendered content updates + - AlpineJS for client-side interactivity + - DaisyUI for consistent styling and components +- **Modal System**: Custom modal handling for forms and configuration + +### 2. Application Layer + +The Flask application follows a modular blueprint architecture: + +#### Flask App Factory +Implements the factory pattern for creating Flask application instances with configuration support. + +#### Main Blueprint +Handles core application functionality: +- Folder CRUD operations +- IMAP configuration and testing +- Folder synchronization +- User interface endpoints + +#### Auth Blueprint +Manages user authentication: +- User registration and login +- Password validation and hashing +- Session management + +### 3. Data Layer + +The system uses PostgreSQL with SQLAlchemy ORM for data persistence: + +#### User Model +Stores user account information and authentication data: +- Primary key: Integer auto-increment ID +- Personal information: First name, last name, email +- Authentication: Password hash +- Configuration: IMAP server settings in JSON format +- Timestamps: Creation and update times + +#### Folder Model +Stores email organization rules and metadata: +- Primary key: Integer auto-increment ID +- Relationship: Foreign key to user +- Rule definition: Natural language rule text +- Organization settings: Priority, enable/disable flag +- Email metrics: Total count, pending count +- Email metadata: Recent emails information in JSON format + +### 4. External Services + +#### IMAP Service +Handles communication with IMAP servers: +- Connection management and authentication +- Folder listing and synchronization +- Email retrieval and metadata extraction +- Connection testing and validation + +## Data Flow + +### User Authentication Flow + +```mermaid +sequenceDiagram + participant U as User + participant B as Browser + participant A as Auth Blueprint + participant DB as Database + + U->>B: Navigate to login page + B->>A: GET /auth/login + A->>B: Return login form + U->>B: Submit credentials + B->>A: POST /auth/login + A->>DB: Verify credentials + DB-->>A: User data + A->>B: Set session cookie + B->>U: Redirect to main page +``` + +### Folder Management Flow + +```mermaid +sequenceDiagram + participant U as User + participant B as Browser + participant M as Main Blueprint + participant DB as Database + participant I as IMAP Service + + U->>B: Click "Add Folder" + B->>M: GET /api/folders/new + M->>B: Return folder modal + U->>B: Fill and submit form + B->>M: POST /api/folders + M->>DB: Create new folder record + DB-->>M: Success confirmation + M->>B: Return updated folders list +``` + +### IMAP Synchronization Flow + +```mermaid +sequenceDiagram + participant U as User + participant B as Browser + participant M as Main Blueprint + participant I as IMAP Service + participant IMAP as IMAP Server + participant DB as Database + + U->>B: Click "Sync Folders" + B->>M: POST /api/imap/sync + M->>I: Initialize with user config + I->>IMAP: Establish connection + IMAP-->>I: Connection success + I->>IMAP: List all folders + IMAP-->>I: Folder list + I->>DB: Create/update local folders + DB-->>I: Commit changes + I->>M: Return sync results + M->>B: Return updated UI +``` + +## Design Patterns + +### 1. Factory Pattern +- **Implementation**: Flask app factory in [`app/__init__.py`](app/__init__.py:14) +- **Purpose**: Create application instances with different configurations +- **Benefits**: Supports multiple environments (development, testing, production) + +### 2. Blueprint Pattern +- **Implementation**: Separated functionality in [`app/routes.py`](app/routes.py:9) and [`app/auth.py`](app/auth.py:9) +- **Purpose**: Modularize application features +- **Benefits**: Code organization, easier testing, scalability + +### 3. Service Layer Pattern +- **Implementation**: IMAP service in [`app/imap_service.py`](app/imap_service.py:10) +- **Purpose**: Encapsulate business logic and external communication +- **Benefits**: Separation of concerns, reusability, testability + +### 4. Repository Pattern +- **Implementation**: SQLAlchemy models in [`app/models.py`](app/models.py:13) +- **Purpose**: Abstract data access layer +- **Benefits**: Database independence, centralized query logic + +## Security Considerations + +### 1. Authentication +- Password hashing using Werkzeug's security functions +- Session management with Flask-Login +- Input validation on all user submissions + +### 2. IMAP Security +- SSL/TLS encryption for IMAP connections +- Secure credential storage (JSON format) +- Connection timeout handling + +### 3. Data Protection +- Server-side validation for all form inputs +- Protection against SQL injection through SQLAlchemy ORM +- Error handling that doesn't expose sensitive information + +## Performance Considerations + +### 1. Database +- PostgreSQL for reliable data storage +- SQLAlchemy ORM for efficient querying +- Alembic for database migrations + +### 2. UI Performance +- HTMX for partial page updates +- Lazy loading of folder content +- Efficient rendering with DaisyUI components + +### 3. IMAP Operations +- Connection pooling for efficiency +- Timeout handling for reliability +- Batch processing for folder synchronization + +## Scalability + +### 1. Architecture +- Modular design supports feature expansion +- Service layer allows for additional email providers +- Database schema designed for growth + +### 2. Future Enhancements +- AI-powered rule recommendations +- Additional email provider support +- Multi-tenant architecture support + +## Error Handling + +### 1. User-Friendly Errors +- Clear validation messages for form inputs +- Graceful degradation for IMAP connection issues +- Modal-based error display + +### 2. Logging +- Comprehensive logging for debugging +- Error tracking for IMAP operations +- Database operation logging + +### 3. Recovery +- Transaction rollbacks for database operations +- Connection cleanup for IMAP service +- Session management for user errors \ No newline at end of file