email-organizer/docs/design/system-architecture.md

# Email Organizer System Architecture

## Overview

The Email Organizer is a self-hosted AI-powered email organization system that automates folder sorting, prioritization, and rule recommendations through natural language configuration. This document provides a comprehensive overview of the system architecture, components, and data flow.

## High-Level Architecture

```mermaid
graph TB
    subgraph "Frontend Layer"
        A[Browser] --> B[Base Template]
        B --> C[HTMX/AlpineJS/DaisyUI]
        C --> D[Dynamic UI Components]
        C --> E[Modal System]
    end
    
    subgraph "Application Layer"
        F[Flask App] --> G[Main Blueprint]
        F --> H[Auth Blueprint]
        G --> I[IMAP Service]
        G --> J[Folder Management]
        H --> K[User Authentication]
    end
    
    subgraph "Data Layer"
        L[PostgreSQL] --> M[User Model]
        L --> N[Folder Model]
        M --> O[IMAP Configuration]
        N --> P[Folder Rules]
        N --> Q[Email Metadata]
    end
    
    subgraph "External Services"
        R[IMAP Server] --> I
        S[Future AI Service] --> I
    end
    
    D --> F
    F --> L
    I --> R
```

## System Components

### 1. Frontend Layer

The frontend is built using a modern, lightweight stack that provides a responsive and interactive user experience:

- **Base Template**: Foundation with DaisyUI theme, HTMX, and AlpineJS
- **Dynamic UI Components**: 
  - HTMX for server-side rendered content updates
  - AlpineJS for client-side interactivity
  - DaisyUI for consistent styling and components
- **Modal System**: Custom modal handling for forms and configuration

### 2. Application Layer

The Flask application follows a modular blueprint architecture:

#### Flask App Factory
Implements the factory pattern for creating Flask application instances with configuration support.

#### Main Blueprint
Handles core application functionality:
- Folder CRUD operations
- IMAP configuration and testing
- Folder synchronization
- User interface endpoints

#### Auth Blueprint
Manages user authentication:
- User registration and login
- Password validation and hashing
- Session management

### 3. Data Layer

The system uses PostgreSQL with SQLAlchemy ORM for data persistence:

#### User Model
Stores user account information and authentication data:
- Primary key: Integer auto-increment ID
- Personal information: First name, last name, email
- Authentication: Password hash
- Configuration: IMAP server settings in JSON format
- Timestamps: Creation and update times

#### Folder Model
Stores email organization rules and metadata:
- Primary key: Integer auto-increment ID
- Relationship: Foreign key to user
- Rule definition: Natural language rule text
- Organization settings: Priority, enable/disable flag
- Email metrics: Total count, pending count
- Email metadata: Recent emails information in JSON format

### 4. External Services

#### IMAP Service
Handles communication with IMAP servers:
- Connection management and authentication
- Folder listing and synchronization
- Email retrieval and metadata extraction
- Connection testing and validation

## Data Flow

### User Authentication Flow

```mermaid
sequenceDiagram
    participant U as User
    participant B as Browser
    participant A as Auth Blueprint
    participant DB as Database
    
    U->>B: Navigate to login page
    B->>A: GET /auth/login
    A->>B: Return login form
    U->>B: Submit credentials
    B->>A: POST /auth/login
    A->>DB: Verify credentials
    DB-->>A: User data
    A->>B: Set session cookie
    B->>U: Redirect to main page
```

### Folder Management Flow

```mermaid
sequenceDiagram
    participant U as User
    participant B as Browser
    participant M as Main Blueprint
    participant DB as Database
    participant I as IMAP Service
    
    U->>B: Click "Add Folder"
    B->>M: GET /api/folders/new
    M->>B: Return folder modal
    U->>B: Fill and submit form
    B->>M: POST /api/folders
    M->>DB: Create new folder record
    DB-->>M: Success confirmation
    M->>B: Return updated folders list
```

### IMAP Synchronization Flow

```mermaid
sequenceDiagram
    participant U as User
    participant B as Browser
    participant M as Main Blueprint
    participant I as IMAP Service
    participant IMAP as IMAP Server
    participant DB as Database
    
    U->>B: Click "Sync Folders"
    B->>M: POST /api/imap/sync
    M->>I: Initialize with user config
    I->>IMAP: Establish connection
    IMAP-->>I: Connection success
    I->>IMAP: List all folders
    IMAP-->>I: Folder list
    I->>DB: Create/update local folders
    DB-->>I: Commit changes
    I->>M: Return sync results
    M->>B: Return updated UI
```

## Design Patterns

### 1. Factory Pattern
- **Implementation**: Flask app factory in [`app/__init__.py`](app/__init__.py:14)
- **Purpose**: Create application instances with different configurations
- **Benefits**: Supports multiple environments (development, testing, production)

### 2. Blueprint Pattern
- **Implementation**: Separated functionality in [`app/routes.py`](app/routes.py:9) and [`app/auth.py`](app/auth.py:9)
- **Purpose**: Modularize application features
- **Benefits**: Code organization, easier testing, scalability

### 3. Service Layer Pattern
- **Implementation**: IMAP service in [`app/imap_service.py`](app/imap_service.py:10)
- **Purpose**: Encapsulate business logic and external communication
- **Benefits**: Separation of concerns, reusability, testability

### 4. Repository Pattern
- **Implementation**: SQLAlchemy models in [`app/models.py`](app/models.py:13)
- **Purpose**: Abstract data access layer
- **Benefits**: Database independence, centralized query logic

## Security Considerations

### 1. Authentication
- Password hashing using Werkzeug's security functions
- Session management with Flask-Login
- Input validation on all user submissions

### 2. IMAP Security
- SSL/TLS encryption for IMAP connections
- Secure credential storage (JSON format)
- Connection timeout handling

### 3. Data Protection
- Server-side validation for all form inputs
- Protection against SQL injection through SQLAlchemy ORM
- Error handling that doesn't expose sensitive information

## Performance Considerations

### 1. Database
- PostgreSQL for reliable data storage
- SQLAlchemy ORM for efficient querying
- Alembic for database migrations

### 2. UI Performance
- HTMX for partial page updates
- Lazy loading of folder content
- Efficient rendering with DaisyUI components

### 3. IMAP Operations
- Connection pooling for efficiency
- Timeout handling for reliability
- Batch processing for folder synchronization

## Scalability

### 1. Architecture
- Modular design supports feature expansion
- Service layer allows for additional email providers
- Database schema designed for growth

### 2. Future Enhancements
- AI-powered rule recommendations
- Additional email provider support
- Multi-tenant architecture support

## Error Handling

### 1. User-Friendly Errors
- Clear validation messages for form inputs
- Graceful degradation for IMAP connection issues
- Modal-based error display

### 2. Logging
- Comprehensive logging for debugging
- Error tracking for IMAP operations
- Database operation logging

### 3. Recovery
- Transaction rollbacks for database operations
- Connection cleanup for IMAP service
- Session management for user errors