email-organizer/docs/design/system-architecture.md

# Email Organizer System Architecture

## Overview

The Email Organizer is a self-hosted AI-powered email organization system that automates folder sorting, prioritization, and rule recommendations through natural language configuration. This document provides a comprehensive overview of the system architecture, components, and data flow.

## High-Level Architecture

```mermaid
graph TB
    subgraph "Frontend Layer"
        A[Browser] --> B[Base Template]
        B --> C[HTMX/AlpineJS/DaisyUI]
        C --> D[Dynamic UI Components]
        C --> E[Modal System]
    end
    
    subgraph "Application Layer"
        F[Flask App] --> G[Main Blueprint]
        F --> H[Auth Blueprint]
        G --> I[Folders Blueprint]
        G --> J[IMAP Blueprint]
        G --> K[Emails Blueprint]
        I --> L[Folder CRUD Operations]
        J --> M[IMAP Configuration & Sync]
        K --> N[Email Processing]
        H --> O[User Authentication]
    end
    
    subgraph "Service Layer"
        P[IMAP Service] --> Q[Email Operations]
        R[Processed Emails Service] --> S[Email Tracking]
    end
    
    subgraph "Data Layer"
        T[PostgreSQL] --> U[User Model]
        T --> V[Folder Model]
        T --> W[ProcessedEmail Model]
        U --> X[IMAP Configuration]
        V --> Y[Folder Rules & Metadata]
        W --> Z[Email Processing Status]
    end
    
    subgraph "External Services"
        AA[IMAP Server] --> P
    end
    
    D --> F
    F --> T
    P --> AA
    Q --> R
```

## System Components

### 1. Frontend Layer

The frontend is built using a modern, lightweight stack that provides a responsive and interactive user experience:

- **Base Template**: Foundation with DaisyUI theme, HTMX, and AlpineJS
- **Dynamic UI Components**: 
  - HTMX for server-side rendered content updates
  - AlpineJS for client-side interactivity
  - DaisyUI for consistent styling and components
- **Modal System**: Custom modal handling for forms and configuration

### 2. Application Layer

The Flask application follows a modular blueprint architecture:

#### Flask App Factory
Implements the factory pattern for creating Flask application instances with configuration support.

#### Main Blueprint
The main application blueprint coordinates all core functionality through sub-blueprints:
- **Folders Blueprint**: Handles folder CRUD operations, type management, and user interface
- **IMAP Blueprint**: Manages IMAP configuration, connection testing, and folder synchronization
- **Emails Blueprint**: Processes email operations and pending email management

#### Auth Blueprint
Manages user authentication:
- User registration and login
- Password validation and hashing
- Session management

### 3. Service Layer

The application implements a service layer pattern for business logic:

#### IMAP Service
Handles communication with IMAP servers:
- Connection management and authentication
- Folder listing and synchronization
- Email retrieval and metadata extraction
- Connection testing and validation
- Email UID tracking for processed emails

#### Processed Emails Service
Manages email processing status tracking:
- Records which emails have been processed
- Provides pending email counts
- Handles bulk email processing operations
- Maintains email processing history

### 4. Data Layer

The system uses PostgreSQL with SQLAlchemy ORM for data persistence:

#### User Model
Stores user account information and authentication data:
- Primary key: Integer auto-increment ID
- Personal information: First name, last name, email
- Authentication: Password hash
- Configuration: IMAP server settings in JSON format
- Timestamps: Creation and update times

#### Folder Model
Stores email organization rules and metadata:
- Primary key: Integer auto-increment ID
- Relationship: Foreign key to user
- Rule definition: Natural language rule text
- Organization settings: Priority, enable/disable flag
- Email metrics: Total count, pending count, emails count
- Email metadata: Recent emails information in JSON format
- Folder types: Tidy, Destination, or Ignore

#### ProcessedEmail Model
Tracks email processing status:
- Primary key: Integer auto-increment ID
- Relationships: Foreign keys to User and Folder
- Email UID tracking: Unique identifiers from IMAP server
- Processing status: Pending or processed flags
- Timestamps: First seen, processed, and update times

### 4. External Services

#### IMAP Service
Handles communication with IMAP servers:
- Connection management and authentication
- Folder listing and synchronization
- Email retrieval and metadata extraction
- Connection testing and validation

## Data Flow

### User Authentication Flow

```mermaid
sequenceDiagram
    participant U as User
    participant B as Browser
    participant A as Auth Blueprint
    participant DB as Database
    
    U->>B: Navigate to login page
    B->>A: GET /auth/login
    A->>B: Return login form
    U->>B: Submit credentials
    B->>A: POST /auth/login
    A->>DB: Verify credentials
    DB-->>A: User data
    A->>B: Set session cookie
    B->>U: Redirect to main page
```

### Folder Management Flow

```mermaid
sequenceDiagram
    participant U as User
    participant B as Browser
    participant M as Main Blueprint
    participant DB as Database
    participant I as IMAP Service
    
    U->>B: Click "Add Folder"
    B->>M: GET /api/folders/new
    M->>B: Return folder modal
    U->>B: Fill and submit form
    B->>M: POST /api/folders
    M->>DB: Create new folder record
    DB-->>M: Success confirmation
    M->>B: Return updated folders list
```

### IMAP Synchronization Flow

```mermaid
sequenceDiagram
    participant U as User
    participant B as Browser
    participant M as Main Blueprint
    participant I as IMAP Service
    participant IMAP as IMAP Server
    participant DB as Database
    
    U->>B: Click "Sync Folders"
    B->>M: POST /api/imap/sync
    M->>I: Initialize with user config
    I->>IMAP: Establish connection
    IMAP-->>I: Connection success
    I->>IMAP: List all folders
    IMAP-->>I: Folder list
    I->>DB: Create/update local folders
    DB-->>I: Commit changes
    I->>M: Return sync results
    M->>B: Return updated UI
```

## Design Patterns

### 1. Factory Pattern
- **Implementation**: Flask app factory in [`app/__init__.py`](app/__init__.py:14)
- **Purpose**: Create application instances with different configurations
- **Benefits**: Supports multiple environments (development, testing, production)

### 2. Blueprint Pattern
- **Implementation**: Modular blueprint structure in [`app/routes.py`](app/routes.py:12)
- **Purpose**: Modularize application features into logical groups
- **Benefits**: Code organization, easier testing, scalability, clear separation of concerns

### 3. Service Layer Pattern
- **Implementation**: IMAP service in [`app/imap_service.py`](app/imap_service.py:11) and Processed Emails service in [`app/processed_emails_service.py`](app/processed_emails_service.py:7)
- **Purpose**: Encapsulate business logic and external communication
- **Benefits**: Separation of concerns, reusability, testability, centralized business rules

### 4. Repository Pattern
- **Implementation**: SQLAlchemy models in [`app/models.py`](app/models.py:13)
- **Purpose**: Abstract data access layer
- **Benefits**: Database independence, centralized query logic, type safety

## Security Considerations

### 1. Authentication
- Password hashing using Werkzeug's security functions
- Session management with Flask-Login
- Input validation on all user submissions

### 2. IMAP Security
- SSL/TLS encryption for IMAP connections
- Secure credential storage (JSON format)
- Connection timeout handling

### 3. Data Protection
- Server-side validation for all form inputs
- Protection against SQL injection through SQLAlchemy ORM
- Error handling that doesn't expose sensitive information

## Performance Considerations

### 1. Database
- PostgreSQL for reliable data storage
- SQLAlchemy ORM for efficient querying
- Alembic for database migrations

### 2. UI Performance
- HTMX for partial page updates
- Lazy loading of folder content
- Efficient rendering with DaisyUI components

### 3. IMAP Operations
- Connection pooling for efficiency
- Timeout handling for reliability
- Batch processing for folder synchronization

## Scalability

### 1. Architecture
- Modular blueprint structure supports feature expansion
- Service layer allows for additional email providers and processing logic
- Database schema designed for growth with processed email tracking
- Current folder type system (Tidy, Destination, Ignore) provides flexible organization

### 2. Current Implementation Status
- IMAP connectivity and folder synchronization fully implemented
- Email processing status tracking operational
- Folder type management with UI components complete
- Authentication system with session management functional

### 3. Future Enhancements
- AI-powered rule recommendations
- Additional email provider support
- Multi-tenant architecture support
- Enhanced email analytics and reporting

## Error Handling

### 1. User-Friendly Errors
- Clear validation messages for form inputs
- Graceful degradation for IMAP connection issues
- Modal-based error display

### 2. Logging
- Comprehensive logging for debugging
- Error tracking for IMAP operations
- Database operation logging

### 3. Recovery
- Transaction rollbacks for database operations
- Connection cleanup for IMAP service
- Session management for user errors