email-organizer/docs/design/system-architecture.md

Email Organizer System Architecture

Overview

The Email Organizer is a self-hosted AI-powered email organization system that automates folder sorting, prioritization, and rule recommendations through natural language configuration. This document provides a comprehensive overview of the system architecture, components, and data flow.

High-Level Architecture

graph TB
    subgraph "Frontend Layer"
        A[Browser] --> B[Base Template]
        B --> C[HTMX/AlpineJS/DaisyUI]
        C --> D[Dynamic UI Components]
        C --> E[Modal System]
    end
    
    subgraph "Application Layer"
        F[Flask App] --> G[Main Blueprint]
        F --> H[Auth Blueprint]
        G --> I[Folders Blueprint]
        G --> J[IMAP Blueprint]
        G --> K[Emails Blueprint]
        I --> L[Folder CRUD Operations]
        J --> M[IMAP Configuration & Sync]
        K --> N[Email Processing]
        H --> O[User Authentication]
    end
    
    subgraph "Service Layer"
        P[IMAP Service] --> Q[Email Operations]
        R[Processed Emails Service] --> S[Email Tracking]
    end
    
    subgraph "Data Layer"
        T[PostgreSQL] --> U[User Model]
        T --> V[Folder Model]
        T --> W[ProcessedEmail Model]
        U --> X[IMAP Configuration]
        V --> Y[Folder Rules & Metadata]
        W --> Z[Email Processing Status]
    end
    
    subgraph "External Services"
        AA[IMAP Server] --> P
    end
    
    D --> F
    F --> T
    P --> AA
    Q --> R

System Components

1. Frontend Layer

The frontend is built using a modern, lightweight stack that provides a responsive and interactive user experience:

• Base Template: Foundation with DaisyUI theme, HTMX, and AlpineJS
• Dynamic UI Components:
  • HTMX for server-side rendered content updates
  • AlpineJS for client-side interactivity
  • DaisyUI for consistent styling and components
• Modal System: Custom modal handling for forms and configuration

2. Application Layer

The Flask application follows a modular blueprint architecture:

Flask App Factory

Implements the factory pattern for creating Flask application instances with configuration support.

Main Blueprint

The main application blueprint coordinates all core functionality through sub-blueprints:

• Folders Blueprint: Handles folder CRUD operations, type management, and user interface
• IMAP Blueprint: Manages IMAP configuration, connection testing, and folder synchronization
• Emails Blueprint: Processes email operations and pending email management

Auth Blueprint

Manages user authentication:

• User registration and login
• Password validation and hashing
• Session management

3. Service Layer

The application implements a service layer pattern for business logic:

IMAP Service

Handles communication with IMAP servers:

• Connection management and authentication
• Folder listing and synchronization
• Email retrieval and metadata extraction
• Connection testing and validation
• Email UID tracking for processed emails

Processed Emails Service

Manages email processing status tracking:

• Records which emails have been processed
• Provides pending email counts
• Handles bulk email processing operations
• Maintains email processing history

4. Data Layer

The system uses PostgreSQL with SQLAlchemy ORM for data persistence:

User Model

Stores user account information and authentication data:

• Primary key: Integer auto-increment ID
• Personal information: First name, last name, email
• Authentication: Password hash
• Configuration: IMAP server settings in JSON format
• Timestamps: Creation and update times

Folder Model

Stores email organization rules and metadata:

• Primary key: Integer auto-increment ID
• Relationship: Foreign key to user
• Rule definition: Natural language rule text
• Organization settings: Priority, enable/disable flag
• Email metrics: Total count, pending count, emails count
• Email metadata: Recent emails information in JSON format
• Folder types: Tidy, Destination, or Ignore

ProcessedEmail Model

Tracks email processing status:

• Primary key: Integer auto-increment ID
• Relationships: Foreign keys to User and Folder
• Email UID tracking: Unique identifiers from IMAP server
• Processing status: Pending or processed flags
• Timestamps: First seen, processed, and update times

4. External Services

IMAP Service

Handles communication with IMAP servers:

• Connection management and authentication
• Folder listing and synchronization
• Email retrieval and metadata extraction
• Connection testing and validation

Data Flow

User Authentication Flow

sequenceDiagram
    participant U as User
    participant B as Browser
    participant A as Auth Blueprint
    participant DB as Database
    
    U->>B: Navigate to login page
    B->>A: GET /auth/login
    A->>B: Return login form
    U->>B: Submit credentials
    B->>A: POST /auth/login
    A->>DB: Verify credentials
    DB-->>A: User data
    A->>B: Set session cookie
    B->>U: Redirect to main page

Folder Management Flow

sequenceDiagram
    participant U as User
    participant B as Browser
    participant M as Main Blueprint
    participant DB as Database
    participant I as IMAP Service
    
    U->>B: Click "Add Folder"
    B->>M: GET /api/folders/new
    M->>B: Return folder modal
    U->>B: Fill and submit form
    B->>M: POST /api/folders
    M->>DB: Create new folder record
    DB-->>M: Success confirmation
    M->>B: Return updated folders list

IMAP Synchronization Flow

sequenceDiagram
    participant U as User
    participant B as Browser
    participant M as Main Blueprint
    participant I as IMAP Service
    participant IMAP as IMAP Server
    participant DB as Database
    
    U->>B: Click "Sync Folders"
    B->>M: POST /api/imap/sync
    M->>I: Initialize with user config
    I->>IMAP: Establish connection
    IMAP-->>I: Connection success
    I->>IMAP: List all folders
    IMAP-->>I: Folder list
    I->>DB: Create/update local folders
    DB-->>I: Commit changes
    I->>M: Return sync results
    M->>B: Return updated UI

Design Patterns

1. Factory Pattern

• Implementation: Flask app factory in app/__init__.py
• Purpose: Create application instances with different configurations
• Benefits: Supports multiple environments (development, testing, production)

2. Blueprint Pattern

• Implementation: Modular blueprint structure in app/routes.py
• Purpose: Modularize application features into logical groups
• Benefits: Code organization, easier testing, scalability, clear separation of concerns

3. Service Layer Pattern

• Implementation: IMAP service in app/imap_service.py and Processed Emails service in app/processed_emails_service.py
• Purpose: Encapsulate business logic and external communication
• Benefits: Separation of concerns, reusability, testability, centralized business rules

4. Repository Pattern

• Implementation: SQLAlchemy models in app/models.py
• Purpose: Abstract data access layer
• Benefits: Database independence, centralized query logic, type safety

Security Considerations

1. Authentication

• Password hashing using Werkzeug's security functions
• Session management with Flask-Login
• Input validation on all user submissions

2. IMAP Security

• SSL/TLS encryption for IMAP connections
• Secure credential storage (JSON format)
• Connection timeout handling

3. Data Protection

• Server-side validation for all form inputs
• Protection against SQL injection through SQLAlchemy ORM
• Error handling that doesn't expose sensitive information

Performance Considerations

1. Database

• PostgreSQL for reliable data storage
• SQLAlchemy ORM for efficient querying
• Alembic for database migrations

2. UI Performance

• HTMX for partial page updates
• Lazy loading of folder content
• Efficient rendering with DaisyUI components

3. IMAP Operations

• Connection pooling for efficiency
• Timeout handling for reliability
• Batch processing for folder synchronization

Scalability

1. Architecture

• Modular blueprint structure supports feature expansion
• Service layer allows for additional email providers and processing logic
• Database schema designed for growth with processed email tracking
• Current folder type system (Tidy, Destination, Ignore) provides flexible organization

2. Current Implementation Status

• IMAP connectivity and folder synchronization fully implemented
• Email processing status tracking operational
• Folder type management with UI components complete
• Authentication system with session management functional

3. Future Enhancements

• AI-powered rule recommendations
• Additional email provider support
• Multi-tenant architecture support
• Enhanced email analytics and reporting

Error Handling

1. User-Friendly Errors

• Clear validation messages for form inputs
• Graceful degradation for IMAP connection issues
• Modal-based error display

2. Logging

• Comprehensive logging for debugging
• Error tracking for IMAP operations
• Database operation logging

3. Recovery

• Transaction rollbacks for database operations
• Connection cleanup for IMAP service
• Session management for user errors