email-organizer/docs/design/system-architecture.md

Email Organizer System Architecture

Overview

The Email Organizer is a self-hosted AI-powered email organization system that automates folder sorting, prioritization, and rule recommendations through natural language configuration. This document provides a comprehensive overview of the system architecture, components, and data flow.

High-Level Architecture

graph TB
    subgraph "Frontend Layer"
        A[Browser] --> B[Base Template]
        B --> C[HTMX/AlpineJS/DaisyUI]
        C --> D[Dynamic UI Components]
        C --> E[Modal System]
    end
    
    subgraph "Application Layer"
        F[Flask App] --> G[Main Blueprint]
        F --> H[Auth Blueprint]
        G --> I[IMAP Service]
        G --> J[Folder Management]
        H --> K[User Authentication]
    end
    
    subgraph "Data Layer"
        L[PostgreSQL] --> M[User Model]
        L --> N[Folder Model]
        M --> O[IMAP Configuration]
        N --> P[Folder Rules]
        N --> Q[Email Metadata]
    end
    
    subgraph "External Services"
        R[IMAP Server] --> I
        S[Future AI Service] --> I
    end
    
    D --> F
    F --> L
    I --> R

System Components

1. Frontend Layer

The frontend is built using a modern, lightweight stack that provides a responsive and interactive user experience:

• Base Template: Foundation with DaisyUI theme, HTMX, and AlpineJS
• Dynamic UI Components:
  • HTMX for server-side rendered content updates
  • AlpineJS for client-side interactivity
  • DaisyUI for consistent styling and components
• Modal System: Custom modal handling for forms and configuration

2. Application Layer

The Flask application follows a modular blueprint architecture:

Flask App Factory

Implements the factory pattern for creating Flask application instances with configuration support.

Main Blueprint

Handles core application functionality:

• Folder CRUD operations
• IMAP configuration and testing
• Folder synchronization
• User interface endpoints

Auth Blueprint

Manages user authentication:

• User registration and login
• Password validation and hashing
• Session management

3. Data Layer

The system uses PostgreSQL with SQLAlchemy ORM for data persistence:

User Model

Stores user account information and authentication data:

• Primary key: Integer auto-increment ID
• Personal information: First name, last name, email
• Authentication: Password hash
• Configuration: IMAP server settings in JSON format
• Timestamps: Creation and update times

Folder Model

Stores email organization rules and metadata:

• Primary key: Integer auto-increment ID
• Relationship: Foreign key to user
• Rule definition: Natural language rule text
• Organization settings: Priority, enable/disable flag
• Email metrics: Total count, pending count
• Email metadata: Recent emails information in JSON format

4. External Services

IMAP Service

Handles communication with IMAP servers:

• Connection management and authentication
• Folder listing and synchronization
• Email retrieval and metadata extraction
• Connection testing and validation

Data Flow

User Authentication Flow

sequenceDiagram
    participant U as User
    participant B as Browser
    participant A as Auth Blueprint
    participant DB as Database
    
    U->>B: Navigate to login page
    B->>A: GET /auth/login
    A->>B: Return login form
    U->>B: Submit credentials
    B->>A: POST /auth/login
    A->>DB: Verify credentials
    DB-->>A: User data
    A->>B: Set session cookie
    B->>U: Redirect to main page

Folder Management Flow

sequenceDiagram
    participant U as User
    participant B as Browser
    participant M as Main Blueprint
    participant DB as Database
    participant I as IMAP Service
    
    U->>B: Click "Add Folder"
    B->>M: GET /api/folders/new
    M->>B: Return folder modal
    U->>B: Fill and submit form
    B->>M: POST /api/folders
    M->>DB: Create new folder record
    DB-->>M: Success confirmation
    M->>B: Return updated folders list

IMAP Synchronization Flow

sequenceDiagram
    participant U as User
    participant B as Browser
    participant M as Main Blueprint
    participant I as IMAP Service
    participant IMAP as IMAP Server
    participant DB as Database
    
    U->>B: Click "Sync Folders"
    B->>M: POST /api/imap/sync
    M->>I: Initialize with user config
    I->>IMAP: Establish connection
    IMAP-->>I: Connection success
    I->>IMAP: List all folders
    IMAP-->>I: Folder list
    I->>DB: Create/update local folders
    DB-->>I: Commit changes
    I->>M: Return sync results
    M->>B: Return updated UI

Design Patterns

1. Factory Pattern

• Implementation: Flask app factory in app/__init__.py
• Purpose: Create application instances with different configurations
• Benefits: Supports multiple environments (development, testing, production)

2. Blueprint Pattern

• Implementation: Separated functionality in app/routes.py and app/auth.py
• Purpose: Modularize application features
• Benefits: Code organization, easier testing, scalability

3. Service Layer Pattern

• Implementation: IMAP service in app/imap_service.py
• Purpose: Encapsulate business logic and external communication
• Benefits: Separation of concerns, reusability, testability

4. Repository Pattern

• Implementation: SQLAlchemy models in app/models.py
• Purpose: Abstract data access layer
• Benefits: Database independence, centralized query logic

Security Considerations

1. Authentication

• Password hashing using Werkzeug's security functions
• Session management with Flask-Login
• Input validation on all user submissions

2. IMAP Security

• SSL/TLS encryption for IMAP connections
• Secure credential storage (JSON format)
• Connection timeout handling

3. Data Protection

• Server-side validation for all form inputs
• Protection against SQL injection through SQLAlchemy ORM
• Error handling that doesn't expose sensitive information

Performance Considerations

1. Database

• PostgreSQL for reliable data storage
• SQLAlchemy ORM for efficient querying
• Alembic for database migrations

2. UI Performance

• HTMX for partial page updates
• Lazy loading of folder content
• Efficient rendering with DaisyUI components

3. IMAP Operations

• Connection pooling for efficiency
• Timeout handling for reliability
• Batch processing for folder synchronization

Scalability

1. Architecture

• Modular design supports feature expansion
• Service layer allows for additional email providers
• Database schema designed for growth

2. Future Enhancements

• AI-powered rule recommendations
• Additional email provider support
• Multi-tenant architecture support

Error Handling

1. User-Friendly Errors

• Clear validation messages for form inputs
• Graceful degradation for IMAP connection issues
• Modal-based error display

2. Logging

• Comprehensive logging for debugging
• Error tracking for IMAP operations
• Database operation logging

3. Recovery

• Transaction rollbacks for database operations
• Connection cleanup for IMAP service
• Session management for user errors