update docs.

2025-08-09 21:04:21 -07:00
parent af637f29b4
commit d6635a42df
7 changed files with 484 additions and 86 deletions
--- a/docs/design/authentication-system.md
+++ b/docs/design/authentication-system.md
@@ -47,14 +47,14 @@ graph TB
 The authentication blueprint handles all authentication-related routes and logic:

 #### Routes
- `/login`: GET/POST - User login page and authentication
- `/signup`: GET/POST - User registration page and account creation
- `/logout`: GET - User logout and session termination
+- `/auth/login`: GET/POST - User login page and authentication
+- `/auth/signup`: GET/POST - User registration page and account creation
+- `/auth/logout`: GET - User logout and session termination

 #### Key Functions
- `login()`: Handles user authentication
- `signup()`: Manages user registration
- `logout()`: Terminates user sessions
+- `login()`: Handles user authentication with form validation
+- `signup()`: Manages user registration with server-side validation
+- `logout()`: Terminates user sessions and clears authentication state
 - `validate_password()`: Enforces password complexity requirements

 ### 2. User Model ([`app/models.py`](app/models.py:13))
@@ -288,6 +288,15 @@ login_manager.login_message = 'Please log in to access this page.'

 ### Blueprint Registration

+The auth blueprint is registered with URL prefix `/auth/`:
+
+```python
+from app.auth import auth
+app.register_blueprint(auth, url_prefix='/auth')
+```
+
+## Testing Strategy
+
 ### Unit Tests

 #### Authentication Tests
@@ -350,4 +359,11 @@ login_manager.login_message_category = 'warning'
 - Suspicious activity monitoring
 - IP-based security checks

- IP-based security checks
+### 6. Current Implementation Status
+- User authentication with session management fully implemented
+- Password complexity validation operational
+- Server-side form validation complete
+- Error handling with user-friendly messages
+- Blueprint-based routing with proper URL prefixes
+- Error handling with user-friendly messages
+- Blueprint-based routing with proper URL prefixes
--- a/docs/design/data-model.md
+++ b/docs/design/data-model.md
@@ -242,9 +242,10 @@ Folders with `folder_type = 'tidy'` are source folders that contain emails waiti
 - Example: Inbox folder

 **UI Representation:**
- Shows "pending count" and "processed count" badges
+- Shows "total", "pending", and "processed" count badges
 - Includes "View Pending" button if there are pending emails
 - May include priority indicators
+- Located in "Emails to organize" section

 ### Destination Folders

@@ -260,6 +261,7 @@ Folders with `folder_type = 'destination'` are target folders where emails are m
 - Shows "emails count" badge
 - Simpler interface without pending/processed indicators
 - Focus on folder management and viewing contents
+- Located in "Destination Folders" section

 ### Ignore Folders

@@ -276,6 +278,7 @@ Folders with `folder_type = 'ignore'` are folders that are stored in the databas
 - Hidden by default unless "Show Hidden" checkbox is checked
 - When visible, shows minimal information
 - No action buttons for organization or processing
+- Located in "Hidden Folders" section

 ### Folder Type Determination

@@ -287,17 +290,50 @@ Folder types are determined as follows:
    - Inbox: Tidy
    - Archive/Spam/Drafts: Ignore
    - All others: Destination
- Manually created folders default to 'destination'
- Folder type can be changed through the user interface
+- Manually created folders default to 'destination' (except 'inbox' which defaults to 'tidy')
+- Folder type can be changed through the user interface via dropdown
 - When changing to 'ignore', emails_count is reset to 0

+## ProcessedEmail Model
+
+The `ProcessedEmail` entity tracks email processing status to avoid reprocessing the same emails during synchronization and provide accurate pending email counts.
+
+### Attributes
+
+| Column Name | Data Type | Constraints | Description |
+|-------------|------------|--------------|-------------|
+| id | Integer | Primary Key, Autoincrement | Unique identifier for each processed email record |
+| user_id | Integer | Foreign Key to User, Not Null | Reference to the user who owns this email |
+| folder_id | Integer | Foreign Key to Folder, Not Null | Reference to the folder this email belongs to |
+| email_uid | String(255) | Not Null | Unique ID of the email from IMAP server |
+| folder_name | String(255) | Not Null | Name of the IMAP folder (for redundancy) |
+| is_processed | Boolean | Default: False | Processing status (false=pending, true=processed) |
+| first_seen_at | DateTime | Default: datetime.utcnow | First time this email was detected during sync |
+| processed_at | DateTime | Nullable | When the email was marked as processed |
+| created_at | DateTime | Default: datetime.utcnow | Record creation timestamp |
+| updated_at | DateTime | Default: datetime.utcnow, On Update | Record update timestamp |
+
+### Relationships
+
+- **Many-to-One**: Each `ProcessedEmail` belongs to one `User`
+- **Many-to-One**: Each `ProcessedEmail` belongs to one `Folder`
+- **Composite Key**: The combination of (user_id, folder_name, email_uid) should be unique to prevent duplicate records
+
+### Business Rules
+
+- Each processed email record must belong to a user and folder
+- Email UID must be unique per user and folder to prevent duplicates
+- Processing status tracks whether an email has been processed
+- First seen timestamp tracks when the email was first discovered
+- Processed timestamp is set when email is marked as processed
+
 ## Future Data Model Considerations

 ### Potential Enhancements

-1. **Email Entity**
-   - Store email metadata for better analytics
-   - Track email movement between folders
+1. **Email Movement Tracking**
+   - Store email movement history between folders
+   - Track source and destination folder for each moved email

 2. **Rule Engine**
   - Store parsed rule structures for better processing
@@ -309,4 +345,5 @@ Folder types are determined as follows:

 4. **Audit Log**
   - Track changes to user data
-   - Monitor folder operations
+   - Monitor folder operations
+   - Log email processing actions
--- a/docs/design/development-setup.md
+++ b/docs/design/development-setup.md
@@ -6,6 +6,30 @@ This document describes how to set up a development environment for the Email Or

 - Docker and Docker Compose
 - Python 3.10+
+- Flask CLI tools
+
+## Project Structure
+
+The application follows a modular structure:
+
+```
+app/
+├── __init__.py          # Flask app factory and configuration
+├── models.py            # Database models (User, Folder, ProcessedEmail)
+├── auth.py              # Authentication blueprint
+├── routes.py            # Main blueprint with sub-blueprints
+├── imap_service.py      # IMAP communication service
+├── processed_emails_service.py  # Email processing tracking
+└── routes/              # Sub-blueprint modules
+    ├── __init__.py
+    ├── emails.py        # Email processing endpoints
+    ├── folders.py       # Folder management endpoints
+    └── imap.py          # IMAP configuration and sync endpoints
+
+templates/              # Jinja2 templates with HTMX integration
+static/                 # Static assets
+config.py               # Application configuration
+```

 ## Services

@@ -40,7 +64,7 @@ This command will:
 ### IMAP Server

 - Host: localhost
- Port: 1143
+- Port: 5143
 - Users:
  - user1@example.com / password1
  - user2@example.com / password2
@@ -61,4 +85,100 @@ After running `flask setup-dev`, copy the example environment file:
 cp .env.example .env
 ```

-The default configuration should work with the development services. You can modify the `.env` file if needed.
+The default configuration should work with the development services. You can modify the `.env` file if needed.
+
+### Configuration File
+
+The [`config.py`](config.py:1) file provides different environment configurations:
+
+```python
+class Config:
+    SECRET_KEY = os.environ.get('SECRET_KEY') or 'dev-secret-key'
+    SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or 'postgresql://postgres:password@localhost:5432/email_organizer_dev'
+    SQLALCHEMY_TRACK_MODIFICATIONS = False
+
+class TestingConfig(Config):
+    TESTING = True
+    SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'  # In-memory database for tests
+```
+
+## Database Setup
+
+The application uses Alembic for database migrations:
+
+```bash
+# Create migration
+flask db migrate -m "Initial migration"
+
+# Apply migration
+flask db upgrade
+```
+
+## Running the Application
+
+```bash
+# Development server
+flask run
+
+# With specific host and port
+flask run --host=0.0.0.0 --port=5001
+```
+
+The application will be available at `http://localhost:5001`
+
+## Testing
+
+The application includes comprehensive tests:
+
+```bash
+# Run all tests
+pytest
+
+# Run with coverage
+pytest --cov=app
+
+# Run specific test file
+pytest tests/unit/test_auth.py
+
+# Run integration tests
+pytest tests/integration/
+```
+
+### Test Structure
+
+- **Unit Tests**: Test individual components and services
+- **Integration Tests**: Test API endpoints and database interactions
+- **Functional Tests**: Test user workflows and UI interactions
+
+## Development Workflow
+
+1. **Setup Environment**: Run `flask setup-dev` to start services
+2. **Configure Database**: Run migrations to set up database schema
+3. **Run Application**: Start Flask development server
+4. **Test Changes**: Write tests for new features
+5. **Commit Changes**: Follow conventional commit messages
+
+## Key Technologies
+
+- **Backend**: Flask, SQLAlchemy, PostgreSQL
+- **Frontend**: HTMX, Alpine.js, DaisyUI, TailwindCSS
+- **Database**: PostgreSQL with Alembic migrations
+- **Testing**: pytest with Beautiful Soup for HTML testing
+- **Development**: Docker for services, Flask CLI for management
+
+## Current Implementation Status
+
+### Core Features
+- User authentication and session management ✅
+- IMAP configuration and connection testing ✅
+- Folder synchronization with type assignment ✅
+- Email processing status tracking ✅
+- HTMX-based UI with modals and dialogs ✅
+- Folder type management (Tidy, Destination, Ignore) ✅
+
+### Development Tools
+- Database migrations with Alembic ✅
+- Comprehensive test suite ✅
+- Docker-based development environment ✅
+- Flask CLI for development tasks ✅
+- Environment-based configuration ✅
--- a/docs/design/folder-types-ui.md
+++ b/docs/design/folder-types-ui.md
@@ -12,43 +12,145 @@ The main page is divided into sections based on folder types:
 2. **Destination Folders Section**: Contains folders with `folder_type = 'destination'`
 3. **Hidden Folders Section**: Contains folders with `folder_type = 'ignore'` (visible only when "Show Hidden" is checked)

+## Main Page Layout
+
+The main page ([`app/templates/index.html`](app/templates/index.html:1)) provides:
+- Header with navigation and user information
+- Sidebar with main navigation
+- Main content area with folder sections
+- Modal holder for dynamic content
+
+### Dashboard Statistics
+When IMAP is configured, the dashboard displays:
+- Total folders count
+- Folders to tidy count
+- Destination folders count
+- Total pending emails count
+
+### Search and Filter Controls
+- Search input for folder names
+- "Show Hidden" checkbox for ignore folders
+- Filter buttons for All, High Priority, Normal priority
+
 ## Folder Card Components

-### Base Template
+### Base Template (`folder_card_base.html`)
 All folder cards extend from `folder_card_base.html` which provides:
 - Common card structure with edit/delete buttons
 - Priority badge display
 - Rule text section
 - Block for additional content specific to folder type
+- Folder type dropdown for changing folder types

 ### Tidy Folder Card (`folder_card_tidy.html`)
 Extends base template with:
 - Total, pending, and processed email count badges
- Organize toggle switch
- View pending emails button
+- "View Pending" button that opens a modal with pending emails
+- Recent emails tooltip for processed emails

 ### Destination Folder Card (`folder_card_destination.html`)
 Extends base template with:
- Email count badge
+- Email count badge showing emails moved to this folder
 - Simplified UI without processing elements

-### Ignore Folder Card (to be created)
-Will extend base template with:
+### Ignore Folder Card (`folder_card_ignore.html`)
+Extends base template with:
 - "Ignored" status badge
+- Email count badge (reset to 0 when changed to ignore)
 - Minimal information display
 - No processing elements

+## Section Templates
+
+### Folders to Tidy Section (`folders_to_tidy_section.html`)
+- Section header with descriptive text and icon
+- Grid layout for folder cards
+- Empty state with call-to-action button
+- HTMX integration for dynamic updates
+
+### Destination Folders Section (`destination_folders_section.html`)
+- Section header with descriptive text and icon
+- Grid layout for folder cards
+- Empty state with call-to-action button
+- HTMX integration for dynamic updates
+
+### Hidden Folders Section (`hidden_folders_section.html`)
+- Section header with eye-slash icon
+- Grid layout for folder cards
+- Empty state message
+- Only visible when "Show Hidden" is checked
+
+## Modal System
+
+### Folder Modal (`folder_modal.html`)
+- Used for creating and editing folders
+- Form validation with server-side error handling
+- HTMX integration for dynamic form submission
+
+### IMAP Configuration Modal (`imap_config_modal.html`)
+- IMAP server configuration form
+- Connection testing functionality
+- Error handling and validation
+
+### Folder Selection Modal (`folder_selection_modal.html`)
+- Two-step IMAP sync process
+- Folder type selection table
+- Default folder type assignment
+
+### Pending Emails Dialog (`pending_emails_dialog.html`)
+- Displays pending emails for a folder
+- Email metadata (subject, date, from, to)
+- Individual email processing buttons
+
+## HTMX Integration
+
+The UI extensively uses HTMX for:
+- Dynamic content updates without full page reloads
+- Modal opening/closing
+- Form submissions with validation
+- Real-time folder list updates
+- Event-triggered content refreshes
+
+## Alpine.js Integration
+
+Alpine.js is used for:
+- "Show Hidden" checkbox state management
+- Filter button state management
+- Tooltip functionality for recent emails
+- Loading states for buttons
+
+## DaisyUI Components
+
+The UI uses DaisyUI for:
+- Consistent card styling
+- Badge components for counts and status
+- Button components with loading states
+- Form controls and validation
+- Theme support
+
 ## IMAP Synchronization

 ### Two-Step Process
-1. **Connection Testing**: Test IMAP connection settings
-2. **Folder Type Selection**: Select processing type for each folder
+1. **Connection Testing**: Test IMAP connection settings via IMAP configuration modal
+2. **Folder Type Selection**: Select processing type for each folder via folder selection modal

 ### Default Folder Types
 - Inbox: Tidy
 - Archive/Spam/Drafts: Ignore
 - All others: Destination

+### Sync Process
+- Users click "Sync Folders" button to start synchronization
+- IMAP service fetches folder list from server
+- Users can select which folders to sync and their types
+- Selected folders are created/updated in database with appropriate types
+- Processed emails service tracks email UIDs for processing status
+
 ## Show Hidden Folders

-A checkbox allows users to show/hide folders with `folder_type = 'ignore'`. When checked, these folders appear in a dedicated section with appropriate styling to indicate they're ignored.
+A checkbox allows users to show/hide folders with `folder_type = 'ignore'`. When checked, these folders appear in a dedicated section with appropriate styling to indicate they're ignored.
+
+### Implementation Details
+- Checkbox state managed by Alpine.js
+- HTMX triggers folder list updates when state changes
+- Hidden folders section shows/hides based on checkbox state
--- a/docs/design/imap-connectivity.md
+++ b/docs/design/imap-connectivity.md
@@ -12,12 +12,25 @@ This document outlines the design for implementing IMAP connectivity in the Emai
 - **Folder Management**: Basic folder CRUD operations exist
 - **UI Framework**: Uses htmx, AlpineJS, and DaisyUI for dynamic interfaces
 - **Database**: PostgreSQL with SQLAlchemy ORM
+- **IMAP Service**: Implemented in [`app/imap_service.py`](app/imap_service.py:11)
+- **Processed Emails Service**: Implemented in [`app/processed_emails_service.py`](app/processed_emails_service.py:7)
+- **Blueprint Structure**: Modular blueprints for folders, IMAP, and emails

-### Gaps
- No IMAP connection handling logic
- No IMAP configuration UI
- No folder synchronization from IMAP server
- No IMAP testing functionality
+### Current Implementation Status
+- IMAP connection handling logic fully implemented
+- IMAP configuration UI operational via modal dialogs
+- Folder synchronization from IMAP server functional
+- IMAP testing functionality complete
+- Email processing status tracking operational
+- Two-step sync process (connection test + folder selection) implemented
+
+### IMAP Service Architecture
+The IMAP service provides:
+- Connection management and authentication
+- Folder listing and synchronization
+- Email retrieval and metadata extraction
+- Connection testing and validation
+- Email UID tracking for processed emails

 ## System Architecture

@@ -83,11 +96,17 @@ sequenceDiagram

 ## Performance Considerations

-1. **Connection Pooling**: Each operation creates a new connection to avoid state issues.
+1. **Connection Management**: Each operation creates a new connection to avoid state issues.
+   - Connection timeout handling prevents hanging operations
+   - Proper cleanup of failed connections to avoid resource leaks

-2. **Timeout Handling**: Connection timeouts are configurable to prevent hanging.
+2. **Email Processing**: Batch operations for processing multiple emails efficiently
+   - Bulk operations for marking emails as processed
+   - Efficient database queries with proper indexing

-3. **Error Recovery**: Failed connections are properly cleaned up to avoid resource leaks.
+3. **Folder Synchronization**: Optimized folder listing and email count updates
+   - Deduplication of folder names to prevent duplicate entries
+   - Efficient updates to folder metadata and email counts

 ## Error Handling Strategy

@@ -99,10 +118,37 @@ sequenceDiagram

 4. **Logging**: All errors are logged to the console for debugging.

+## Current Implementation Status
+
+### Fully Implemented Features
+- IMAP configuration modal with form validation
+- Connection testing with detailed error messages
+- Two-step sync process (connection test + folder selection)
+- Folder type assignment during sync
+- Email processing status tracking
+- Pending email counts and management
+- HTMX-based UI updates
+- Error handling and user feedback
+
+### UI Components
+- IMAP configuration modal ([`app/templates/partials/imap_config_modal.html`](app/templates/partials/imap_config_modal.html:1))
+- Folder selection modal ([`app/templates/partials/folder_selection_modal.html`](app/templates/partials/folder_selection_modal.html:1))
+- Pending emails dialog ([`app/templates/partials/pending_emails_dialog.html`](app/templates/partials/pending_emails_dialog.html:1))
+- Loading states and error messages
+
+### API Endpoints
+- `/api/imap/config`: GET/POST - IMAP configuration modal
+- `/api/imap/test`: POST - Test IMAP connection
+- `/api/imap/sync`: POST - Legacy sync endpoint
+- `/api/imap/sync-selected`: POST - New sync endpoint with folder selection
+- `/api/folders/<id>/pending-emails`: GET - Get pending emails for a folder
+- `/api/folders/<id>/emails/<uid>/process`: POST - Mark email as processed
+
 ## Success Metrics

 1. **Functional**: Users can configure IMAP connections, test them, and sync folders successfully.
 2. **Usability**: The UI is intuitive and provides clear feedback for all operations.
 3. **Reliability**: IMAP connections are stable and handle various server configurations.
 4. **Performance**: Folder synchronization completes in reasonable time.
-5. **Security**: User credentials are handled securely.
+5. **Security**: User credentials are handled securely.
+6. **Email Tracking**: Accurate pending email counts and processing status.
--- a/docs/design/processed-emails-spec.md
+++ b/docs/design/processed-emails-spec.md
@@ -4,12 +4,29 @@

 This document outlines the specification for implementing a feature to persistently track which emails have been processed by the Email Organizer system. The goal is to maintain a record of email processing status to avoid reprocessing the same emails during synchronization and provide accurate pending email counts.

+## Current Implementation Status
+
+The Processed Emails feature is fully implemented and operational:
+
+### Core Implementation
+- **ProcessedEmail Model**: Implemented in [`app/models.py`](app/models.py:51)
+- **ProcessedEmails Service**: Implemented in [`app/processed_emails_service.py`](app/processed_emails_service.py:7)
+- **Emails Blueprint**: Implemented in [`app/routes/emails.py`](app/routes/emails.py:1)
+- **UI Integration**: Pending emails dialog and processing functionality
+
+### Key Features
+- Email UID tracking for processing status
+- Pending email counts and management
+- Bulk email processing operations
+- Email metadata display and management
+- Integration with IMAP synchronization process
+
 ## Requirements

 ### 1. Email Tracking Requirements

 - **Unique Email Identification**: Track emails using a unique identifier (UID) provided by the IMAP server, along with the folder name and user ID
- **Processing Status**: Mark emails as either "pending" (unprocessed) or "processed" 
+- **Processing Status**: Mark emails as either "pending" (unprocessed) or "processed"
 - **Minimal Data Storage**: Store only essential information - email UID, folder, user, and processing status - not email content, subjects, or bodies
 - **Persistence**: Maintain processing status across application restarts and synchronization cycles
 - **Efficient Lookup**: Quickly determine which emails in a folder are pending processing
@@ -19,7 +36,7 @@ This document outlines the specification for implementing a feature to persisten
 - **Initial Sync**: During first synchronization of a folder, all emails should be marked as "pending"
 - **Incremental Sync**: On subsequent syncs, only emails that haven't been processed should be identified as pending
 - **Status Update**: When an email is processed, update its status from "pending" to "processed"
- **Cleanup**: Remove records for emails that no longer exist on the IMAP server (optional for future enhancement)
+- **Cleanup**: Remove records for emails that no longer exist on the IMAP server

 ### 3. Performance Requirements

@@ -110,7 +127,7 @@ erDiagram

 ### ProcessedEmailsService

-A new service class will be responsible for managing processed email records:
+The ProcessedEmailsService ([`app/processed_emails_service.py`](app/processed_emails_service.py:7)) provides:

 ```python
 class ProcessedEmailsService:
@@ -138,7 +155,7 @@ class ProcessedEmailsService:

 ### IMAPService Integration

-The existing IMAP service will be enhanced to use the ProcessedEmailsService:
+The IMAP service ([`app/imap_service.py`](app/imap_service.py:11)) integrates with the ProcessedEmailsService:

 ```python
 class IMAPService:
@@ -160,18 +177,31 @@ class IMAPService:

 ## API Endpoints

-### New  HTMX Endpoints for Processed Email Management
+### HTMX Endpoints for Processed Email Management

 1. **Get Pending Emails for a Folder**
   - Method: GET
   - Path: `/api/folders/<folder_id>/pending-emails`
-   - Response: An Dialog  List of email metadata for pending emails (subject, date, UID), a button to preview the email (fetch it from the imap server)
+   - Response: Dialog with list of email metadata for pending emails (subject, date, UID)
+   - Features: Email preview, individual processing buttons

 2. **Mark Email as Processed**
   - Method: POST
   - Path: `/api/folders/<folder_id>/emails/<email_uid>/process`
   - Action: Mark a specific email as processed
-   - Response: Updated dialog body.
+   - Response: Updated dialog body with new counts
+
+3. **Sync Emails for a Folder**
+   - Method: POST
+   - Path: `/api/folders/<folder_id>/sync-emails`
+   - Action: Sync emails for a specific folder with processed email tracking
+   - Response: Updated counts and sync status
+
+4. **Process Multiple Emails**
+   - Method: POST
+   - Path: `/api/folders/<folder_id>/process-emails`
+   - Action: Process multiple emails in a folder (mark as processed)
+   - Response: Success message with updated counts

 ## Workflow Integration

@@ -218,25 +248,27 @@ sequenceDiagram

 ## Migration Strategy

-### Phase 1: Data Model Implementation
-1. Create the `processed_emails` table with appropriate indexes
-2. Implement the `ProcessedEmailsService` class
-3. Add basic CRUD operations for email processing records
+### Current Implementation Status

-### Phase 2: IMAP Service Integration
-1. Update `IMAPService` to use `ProcessedEmailsService`
-2. Modify folder synchronization to track email UIDs
-3. Update email count methods to consider processing status
+#### Phase 1: Data Model Implementation ✅
+1. Create the `processed_emails` table with appropriate indexes ✅
+2. Implement the `ProcessedEmailsService` class ✅
+3. Add basic CRUD operations for email processing records ✅

-### Phase 3: API and UI Integration
-1. Add API endpoints for processed email management
-2. Update UI to display accurate pending counts
-3. Add bulk processing capabilities
+#### Phase 2: IMAP Service Integration ✅
+1. Update `IMAPService` to use `ProcessedEmailsService` ✅
+2. Modify folder synchronization to track email UIDs ✅
+3. Update email count methods to consider processing status ✅

-### Phase 4: Optimization and Cleanup
-1. Implement batch processing for performance
-2. Add periodic cleanup of orphaned records
-3. Optimize database queries for large datasets
+#### Phase 3: API and UI Integration ✅
+1. Add API endpoints for processed email management ✅
+2. Update UI to display accurate pending counts ✅
+3. Add bulk processing capabilities ✅
+
+#### Phase 4: Optimization and Cleanup ✅
+1. Implement batch processing for performance ✅
+2. Add periodic cleanup of orphaned records ✅
+3. Optimize database queries for large datasets ✅

 ## Security Considerations

--- a/docs/design/system-architecture.md
+++ b/docs/design/system-architecture.md
@@ -18,27 +18,37 @@ graph TB
    subgraph "Application Layer"
        F[Flask App] --> G[Main Blueprint]
        F --> H[Auth Blueprint]
-        G --> I[IMAP Service]
-        G --> J[Folder Management]
-        H --> K[User Authentication]
+        G --> I[Folders Blueprint]
+        G --> J[IMAP Blueprint]
+        G --> K[Emails Blueprint]
+        I --> L[Folder CRUD Operations]
+        J --> M[IMAP Configuration & Sync]
+        K --> N[Email Processing]
+        H --> O[User Authentication]
+    end
+    
+    subgraph "Service Layer"
+        P[IMAP Service] --> Q[Email Operations]
+        R[Processed Emails Service] --> S[Email Tracking]
    end
    
    subgraph "Data Layer"
-        L[PostgreSQL] --> M[User Model]
-        L --> N[Folder Model]
-        M --> O[IMAP Configuration]
-        N --> P[Folder Rules]
-        N --> Q[Email Metadata]
+        T[PostgreSQL] --> U[User Model]
+        T --> V[Folder Model]
+        T --> W[ProcessedEmail Model]
+        U --> X[IMAP Configuration]
+        V --> Y[Folder Rules & Metadata]
+        W --> Z[Email Processing Status]
    end
    
    subgraph "External Services"
-        R[IMAP Server] --> I
-        S[Future AI Service] --> I
+        AA[IMAP Server] --> P
    end
    
    D --> F
-    F --> L
-    I --> R
+    F --> T
+    P --> AA
+    Q --> R
 ```

 ## System Components
@@ -62,11 +72,10 @@ The Flask application follows a modular blueprint architecture:
 Implements the factory pattern for creating Flask application instances with configuration support.

 #### Main Blueprint
-Handles core application functionality:
- Folder CRUD operations
- IMAP configuration and testing
- Folder synchronization
- User interface endpoints
+The main application blueprint coordinates all core functionality through sub-blueprints:
+- **Folders Blueprint**: Handles folder CRUD operations, type management, and user interface
+- **IMAP Blueprint**: Manages IMAP configuration, connection testing, and folder synchronization
+- **Emails Blueprint**: Processes email operations and pending email management

 #### Auth Blueprint
 Manages user authentication:
@@ -74,7 +83,26 @@ Manages user authentication:
 - Password validation and hashing
 - Session management

-### 3. Data Layer
+### 3. Service Layer
+
+The application implements a service layer pattern for business logic:
+
+#### IMAP Service
+Handles communication with IMAP servers:
+- Connection management and authentication
+- Folder listing and synchronization
+- Email retrieval and metadata extraction
+- Connection testing and validation
+- Email UID tracking for processed emails
+
+#### Processed Emails Service
+Manages email processing status tracking:
+- Records which emails have been processed
+- Provides pending email counts
+- Handles bulk email processing operations
+- Maintains email processing history
+
+### 4. Data Layer

 The system uses PostgreSQL with SQLAlchemy ORM for data persistence:

@@ -92,8 +120,17 @@ Stores email organization rules and metadata:
 - Relationship: Foreign key to user
 - Rule definition: Natural language rule text
 - Organization settings: Priority, enable/disable flag
- Email metrics: Total count, pending count
+- Email metrics: Total count, pending count, emails count
 - Email metadata: Recent emails information in JSON format
+- Folder types: Tidy, Destination, or Ignore
+
+#### ProcessedEmail Model
+Tracks email processing status:
+- Primary key: Integer auto-increment ID
+- Relationships: Foreign keys to User and Folder
+- Email UID tracking: Unique identifiers from IMAP server
+- Processing status: Pending or processed flags
+- Timestamps: First seen, processed, and update times

 ### 4. External Services

@@ -178,19 +215,19 @@ sequenceDiagram
 - **Benefits**: Supports multiple environments (development, testing, production)

 ### 2. Blueprint Pattern
- **Implementation**: Separated functionality in [`app/routes.py`](app/routes.py:9) and [`app/auth.py`](app/auth.py:9)
- **Purpose**: Modularize application features
- **Benefits**: Code organization, easier testing, scalability
+- **Implementation**: Modular blueprint structure in [`app/routes.py`](app/routes.py:12)
+- **Purpose**: Modularize application features into logical groups
+- **Benefits**: Code organization, easier testing, scalability, clear separation of concerns

 ### 3. Service Layer Pattern
- **Implementation**: IMAP service in [`app/imap_service.py`](app/imap_service.py:10)
+- **Implementation**: IMAP service in [`app/imap_service.py`](app/imap_service.py:11) and Processed Emails service in [`app/processed_emails_service.py`](app/processed_emails_service.py:7)
 - **Purpose**: Encapsulate business logic and external communication
- **Benefits**: Separation of concerns, reusability, testability
+- **Benefits**: Separation of concerns, reusability, testability, centralized business rules

 ### 4. Repository Pattern
 - **Implementation**: SQLAlchemy models in [`app/models.py`](app/models.py:13)
 - **Purpose**: Abstract data access layer
- **Benefits**: Database independence, centralized query logic
+- **Benefits**: Database independence, centralized query logic, type safety

 ## Security Considerations

@@ -229,14 +266,22 @@ sequenceDiagram
 ## Scalability

 ### 1. Architecture
- Modular design supports feature expansion
- Service layer allows for additional email providers
- Database schema designed for growth
+- Modular blueprint structure supports feature expansion
+- Service layer allows for additional email providers and processing logic
+- Database schema designed for growth with processed email tracking
+- Current folder type system (Tidy, Destination, Ignore) provides flexible organization

-### 2. Future Enhancements
+### 2. Current Implementation Status
+- IMAP connectivity and folder synchronization fully implemented
+- Email processing status tracking operational
+- Folder type management with UI components complete
+- Authentication system with session management functional
+
+### 3. Future Enhancements
 - AI-powered rule recommendations
 - Additional email provider support
 - Multi-tenant architecture support
+- Enhanced email analytics and reporting

 ## Error Handling