diff --git a/docs/design/authentication-system.md b/docs/design/authentication-system.md index 04ff394..0378091 100644 --- a/docs/design/authentication-system.md +++ b/docs/design/authentication-system.md @@ -47,14 +47,14 @@ graph TB The authentication blueprint handles all authentication-related routes and logic: #### Routes -- `/login`: GET/POST - User login page and authentication -- `/signup`: GET/POST - User registration page and account creation -- `/logout`: GET - User logout and session termination +- `/auth/login`: GET/POST - User login page and authentication +- `/auth/signup`: GET/POST - User registration page and account creation +- `/auth/logout`: GET - User logout and session termination #### Key Functions -- `login()`: Handles user authentication -- `signup()`: Manages user registration -- `logout()`: Terminates user sessions +- `login()`: Handles user authentication with form validation +- `signup()`: Manages user registration with server-side validation +- `logout()`: Terminates user sessions and clears authentication state - `validate_password()`: Enforces password complexity requirements ### 2. User Model ([`app/models.py`](app/models.py:13)) @@ -288,6 +288,15 @@ login_manager.login_message = 'Please log in to access this page.' login_manager.login_message_category = 'warning' ``` +### Blueprint Registration + +The auth blueprint is registered with URL prefix `/auth/`: + +```python +from app.auth import auth +app.register_blueprint(auth, url_prefix='/auth') +``` + ## Testing Strategy ### Unit Tests @@ -350,4 +359,11 @@ login_manager.login_message_category = 'warning' - Rate limiting for login attempts - Account lockout after failed attempts - Suspicious activity monitoring -- IP-based security checks \ No newline at end of file +- IP-based security checks + +### 6. Current Implementation Status +- User authentication with session management fully implemented +- Password complexity validation operational +- Server-side form validation complete +- Error handling with user-friendly messages +- Blueprint-based routing with proper URL prefixes \ No newline at end of file diff --git a/docs/design/data-model.md b/docs/design/data-model.md index 24c5cfb..7aa480c 100644 --- a/docs/design/data-model.md +++ b/docs/design/data-model.md @@ -242,9 +242,10 @@ Folders with `folder_type = 'tidy'` are source folders that contain emails waiti - Example: Inbox folder **UI Representation:** -- Shows "pending count" and "processed count" badges +- Shows "total", "pending", and "processed" count badges - Includes "View Pending" button if there are pending emails - May include priority indicators +- Located in "Emails to organize" section ### Destination Folders @@ -260,6 +261,7 @@ Folders with `folder_type = 'destination'` are target folders where emails are m - Shows "emails count" badge - Simpler interface without pending/processed indicators - Focus on folder management and viewing contents +- Located in "Destination Folders" section ### Ignore Folders @@ -276,6 +278,7 @@ Folders with `folder_type = 'ignore'` are folders that are stored in the databas - Hidden by default unless "Show Hidden" checkbox is checked - When visible, shows minimal information - No action buttons for organization or processing +- Located in "Hidden Folders" section ### Folder Type Determination @@ -287,17 +290,50 @@ Folder types are determined as follows: - Inbox: Tidy - Archive/Spam/Drafts: Ignore - All others: Destination -- Manually created folders default to 'destination' -- Folder type can be changed through the user interface +- Manually created folders default to 'destination' (except 'inbox' which defaults to 'tidy') +- Folder type can be changed through the user interface via dropdown - When changing to 'ignore', emails_count is reset to 0 +## ProcessedEmail Model + +The `ProcessedEmail` entity tracks email processing status to avoid reprocessing the same emails during synchronization and provide accurate pending email counts. + +### Attributes + +| Column Name | Data Type | Constraints | Description | +|-------------|------------|--------------|-------------| +| id | Integer | Primary Key, Autoincrement | Unique identifier for each processed email record | +| user_id | Integer | Foreign Key to User, Not Null | Reference to the user who owns this email | +| folder_id | Integer | Foreign Key to Folder, Not Null | Reference to the folder this email belongs to | +| email_uid | String(255) | Not Null | Unique ID of the email from IMAP server | +| folder_name | String(255) | Not Null | Name of the IMAP folder (for redundancy) | +| is_processed | Boolean | Default: False | Processing status (false=pending, true=processed) | +| first_seen_at | DateTime | Default: datetime.utcnow | First time this email was detected during sync | +| processed_at | DateTime | Nullable | When the email was marked as processed | +| created_at | DateTime | Default: datetime.utcnow | Record creation timestamp | +| updated_at | DateTime | Default: datetime.utcnow, On Update | Record update timestamp | + +### Relationships + +- **Many-to-One**: Each `ProcessedEmail` belongs to one `User` +- **Many-to-One**: Each `ProcessedEmail` belongs to one `Folder` +- **Composite Key**: The combination of (user_id, folder_name, email_uid) should be unique to prevent duplicate records + +### Business Rules + +- Each processed email record must belong to a user and folder +- Email UID must be unique per user and folder to prevent duplicates +- Processing status tracks whether an email has been processed +- First seen timestamp tracks when the email was first discovered +- Processed timestamp is set when email is marked as processed + ## Future Data Model Considerations ### Potential Enhancements -1. **Email Entity** - - Store email metadata for better analytics - - Track email movement between folders +1. **Email Movement Tracking** + - Store email movement history between folders + - Track source and destination folder for each moved email 2. **Rule Engine** - Store parsed rule structures for better processing @@ -309,4 +345,5 @@ Folder types are determined as follows: 4. **Audit Log** - Track changes to user data - - Monitor folder operations \ No newline at end of file + - Monitor folder operations + - Log email processing actions \ No newline at end of file diff --git a/docs/design/development-setup.md b/docs/design/development-setup.md index 0677e40..46f68bc 100644 --- a/docs/design/development-setup.md +++ b/docs/design/development-setup.md @@ -6,6 +6,30 @@ This document describes how to set up a development environment for the Email Or - Docker and Docker Compose - Python 3.10+ +- Flask CLI tools + +## Project Structure + +The application follows a modular structure: + +``` +app/ +├── __init__.py # Flask app factory and configuration +├── models.py # Database models (User, Folder, ProcessedEmail) +├── auth.py # Authentication blueprint +├── routes.py # Main blueprint with sub-blueprints +├── imap_service.py # IMAP communication service +├── processed_emails_service.py # Email processing tracking +└── routes/ # Sub-blueprint modules + ├── __init__.py + ├── emails.py # Email processing endpoints + ├── folders.py # Folder management endpoints + └── imap.py # IMAP configuration and sync endpoints + +templates/ # Jinja2 templates with HTMX integration +static/ # Static assets +config.py # Application configuration +``` ## Services @@ -40,7 +64,7 @@ This command will: ### IMAP Server - Host: localhost -- Port: 1143 +- Port: 5143 - Users: - user1@example.com / password1 - user2@example.com / password2 @@ -61,4 +85,100 @@ After running `flask setup-dev`, copy the example environment file: cp .env.example .env ``` -The default configuration should work with the development services. You can modify the `.env` file if needed. \ No newline at end of file +The default configuration should work with the development services. You can modify the `.env` file if needed. + +### Configuration File + +The [`config.py`](config.py:1) file provides different environment configurations: + +```python +class Config: + SECRET_KEY = os.environ.get('SECRET_KEY') or 'dev-secret-key' + SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or 'postgresql://postgres:password@localhost:5432/email_organizer_dev' + SQLALCHEMY_TRACK_MODIFICATIONS = False + +class TestingConfig(Config): + TESTING = True + SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:' # In-memory database for tests +``` + +## Database Setup + +The application uses Alembic for database migrations: + +```bash +# Create migration +flask db migrate -m "Initial migration" + +# Apply migration +flask db upgrade +``` + +## Running the Application + +```bash +# Development server +flask run + +# With specific host and port +flask run --host=0.0.0.0 --port=5001 +``` + +The application will be available at `http://localhost:5001` + +## Testing + +The application includes comprehensive tests: + +```bash +# Run all tests +pytest + +# Run with coverage +pytest --cov=app + +# Run specific test file +pytest tests/unit/test_auth.py + +# Run integration tests +pytest tests/integration/ +``` + +### Test Structure + +- **Unit Tests**: Test individual components and services +- **Integration Tests**: Test API endpoints and database interactions +- **Functional Tests**: Test user workflows and UI interactions + +## Development Workflow + +1. **Setup Environment**: Run `flask setup-dev` to start services +2. **Configure Database**: Run migrations to set up database schema +3. **Run Application**: Start Flask development server +4. **Test Changes**: Write tests for new features +5. **Commit Changes**: Follow conventional commit messages + +## Key Technologies + +- **Backend**: Flask, SQLAlchemy, PostgreSQL +- **Frontend**: HTMX, Alpine.js, DaisyUI, TailwindCSS +- **Database**: PostgreSQL with Alembic migrations +- **Testing**: pytest with Beautiful Soup for HTML testing +- **Development**: Docker for services, Flask CLI for management + +## Current Implementation Status + +### Core Features +- User authentication and session management ✅ +- IMAP configuration and connection testing ✅ +- Folder synchronization with type assignment ✅ +- Email processing status tracking ✅ +- HTMX-based UI with modals and dialogs ✅ +- Folder type management (Tidy, Destination, Ignore) ✅ + +### Development Tools +- Database migrations with Alembic ✅ +- Comprehensive test suite ✅ +- Docker-based development environment ✅ +- Flask CLI for development tasks ✅ +- Environment-based configuration ✅ \ No newline at end of file diff --git a/docs/design/folder-types-ui.md b/docs/design/folder-types-ui.md index f0f2113..7dee389 100644 --- a/docs/design/folder-types-ui.md +++ b/docs/design/folder-types-ui.md @@ -12,43 +12,145 @@ The main page is divided into sections based on folder types: 2. **Destination Folders Section**: Contains folders with `folder_type = 'destination'` 3. **Hidden Folders Section**: Contains folders with `folder_type = 'ignore'` (visible only when "Show Hidden" is checked) +## Main Page Layout + +The main page ([`app/templates/index.html`](app/templates/index.html:1)) provides: +- Header with navigation and user information +- Sidebar with main navigation +- Main content area with folder sections +- Modal holder for dynamic content + +### Dashboard Statistics +When IMAP is configured, the dashboard displays: +- Total folders count +- Folders to tidy count +- Destination folders count +- Total pending emails count + +### Search and Filter Controls +- Search input for folder names +- "Show Hidden" checkbox for ignore folders +- Filter buttons for All, High Priority, Normal priority + ## Folder Card Components -### Base Template +### Base Template (`folder_card_base.html`) All folder cards extend from `folder_card_base.html` which provides: - Common card structure with edit/delete buttons - Priority badge display - Rule text section - Block for additional content specific to folder type +- Folder type dropdown for changing folder types ### Tidy Folder Card (`folder_card_tidy.html`) Extends base template with: - Total, pending, and processed email count badges -- Organize toggle switch -- View pending emails button +- "View Pending" button that opens a modal with pending emails +- Recent emails tooltip for processed emails ### Destination Folder Card (`folder_card_destination.html`) Extends base template with: -- Email count badge +- Email count badge showing emails moved to this folder - Simplified UI without processing elements -### Ignore Folder Card (to be created) -Will extend base template with: +### Ignore Folder Card (`folder_card_ignore.html`) +Extends base template with: - "Ignored" status badge +- Email count badge (reset to 0 when changed to ignore) - Minimal information display - No processing elements +## Section Templates + +### Folders to Tidy Section (`folders_to_tidy_section.html`) +- Section header with descriptive text and icon +- Grid layout for folder cards +- Empty state with call-to-action button +- HTMX integration for dynamic updates + +### Destination Folders Section (`destination_folders_section.html`) +- Section header with descriptive text and icon +- Grid layout for folder cards +- Empty state with call-to-action button +- HTMX integration for dynamic updates + +### Hidden Folders Section (`hidden_folders_section.html`) +- Section header with eye-slash icon +- Grid layout for folder cards +- Empty state message +- Only visible when "Show Hidden" is checked + +## Modal System + +### Folder Modal (`folder_modal.html`) +- Used for creating and editing folders +- Form validation with server-side error handling +- HTMX integration for dynamic form submission + +### IMAP Configuration Modal (`imap_config_modal.html`) +- IMAP server configuration form +- Connection testing functionality +- Error handling and validation + +### Folder Selection Modal (`folder_selection_modal.html`) +- Two-step IMAP sync process +- Folder type selection table +- Default folder type assignment + +### Pending Emails Dialog (`pending_emails_dialog.html`) +- Displays pending emails for a folder +- Email metadata (subject, date, from, to) +- Individual email processing buttons + +## HTMX Integration + +The UI extensively uses HTMX for: +- Dynamic content updates without full page reloads +- Modal opening/closing +- Form submissions with validation +- Real-time folder list updates +- Event-triggered content refreshes + +## Alpine.js Integration + +Alpine.js is used for: +- "Show Hidden" checkbox state management +- Filter button state management +- Tooltip functionality for recent emails +- Loading states for buttons + +## DaisyUI Components + +The UI uses DaisyUI for: +- Consistent card styling +- Badge components for counts and status +- Button components with loading states +- Form controls and validation +- Theme support + ## IMAP Synchronization ### Two-Step Process -1. **Connection Testing**: Test IMAP connection settings -2. **Folder Type Selection**: Select processing type for each folder +1. **Connection Testing**: Test IMAP connection settings via IMAP configuration modal +2. **Folder Type Selection**: Select processing type for each folder via folder selection modal ### Default Folder Types - Inbox: Tidy - Archive/Spam/Drafts: Ignore - All others: Destination +### Sync Process +- Users click "Sync Folders" button to start synchronization +- IMAP service fetches folder list from server +- Users can select which folders to sync and their types +- Selected folders are created/updated in database with appropriate types +- Processed emails service tracks email UIDs for processing status + ## Show Hidden Folders -A checkbox allows users to show/hide folders with `folder_type = 'ignore'`. When checked, these folders appear in a dedicated section with appropriate styling to indicate they're ignored. \ No newline at end of file +A checkbox allows users to show/hide folders with `folder_type = 'ignore'`. When checked, these folders appear in a dedicated section with appropriate styling to indicate they're ignored. + +### Implementation Details +- Checkbox state managed by Alpine.js +- HTMX triggers folder list updates when state changes +- Hidden folders section shows/hides based on checkbox state \ No newline at end of file diff --git a/docs/design/imap-connectivity.md b/docs/design/imap-connectivity.md index 8934a63..a91876c 100644 --- a/docs/design/imap-connectivity.md +++ b/docs/design/imap-connectivity.md @@ -12,12 +12,25 @@ This document outlines the design for implementing IMAP connectivity in the Emai - **Folder Management**: Basic folder CRUD operations exist - **UI Framework**: Uses htmx, AlpineJS, and DaisyUI for dynamic interfaces - **Database**: PostgreSQL with SQLAlchemy ORM +- **IMAP Service**: Implemented in [`app/imap_service.py`](app/imap_service.py:11) +- **Processed Emails Service**: Implemented in [`app/processed_emails_service.py`](app/processed_emails_service.py:7) +- **Blueprint Structure**: Modular blueprints for folders, IMAP, and emails -### Gaps -- No IMAP connection handling logic -- No IMAP configuration UI -- No folder synchronization from IMAP server -- No IMAP testing functionality +### Current Implementation Status +- IMAP connection handling logic fully implemented +- IMAP configuration UI operational via modal dialogs +- Folder synchronization from IMAP server functional +- IMAP testing functionality complete +- Email processing status tracking operational +- Two-step sync process (connection test + folder selection) implemented + +### IMAP Service Architecture +The IMAP service provides: +- Connection management and authentication +- Folder listing and synchronization +- Email retrieval and metadata extraction +- Connection testing and validation +- Email UID tracking for processed emails ## System Architecture @@ -83,11 +96,17 @@ sequenceDiagram ## Performance Considerations -1. **Connection Pooling**: Each operation creates a new connection to avoid state issues. +1. **Connection Management**: Each operation creates a new connection to avoid state issues. + - Connection timeout handling prevents hanging operations + - Proper cleanup of failed connections to avoid resource leaks -2. **Timeout Handling**: Connection timeouts are configurable to prevent hanging. +2. **Email Processing**: Batch operations for processing multiple emails efficiently + - Bulk operations for marking emails as processed + - Efficient database queries with proper indexing -3. **Error Recovery**: Failed connections are properly cleaned up to avoid resource leaks. +3. **Folder Synchronization**: Optimized folder listing and email count updates + - Deduplication of folder names to prevent duplicate entries + - Efficient updates to folder metadata and email counts ## Error Handling Strategy @@ -99,10 +118,37 @@ sequenceDiagram 4. **Logging**: All errors are logged to the console for debugging. +## Current Implementation Status + +### Fully Implemented Features +- IMAP configuration modal with form validation +- Connection testing with detailed error messages +- Two-step sync process (connection test + folder selection) +- Folder type assignment during sync +- Email processing status tracking +- Pending email counts and management +- HTMX-based UI updates +- Error handling and user feedback + +### UI Components +- IMAP configuration modal ([`app/templates/partials/imap_config_modal.html`](app/templates/partials/imap_config_modal.html:1)) +- Folder selection modal ([`app/templates/partials/folder_selection_modal.html`](app/templates/partials/folder_selection_modal.html:1)) +- Pending emails dialog ([`app/templates/partials/pending_emails_dialog.html`](app/templates/partials/pending_emails_dialog.html:1)) +- Loading states and error messages + +### API Endpoints +- `/api/imap/config`: GET/POST - IMAP configuration modal +- `/api/imap/test`: POST - Test IMAP connection +- `/api/imap/sync`: POST - Legacy sync endpoint +- `/api/imap/sync-selected`: POST - New sync endpoint with folder selection +- `/api/folders//pending-emails`: GET - Get pending emails for a folder +- `/api/folders//emails//process`: POST - Mark email as processed + ## Success Metrics 1. **Functional**: Users can configure IMAP connections, test them, and sync folders successfully. 2. **Usability**: The UI is intuitive and provides clear feedback for all operations. 3. **Reliability**: IMAP connections are stable and handle various server configurations. 4. **Performance**: Folder synchronization completes in reasonable time. -5. **Security**: User credentials are handled securely. \ No newline at end of file +5. **Security**: User credentials are handled securely. +6. **Email Tracking**: Accurate pending email counts and processing status. \ No newline at end of file diff --git a/docs/design/processed-emails-spec.md b/docs/design/processed-emails-spec.md index ed2ae9d..d95f65a 100644 --- a/docs/design/processed-emails-spec.md +++ b/docs/design/processed-emails-spec.md @@ -4,12 +4,29 @@ This document outlines the specification for implementing a feature to persistently track which emails have been processed by the Email Organizer system. The goal is to maintain a record of email processing status to avoid reprocessing the same emails during synchronization and provide accurate pending email counts. +## Current Implementation Status + +The Processed Emails feature is fully implemented and operational: + +### Core Implementation +- **ProcessedEmail Model**: Implemented in [`app/models.py`](app/models.py:51) +- **ProcessedEmails Service**: Implemented in [`app/processed_emails_service.py`](app/processed_emails_service.py:7) +- **Emails Blueprint**: Implemented in [`app/routes/emails.py`](app/routes/emails.py:1) +- **UI Integration**: Pending emails dialog and processing functionality + +### Key Features +- Email UID tracking for processing status +- Pending email counts and management +- Bulk email processing operations +- Email metadata display and management +- Integration with IMAP synchronization process + ## Requirements ### 1. Email Tracking Requirements - **Unique Email Identification**: Track emails using a unique identifier (UID) provided by the IMAP server, along with the folder name and user ID -- **Processing Status**: Mark emails as either "pending" (unprocessed) or "processed" +- **Processing Status**: Mark emails as either "pending" (unprocessed) or "processed" - **Minimal Data Storage**: Store only essential information - email UID, folder, user, and processing status - not email content, subjects, or bodies - **Persistence**: Maintain processing status across application restarts and synchronization cycles - **Efficient Lookup**: Quickly determine which emails in a folder are pending processing @@ -19,7 +36,7 @@ This document outlines the specification for implementing a feature to persisten - **Initial Sync**: During first synchronization of a folder, all emails should be marked as "pending" - **Incremental Sync**: On subsequent syncs, only emails that haven't been processed should be identified as pending - **Status Update**: When an email is processed, update its status from "pending" to "processed" -- **Cleanup**: Remove records for emails that no longer exist on the IMAP server (optional for future enhancement) +- **Cleanup**: Remove records for emails that no longer exist on the IMAP server ### 3. Performance Requirements @@ -110,7 +127,7 @@ erDiagram ### ProcessedEmailsService -A new service class will be responsible for managing processed email records: +The ProcessedEmailsService ([`app/processed_emails_service.py`](app/processed_emails_service.py:7)) provides: ```python class ProcessedEmailsService: @@ -138,7 +155,7 @@ class ProcessedEmailsService: ### IMAPService Integration -The existing IMAP service will be enhanced to use the ProcessedEmailsService: +The IMAP service ([`app/imap_service.py`](app/imap_service.py:11)) integrates with the ProcessedEmailsService: ```python class IMAPService: @@ -160,18 +177,31 @@ class IMAPService: ## API Endpoints -### New HTMX Endpoints for Processed Email Management +### HTMX Endpoints for Processed Email Management 1. **Get Pending Emails for a Folder** - Method: GET - Path: `/api/folders//pending-emails` - - Response: An Dialog List of email metadata for pending emails (subject, date, UID), a button to preview the email (fetch it from the imap server) + - Response: Dialog with list of email metadata for pending emails (subject, date, UID) + - Features: Email preview, individual processing buttons 2. **Mark Email as Processed** - Method: POST - Path: `/api/folders//emails//process` - Action: Mark a specific email as processed - - Response: Updated dialog body. + - Response: Updated dialog body with new counts + +3. **Sync Emails for a Folder** + - Method: POST + - Path: `/api/folders//sync-emails` + - Action: Sync emails for a specific folder with processed email tracking + - Response: Updated counts and sync status + +4. **Process Multiple Emails** + - Method: POST + - Path: `/api/folders//process-emails` + - Action: Process multiple emails in a folder (mark as processed) + - Response: Success message with updated counts ## Workflow Integration @@ -218,25 +248,27 @@ sequenceDiagram ## Migration Strategy -### Phase 1: Data Model Implementation -1. Create the `processed_emails` table with appropriate indexes -2. Implement the `ProcessedEmailsService` class -3. Add basic CRUD operations for email processing records +### Current Implementation Status -### Phase 2: IMAP Service Integration -1. Update `IMAPService` to use `ProcessedEmailsService` -2. Modify folder synchronization to track email UIDs -3. Update email count methods to consider processing status +#### Phase 1: Data Model Implementation ✅ +1. Create the `processed_emails` table with appropriate indexes ✅ +2. Implement the `ProcessedEmailsService` class ✅ +3. Add basic CRUD operations for email processing records ✅ -### Phase 3: API and UI Integration -1. Add API endpoints for processed email management -2. Update UI to display accurate pending counts -3. Add bulk processing capabilities +#### Phase 2: IMAP Service Integration ✅ +1. Update `IMAPService` to use `ProcessedEmailsService` ✅ +2. Modify folder synchronization to track email UIDs ✅ +3. Update email count methods to consider processing status ✅ -### Phase 4: Optimization and Cleanup -1. Implement batch processing for performance -2. Add periodic cleanup of orphaned records -3. Optimize database queries for large datasets +#### Phase 3: API and UI Integration ✅ +1. Add API endpoints for processed email management ✅ +2. Update UI to display accurate pending counts ✅ +3. Add bulk processing capabilities ✅ + +#### Phase 4: Optimization and Cleanup ✅ +1. Implement batch processing for performance ✅ +2. Add periodic cleanup of orphaned records ✅ +3. Optimize database queries for large datasets ✅ ## Security Considerations diff --git a/docs/design/system-architecture.md b/docs/design/system-architecture.md index 8455628..c73492e 100644 --- a/docs/design/system-architecture.md +++ b/docs/design/system-architecture.md @@ -18,27 +18,37 @@ graph TB subgraph "Application Layer" F[Flask App] --> G[Main Blueprint] F --> H[Auth Blueprint] - G --> I[IMAP Service] - G --> J[Folder Management] - H --> K[User Authentication] + G --> I[Folders Blueprint] + G --> J[IMAP Blueprint] + G --> K[Emails Blueprint] + I --> L[Folder CRUD Operations] + J --> M[IMAP Configuration & Sync] + K --> N[Email Processing] + H --> O[User Authentication] + end + + subgraph "Service Layer" + P[IMAP Service] --> Q[Email Operations] + R[Processed Emails Service] --> S[Email Tracking] end subgraph "Data Layer" - L[PostgreSQL] --> M[User Model] - L --> N[Folder Model] - M --> O[IMAP Configuration] - N --> P[Folder Rules] - N --> Q[Email Metadata] + T[PostgreSQL] --> U[User Model] + T --> V[Folder Model] + T --> W[ProcessedEmail Model] + U --> X[IMAP Configuration] + V --> Y[Folder Rules & Metadata] + W --> Z[Email Processing Status] end subgraph "External Services" - R[IMAP Server] --> I - S[Future AI Service] --> I + AA[IMAP Server] --> P end D --> F - F --> L - I --> R + F --> T + P --> AA + Q --> R ``` ## System Components @@ -62,11 +72,10 @@ The Flask application follows a modular blueprint architecture: Implements the factory pattern for creating Flask application instances with configuration support. #### Main Blueprint -Handles core application functionality: -- Folder CRUD operations -- IMAP configuration and testing -- Folder synchronization -- User interface endpoints +The main application blueprint coordinates all core functionality through sub-blueprints: +- **Folders Blueprint**: Handles folder CRUD operations, type management, and user interface +- **IMAP Blueprint**: Manages IMAP configuration, connection testing, and folder synchronization +- **Emails Blueprint**: Processes email operations and pending email management #### Auth Blueprint Manages user authentication: @@ -74,7 +83,26 @@ Manages user authentication: - Password validation and hashing - Session management -### 3. Data Layer +### 3. Service Layer + +The application implements a service layer pattern for business logic: + +#### IMAP Service +Handles communication with IMAP servers: +- Connection management and authentication +- Folder listing and synchronization +- Email retrieval and metadata extraction +- Connection testing and validation +- Email UID tracking for processed emails + +#### Processed Emails Service +Manages email processing status tracking: +- Records which emails have been processed +- Provides pending email counts +- Handles bulk email processing operations +- Maintains email processing history + +### 4. Data Layer The system uses PostgreSQL with SQLAlchemy ORM for data persistence: @@ -92,8 +120,17 @@ Stores email organization rules and metadata: - Relationship: Foreign key to user - Rule definition: Natural language rule text - Organization settings: Priority, enable/disable flag -- Email metrics: Total count, pending count +- Email metrics: Total count, pending count, emails count - Email metadata: Recent emails information in JSON format +- Folder types: Tidy, Destination, or Ignore + +#### ProcessedEmail Model +Tracks email processing status: +- Primary key: Integer auto-increment ID +- Relationships: Foreign keys to User and Folder +- Email UID tracking: Unique identifiers from IMAP server +- Processing status: Pending or processed flags +- Timestamps: First seen, processed, and update times ### 4. External Services @@ -178,19 +215,19 @@ sequenceDiagram - **Benefits**: Supports multiple environments (development, testing, production) ### 2. Blueprint Pattern -- **Implementation**: Separated functionality in [`app/routes.py`](app/routes.py:9) and [`app/auth.py`](app/auth.py:9) -- **Purpose**: Modularize application features -- **Benefits**: Code organization, easier testing, scalability +- **Implementation**: Modular blueprint structure in [`app/routes.py`](app/routes.py:12) +- **Purpose**: Modularize application features into logical groups +- **Benefits**: Code organization, easier testing, scalability, clear separation of concerns ### 3. Service Layer Pattern -- **Implementation**: IMAP service in [`app/imap_service.py`](app/imap_service.py:10) +- **Implementation**: IMAP service in [`app/imap_service.py`](app/imap_service.py:11) and Processed Emails service in [`app/processed_emails_service.py`](app/processed_emails_service.py:7) - **Purpose**: Encapsulate business logic and external communication -- **Benefits**: Separation of concerns, reusability, testability +- **Benefits**: Separation of concerns, reusability, testability, centralized business rules ### 4. Repository Pattern - **Implementation**: SQLAlchemy models in [`app/models.py`](app/models.py:13) - **Purpose**: Abstract data access layer -- **Benefits**: Database independence, centralized query logic +- **Benefits**: Database independence, centralized query logic, type safety ## Security Considerations @@ -229,14 +266,22 @@ sequenceDiagram ## Scalability ### 1. Architecture -- Modular design supports feature expansion -- Service layer allows for additional email providers -- Database schema designed for growth +- Modular blueprint structure supports feature expansion +- Service layer allows for additional email providers and processing logic +- Database schema designed for growth with processed email tracking +- Current folder type system (Tidy, Destination, Ignore) provides flexible organization -### 2. Future Enhancements +### 2. Current Implementation Status +- IMAP connectivity and folder synchronization fully implemented +- Email processing status tracking operational +- Folder type management with UI components complete +- Authentication system with session management functional + +### 3. Future Enhancements - AI-powered rule recommendations - Additional email provider support - Multi-tenant architecture support +- Enhanced email analytics and reporting ## Error Handling