notid/email-organizer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

lots of configuration progress.

Browse Source
This commit is contained in:
Bryce
2025-08-06 15:38:49 -07:00
parent c6102dda45
commit 41ea8fb3bd
24 changed files with 2566 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
docs/design/data-model.md
View File

@@ -26,8 +26,10 @@ erDiagram
text rule_text "Natural Language Rule"
int priority "Processing Order"
boolean organize_enabled "Default: True"
string folder_type "Default: 'destination'"
int total_count "Default: 0"
int pending_count "Default: 0"
int emails_count "Default: 0"
json recent_emails "JSON Array"
datetime created_at "Default: UTC Now"
datetime updated_at "Default: UTC Now, On Update"
@@ -83,8 +85,10 @@ The `Folder` entity stores email organization rules and metadata for each user's
| rule_text | Text | Nullable | Natural language description of the folder rule |
| priority | Integer | Nullable | Processing order (0=normal, 1=high) |
| organize_enabled | Boolean | Default: True | Whether the organization rule is active |
| folder_type | String(20) | Default: 'destination' | Folder type: 'tidy' or 'destination' |
| total_count | Integer | Default: 0 | Total number of emails in the folder |
| pending_count | Integer | Default: 0 | Number of emails waiting to be processed |
| emails_count | Integer | Default: 0 | Number of emails moved to this destination folder |
| recent_emails | JSON | Default: [] | Array of recent email metadata |
| created_at | DateTime | Default: datetime.utcnow | Timestamp of folder creation |
| updated_at | DateTime | Default: datetime.utcnow, On Update | Timestamp of last update |
@@ -100,6 +104,9 @@ The `Folder` entity stores email organization rules and metadata for each user's
- Folder name must be unique per user
- Rule text can be null (for manually created folders)
- Priority values: 0 (normal), 1 (high priority)
- Folder types:
- 'tidy': Folders containing emails to be processed (e.g., Inbox)
- 'destination': Folders that are targets for email organization (default)
- Recent emails array stores JSON objects with subject and date information
## Data Constraints
@@ -128,7 +135,8 @@ The `Folder` entity stores email organization rules and metadata for each user's
- `User.created_at`, `User.updated_at`: Current UTC timestamp
- `Folder.created_at`, `Folder.updated_at`: Current UTC timestamp
- `Folder.organize_enabled`: True
- `Folder.total_count`, `Folder.pending_count`: 0
- `Folder.folder_type`: 'destination'
- `Folder.total_count`, `Folder.pending_count`, `Folder.emails_count`: 0
- `Folder.recent_emails`: Empty array
## JSON Data Structures
@@ -219,6 +227,49 @@ The `recent_emails` field stores an array of JSON objects:
- Batch updates for email counts
- JSON operations for recent emails metadata
## Folder Types
The system supports two distinct types of folders, each with different purposes and behaviors:
### Tidy Folders
Folders with `folder_type = 'tidy'` are source folders that contain emails waiting to be processed and organized.
**Characteristics:**
- Display pending and processed email counts
- Can have organization rules enabled/disabled
- Support viewing pending emails
- Example: Inbox folder
**UI Representation:**
- Shows "pending count" and "processed count" badges
- Includes "View Pending" button if there are pending emails
- May include priority indicators
### Destination Folders
Folders with `folder_type = 'destination'` are target folders where emails are moved from other folders during organization.
**Characteristics:**
- Display count of emails moved to this folder
- Typically don't have organization rules (or they're ignored)
- Focus on showing how many emails have been organized into them
- Example: "Projects", "Finance", "Personal" folders
**UI Representation:**
- Shows "emails count" badge
- Simpler interface without pending/processed indicators
- Focus on folder management and viewing contents
### Folder Type Determination
Folder types are determined as follows:
- During IMAP synchronization:
- Folders named "inbox" (case-insensitive) are automatically set as 'tidy'
- All other folders are set as 'destination'
- Manually created folders default to 'destination'
- Folder type can be changed through administrative functions
## Future Data Model Considerations
### Potential Enhancements

228
docs/design/folder-types-ui.md Normal file
View File

@@ -0,0 +1,228 @@
# Folder Types UI Design
## Overview
This document outlines the UI design changes needed to support the new folder types feature: "Folders to Tidy" and "Destination Folders". The UI will be reorganized to clearly separate these two types of folders into distinct sections.
## UI Structure
### Main Page Layout
The main page will be divided into two distinct sections:
1. **Folders to Tidy Section**
- Contains folders with `folder_type = 'tidy'`
- Typically includes the inbox and other source folders
- Shows pending and processed email counts
- Has actions for viewing and processing emails
2. **Destination Folders Section**
- Contains folders with `folder_type = 'destination'`
- Contains target folders for email organization
- Shows count of emails moved to each folder
- Focuses on folder management and viewing contents
### Visual Design
#### Section Headers
Each section will have a clear header with:
- Section title
- Brief description of the section's purpose
- Action button (e.g., "Add Folder" for destination folders)
#### Section Styling
- Different background colors or borders to visually distinguish sections
- Consistent spacing and layout within each section
- Responsive grid layout that adapts to screen size
## Folder Card Components
### Tidy Folder Card
For folders with `folder_type = 'tidy'`, the card will display:
#### Header
- Folder name (prominent)
- Edit and delete buttons
#### Count Badges
- Total emails count
- Pending emails count (with warning styling if > 0)
- Processed emails count (calculated as total - pending)
#### Actions
- "View Pending" button (if pending count > 0)
- Organize toggle switch
- Priority badge
#### Content
- Rule text description
- Recent email previews (if available)
### Destination Folder Card
For folders with `folder_type = 'destination'`, the card will display:
#### Header
- Folder name (prominent)
- Edit and delete buttons
#### Count Badge
- Emails count (number of emails moved to this folder)
#### Content
- Rule text description (if applicable)
#### Actions
- Basic folder management options
- No organize toggle (as it's not applicable)
## UI Components
### Section Templates
#### Folders to Tidy Section
```html
<div class="folders-to-tidy-section">
<div class="section-header">
<h2>Folders to Tidy</h2>
<p>Folders containing emails that need to be processed</p>
<button class="btn btn-primary" hx-get="/api/folders/new">
<i class="fas fa-plus"></i> Add Tidy Folder
</button>
</div>
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
{% for folder in tidy_folders %}
{% include 'partials/folder_card_tidy.html' %}
{% endfor %}
</div>
</div>
```
#### Destination Folders Section
```html
<div class="destination-folders-section">
<div class="section-header">
<h2>Destination Folders</h2>
<p>Folders where emails are organized and stored</p>
<button class="btn btn-primary" hx-get="/api/folders/new">
<i class="fas fa-plus"></i> Add Destination Folder
</button>
</div>
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
{% for folder in destination_folders %}
{% include 'partials/folder_card_destination.html' %}
{% endfor %}
</div>
</div>
```
### Folder Card Templates
#### Tidy Folder Card
```html
<div class="card bg-base-100 shadow-xl border border-base-300">
<div class="card-body">
<div class="flex justify-between items-start mb-2">
<h3 class="text-xl font-bold truncate">{{ folder.name }}</h3>
<div class="flex space-x-2">
<button class="btn btn-sm btn-outline" hx-get="/api/folders/{{ folder.id }}/edit">
<i class="fas fa-edit"></i>
</button>
<button class="btn btn-sm btn-outline btn-error" hx-delete="/api/folders/{{ folder.id }}">
<i class="fas fa-trash"></i>
</button>
</div>
</div>
<div class="flex space-x-2 mb-2">
<span class="badge badge-outline">{{ folder.total_count }} total</span>
{% if folder.pending_count > 0 %}
<button class="badge badge-warning" hx-get="/api/folders/{{ folder.id }}/pending-emails">
{{ folder.pending_count }} pending
</button>
{% else %}
<span class="badge badge-secondary">{{ folder.pending_count }} pending</span>
{% endif %}
<span class="badge badge-success">{{ folder.total_count - folder.pending_count }} processed</span>
</div>
<div class="bg-base-200 rounded-box p-3 mb-3">
<p class="text-sm">{{ folder.rule_text }}</p>
</div>
<div class="flex justify-between items-center">
<div class="flex items-center space-x-2">
<span class="text-xs">Organize:</span>
<input type="checkbox" class="toggle toggle-sm"
{% if folder.organize_enabled %}checked{% endif %}
hx-put="/api/folders/{{ folder.id }}/toggle">
</div>
{% if folder.priority == 1 %}
<span class="badge badge-error text-xs">High Priority</span>
{% endif %}
</div>
</div>
</div>
```
#### Destination Folder Card
```html
<div class="card bg-base-100 shadow-xl border border-base-300">
<div class="card-body">
<div class="flex justify-between items-start mb-2">
<h3 class="text-xl font-bold truncate">{{ folder.name }}</h3>
<div class="flex space-x-2">
<button class="btn btn-sm btn-outline" hx-get="/api/folders/{{ folder.id }}/edit">
<i class="fas fa-edit"></i>
</button>
<button class="btn btn-sm btn-outline btn-error" hx-delete="/api/folders/{{ folder.id }}">
<i class="fas fa-trash"></i>
</button>
</div>
</div>
<div class="flex space-x-2 mb-2">
<span class="badge badge-primary">{{ folder.emails_count }} emails</span>
</div>
<div class="bg-base-200 rounded-box p-3 mb-3">
<p class="text-sm">{{ folder.rule_text }}</p>
</div>
</div>
</div>
```
## Styling Guidelines
### Color Scheme
- **Tidy Folders**: Use warning colors (yellow/orange) for pending emails
- **Destination Folders**: Use primary colors (blue) for email counts
- Consistent use of DaisyUI badge classes for different states
### Responsive Design
- Grid layout adjusts from 1 column on mobile to 3 columns on desktop
- Cards should have consistent width across all screen sizes
- Section headers should remain readable on mobile
### Interactive Elements
- Buttons should have hover states
- Toggle switches should be clearly labeled
- Pending email count should be clickable to open dialog
## Implementation Notes
### Template Organization
- Create separate partial templates for each folder type
- Main index template will include both sections
- Use HTMX for dynamic updates without full page reloads
### Data Loading
- Fetch folders filtered by type from the backend
- Pass filtered lists to appropriate section templates
- Maintain existing functionality for folder operations
### Accessibility
- Ensure all interactive elements have proper ARIA labels
- Use semantic HTML5 elements for section structure
- Maintain keyboard navigation support

261
docs/design/processed-emails-spec.md Normal file
View File

@@ -0,0 +1,261 @@
# Processed Emails Feature Specification
## Overview
This document outlines the specification for implementing a feature to persistently track which emails have been processed by the Email Organizer system. The goal is to maintain a record of email processing status to avoid reprocessing the same emails during synchronization and provide accurate pending email counts.
## Requirements
### 1. Email Tracking Requirements
- **Unique Email Identification**: Track emails using a unique identifier (UID) provided by the IMAP server, along with the folder name and user ID
- **Processing Status**: Mark emails as either "pending" (unprocessed) or "processed"
- **Minimal Data Storage**: Store only essential information - email UID, folder, user, and processing status - not email content, subjects, or bodies
- **Persistence**: Maintain processing status across application restarts and synchronization cycles
- **Efficient Lookup**: Quickly determine which emails in a folder are pending processing
### 2. Synchronization Requirements
- **Initial Sync**: During first synchronization of a folder, all emails should be marked as "pending"
- **Incremental Sync**: On subsequent syncs, only emails that haven't been processed should be identified as pending
- **Status Update**: When an email is processed, update its status from "pending" to "processed"
- **Cleanup**: Remove records for emails that no longer exist on the IMAP server (optional for future enhancement)
### 3. Performance Requirements
- **Efficient Storage**: Use appropriate database indexing for fast lookups
- **Minimal Memory Usage**: Store only essential data to keep memory footprint low
- **Batch Processing**: Support batch operations for processing multiple emails efficiently
## Data Model Design
### ProcessedEmails Table
```mermaid
erDiagram
USER {
int id PK "Primary Key"
string email "Unique, Not Null"
string first_name "Not Null"
string last_name "Not Null"
string password_hash "Not Null"
json imap_config "JSON Configuration"
datetime created_at "Default: UTC Now"
datetime updated_at "Default: UTC Now, On Update"
}
FOLDER {
int id PK "Primary Key"
int user_id FK "Foreign Key to User"
string name "Not Null"
text rule_text "Natural Language Rule"
int priority "Processing Order"
boolean organize_enabled "Default: True"
int total_count "Default: 0"
int pending_count "Default: 0"
json recent_emails "JSON Array"
datetime created_at "Default: UTC Now"
datetime updated_at "Default: UTC Now, On Update"
}
PROCESSED_EMAIL {
int id PK "Primary Key"
int user_id FK "Foreign Key to User"
int folder_id FK "Foreign Key to Folder"
string email_uid "Not Null" "IMAP Email UID"
string folder_name "Not Null" "IMAP Folder Name"
boolean is_processed "Default: False" "Processing Status"
datetime first_seen_at "Default: UTC Now" "First seen during sync"
datetime processed_at "Nullable" "When email was processed"
datetime created_at "Default: UTC Now"
datetime updated_at "Default: UTC Now, On Update"
}
USER ||--o{ FOLDER : "has"
USER ||--o{ PROCESSED_EMAIL : "has"
FOLDER ||--o{ PROCESSED_EMAIL : "has"
```
### Column Specifications
| Table | Column | Data Type | Constraints | Description |
|-------|--------|-----------|--------------|-------------|
| PROCESSED_EMAIL | id | Integer | Primary Key, Autoincrement | Unique identifier for each processed email record |
| PROCESSED_EMAIL | user_id | Integer | Foreign Key to User, Not Null | Reference to the user who owns this email |
| PROCESSED_EMAIL | folder_id | Integer | Foreign Key to Folder, Not Null | Reference to the folder this email belongs to |
| PROCESSED_EMAIL | email_uid | String(255) | Not Null | Unique ID of the email from IMAP server |
| PROCESSED_EMAIL | folder_name | String(255) | Not Null | Name of the IMAP folder (for redundancy) |
| PROCESSED_EMAIL | is_processed | Boolean | Default: False | Processing status (false=pending, true=processed) |
| PROCESSED_EMAIL | first_seen_at | DateTime | Default: datetime.utcnow | First time this email was detected during sync |
| PROCESSED_EMAIL | processed_at | DateTime | Nullable | When the email was marked as processed |
| PROCESSED_EMAIL | created_at | DateTime | Default: datetime.utcnow | Record creation timestamp |
| PROCESSED_EMAIL | updated_at | DateTime | Default: datetime.utcnow, On Update | Record update timestamp |
### Relationships
- **User to ProcessedEmail**: One-to-many relationship - each user can have multiple processed email records
- **Folder to ProcessedEmail**: One-to-many relationship - each folder can have multiple processed email records
- **Composite Key**: The combination of (user_id, folder_name, email_uid) should be unique to prevent duplicate records
### Database Indexes
- Primary key index on `id`
- Foreign key indexes on `user_id` and `folder_id`
- Composite unique index on `(user_id, folder_name, email_uid)`
- Index on `folder_name` for faster folder-based queries
- Index on `is_processed` for filtering pending emails
- Index on `first_seen_at` for tracking recently added emails
## Service Design
### ProcessedEmailsService
A new service class will be responsible for managing processed email records:
```python
class ProcessedEmailsService:
def __init__(self, user: User):
self.user = user
def get_pending_emails(self, folder_name: str) -> List[str]:
"""Get list of email UIDs that are pending processing in a folder."""
def mark_email_processed(self, folder_name: str, email_uid: str) -> bool:
"""Mark an email as processed."""
def mark_emails_processed(self, folder_name: str, email_uids: List[str]) -> int:
"""Mark multiple emails as processed in bulk."""
def sync_folder_emails(self, folder_name: str, email_uids: List[str]) -> int:
"""Sync email UIDs for a folder, adding new ones as pending."""
def get_pending_count(self, folder_name: str) -> int:
"""Get count of pending emails for a folder."""
def cleanup_old_records(self, folder_name: str, current_uids: List[str]) -> int:
"""Remove records for emails that no longer exist in the folder."""
```
### IMAPService Integration
The existing IMAP service will be enhanced to use the ProcessedEmailsService:
```python
class IMAPService:
def __init__(self, user: User):
self.user = user
self.config = user.imap_config or {}
self.connection = None
self.processed_emails_service = ProcessedEmailsService(user)
def get_folder_email_count(self, folder_name: str) -> int:
"""Get the count of emails in a specific folder, considering processed status."""
def get_pending_emails(self, folder_name: str) -> List[str]:
"""Get email UIDs that are pending processing."""
def sync_folders(self) -> Tuple[bool, str]:
"""Sync IMAP folders with local database, tracking email processing status."""
```
## API Endpoints
### New HTMX Endpoints for Processed Email Management
1. **Get Pending Emails for a Folder**
- Method: GET
- Path: `/api/folders/<folder_id>/pending-emails`
- Response: An Dialog List of email metadata for pending emails (subject, date, UID), a button to preview the email (fetch it from the imap server)
2. **Mark Email as Processed**
- Method: POST
- Path: `/api/folders/<folder_id>/emails/<email_uid>/process`
- Action: Mark a specific email as processed
- Response: Updated dialog body.
## Workflow Integration
### Email Processing Flow
```mermaid
sequenceDiagram
participant U as User
participant B as Browser
participant M as Main Blueprint
participant I as IMAP Service
participant P as ProcessedEmails Service
participant DB as Database
U->>B: Click "Sync Folders"
B->>M: POST /api/imap/sync
M->>I: Sync folders with processed email tracking
I->>I: Connect to IMAP server
I->>I: Get list of email UIDs for folder
I->>P: sync_folder_emails(folder_name, email_uids)
P->>DB: Create pending email records
P->>I: Return list of pending email UIDs
I->>M: Return sync results
M->>B: Update UI with pending counts
```
### Email Processing Status Update
```mermaid
sequenceDiagram
participant U as User
participant B as Browser
participant M as Main Blueprint
participant P as ProcessedEmails Service
participant DB as Database
U->>B: Trigger email processing
B->>M: POST /api/folders/<folder_id>/process-emails
M->>P: mark_emails_processed(folder_name, email_uids)
P->>DB: Update email processing status
P->>M: Return success count
M->>B: Update UI with new counts
```
## Migration Strategy
### Phase 1: Data Model Implementation
1. Create the `processed_emails` table with appropriate indexes
2. Implement the `ProcessedEmailsService` class
3. Add basic CRUD operations for email processing records
### Phase 2: IMAP Service Integration
1. Update `IMAPService` to use `ProcessedEmailsService`
2. Modify folder synchronization to track email UIDs
3. Update email count methods to consider processing status
### Phase 3: API and UI Integration
1. Add API endpoints for processed email management
2. Update UI to display accurate pending counts
3. Add bulk processing capabilities
### Phase 4: Optimization and Cleanup
1. Implement batch processing for performance
2. Add periodic cleanup of orphaned records
3. Optimize database queries for large datasets
## Security Considerations
1. **Access Control**: Ensure users can only access their own email processing records
2. **Data Validation**: Validate all email UIDs and folder names to prevent injection attacks
3. **Rate Limiting**: Implement rate limiting for email processing endpoints to prevent abuse
4. **Data Privacy**: Ensure no sensitive email content is stored in the database
## Performance Considerations
1. **Database Indexing**: Proper indexing on frequently queried fields
2. **Batch Operations**: Use batch operations for processing multiple emails
3. **Memory Management**: Process emails in batches to avoid memory issues with large mailboxes
4. **Caching**: Consider caching frequently accessed email processing status
## Future Enhancements
1. **Email Movement Tracking**: Track when emails are moved between folders
2. **Processing History**: Maintain a history of email processing actions
3. **Email Deduplication**: Handle duplicate emails across folders
4. **Automated Cleanup**: Periodic cleanup of old or orphaned processing records
5. **Analytics**: Provide insights into email processing patterns and efficiency

143
docs/plans/folder-types-redesign.md Normal file
View File

@@ -0,0 +1,143 @@
# Folder Types Redesign Plan
## Overview
This plan outlines the redesign of the email organizer to support two distinct types of folders:
1. **Folders to Tidy**: These folders contain emails that need to be processed and organized
2. **Destination Folders**: These folders are targets for organizing emails from other folders
## Current State
Currently, all folders are treated the same way with:
- Pending and processed counts
- Organization rules
- Toggle for enabling/disabling organization
The system doesn't differentiate between source folders (where emails are processed from) and destination folders (where emails are moved to).
## Proposed Changes
### 1. Data Model Updates
#### Folder Model Changes
Add a new field to the `Folder` model:
```python
folder_type = db.Column(db.String(20), default='destination', nullable=False)
```
Possible values:
- `'tidy'`: Folders that contain emails to be processed (e.g., Inbox)
- `'destination'`: Folders that are targets for email organization (default for new folders)
#### New Field Requirements:
- Must be a string to allow for potential future folder types
- Default value should be 'destination' for backward compatibility
- Not nullable to ensure all folders have a defined type
### 2. Business Logic Changes
#### Folder Synchronization Logic
- When synchronizing IMAP folders:
- If folder name is 'inbox' (case-insensitive), set `folder_type = 'tidy'`
- For all other folders, set `folder_type = 'destination'`
- Existing folders will keep their current type unless explicitly changed
#### Folder Count Logic
- **Folders to Tidy**:
- Show `pending_count` (emails waiting to be processed)
- Show `processed_count` (emails that have been processed)
- May show `total_count` (total emails in the folder)
- **Destination Folders**:
- Show `emails_count` (number of emails that have been put into this folder)
- No need to show pending/processed counts
### 3. UI Changes
#### Main Page Layout
Create two distinct sections:
1. **Folders to Tidy Section**
- Contains folders with `folder_type = 'tidy'`
- Shows pending and processed counts
- May include special styling to indicate these are source folders
2. **Destination Folders Section**
- Contains folders with `folder_type = 'destination'`
- Shows count of emails moved to this folder
- May include special styling to indicate these are target folders
#### Folder Card Changes
- **Tidy Folder Card**:
- Display: Folder name, pending count, processed count
- Actions: View pending emails, toggle organization
- **Destination Folder Card**:
- Display: Folder name, emails count
- Actions: View emails in folder (if implemented), basic folder management
### 4. API Changes
#### New Endpoints
- `/api/folders/filter?type={tidy|destination}` - Get folders filtered by type
- `/api/folders/<id>/type` - Update folder type (admin function)
#### Existing Endpoint Updates
- `/api/folders` - Add filtering capability
- `/api/imap/sync` - Implement new folder type logic
- Folder-related templates should adapt based on folder type
### 5. Migration Strategy
#### Database Migration
1. Create a new migration to add the `folder_type` column
2. Set default value to 'destination' for existing folders
3. Special handling for 'inbox' folders to set type to 'tidy'
#### Data Migration
- Scan all existing folders for user with 'inbox' in the name
- Set `folder_type = 'tidy'` for these folders
- Set `folder_type = 'destination'` for all other folders
### 6. Implementation Steps
1. **Phase 1: Data Model**
- Create migration for `folder_type` column
- Update `Folder` model
- Apply migration to database
2. **Phase 2: Backend Logic**
- Update folder synchronization logic
- Modify folder query methods
- Update count calculation logic
3. **Phase 3: Frontend UI**
- Create two new partial templates for folder sections
- Update folder card templates based on type
- Modify main page to show two sections
4. **Phase 4: Testing**
- Test folder synchronization with new types
- Verify UI shows correct sections
- Test folder count calculations
### 7. Future Considerations
#### Potential Enhancements
- Add ability to change folder type through UI
- Implement email movement tracking
- Add analytics for email organization efficiency
- Support for additional folder types in the future
#### Backward Compatibility
- Existing folders will be treated as destination folders
- No breaking changes to existing functionality
- Gradual rollout of new UI sections
## Success Criteria
1. Folders are correctly categorized as either 'tidy' or 'destination'
2. Inbox folders are automatically set as 'tidy' type
3. UI displays two distinct sections for folder types
4. Folder cards show appropriate counts based on type
5. All existing functionality continues to work