notid/email-organizer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

lots of configuration progress.

Browse Source
This commit is contained in:
Bryce
2025-08-06 15:38:49 -07:00
parent c6102dda45
commit 41ea8fb3bd
24 changed files with 2566 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
docs/design/data-model.md
View File

@@ -26,8 +26,10 @@ erDiagram
text rule_text "Natural Language Rule"
int priority "Processing Order"
boolean organize_enabled "Default: True"
string folder_type "Default: 'destination'"
int total_count "Default: 0"
int pending_count "Default: 0"
int emails_count "Default: 0"
json recent_emails "JSON Array"
datetime created_at "Default: UTC Now"
datetime updated_at "Default: UTC Now, On Update"
@@ -83,8 +85,10 @@ The `Folder` entity stores email organization rules and metadata for each user's
| rule_text | Text | Nullable | Natural language description of the folder rule |
| priority | Integer | Nullable | Processing order (0=normal, 1=high) |
| organize_enabled | Boolean | Default: True | Whether the organization rule is active |
| folder_type | String(20) | Default: 'destination' | Folder type: 'tidy' or 'destination' |
| total_count | Integer | Default: 0 | Total number of emails in the folder |
| pending_count | Integer | Default: 0 | Number of emails waiting to be processed |
| emails_count | Integer | Default: 0 | Number of emails moved to this destination folder |
| recent_emails | JSON | Default: [] | Array of recent email metadata |
| created_at | DateTime | Default: datetime.utcnow | Timestamp of folder creation |
| updated_at | DateTime | Default: datetime.utcnow, On Update | Timestamp of last update |
@@ -100,6 +104,9 @@ The `Folder` entity stores email organization rules and metadata for each user's
- Folder name must be unique per user
- Rule text can be null (for manually created folders)
- Priority values: 0 (normal), 1 (high priority)
- Folder types:
- 'tidy': Folders containing emails to be processed (e.g., Inbox)
- 'destination': Folders that are targets for email organization (default)
- Recent emails array stores JSON objects with subject and date information
## Data Constraints
@@ -128,7 +135,8 @@ The `Folder` entity stores email organization rules and metadata for each user's
- `User.created_at`, `User.updated_at`: Current UTC timestamp
- `Folder.created_at`, `Folder.updated_at`: Current UTC timestamp
- `Folder.organize_enabled`: True
- `Folder.total_count`, `Folder.pending_count`: 0
- `Folder.folder_type`: 'destination'
- `Folder.total_count`, `Folder.pending_count`, `Folder.emails_count`: 0
- `Folder.recent_emails`: Empty array
## JSON Data Structures
@@ -219,6 +227,49 @@ The `recent_emails` field stores an array of JSON objects:
- Batch updates for email counts
- JSON operations for recent emails metadata
## Folder Types
The system supports two distinct types of folders, each with different purposes and behaviors:
### Tidy Folders
Folders with `folder_type = 'tidy'` are source folders that contain emails waiting to be processed and organized.
**Characteristics:**
- Display pending and processed email counts
- Can have organization rules enabled/disabled
- Support viewing pending emails
- Example: Inbox folder
**UI Representation:**
- Shows "pending count" and "processed count" badges
- Includes "View Pending" button if there are pending emails
- May include priority indicators
### Destination Folders
Folders with `folder_type = 'destination'` are target folders where emails are moved from other folders during organization.
**Characteristics:**
- Display count of emails moved to this folder
- Typically don't have organization rules (or they're ignored)
- Focus on showing how many emails have been organized into them
- Example: "Projects", "Finance", "Personal" folders
**UI Representation:**
- Shows "emails count" badge
- Simpler interface without pending/processed indicators
- Focus on folder management and viewing contents
### Folder Type Determination
Folder types are determined as follows:
- During IMAP synchronization:
- Folders named "inbox" (case-insensitive) are automatically set as 'tidy'
- All other folders are set as 'destination'
- Manually created folders default to 'destination'
- Folder type can be changed through administrative functions
## Future Data Model Considerations
### Potential Enhancements

228
docs/design/folder-types-ui.md Normal file
View File

@@ -0,0 +1,228 @@
# Folder Types UI Design
## Overview
This document outlines the UI design changes needed to support the new folder types feature: "Folders to Tidy" and "Destination Folders". The UI will be reorganized to clearly separate these two types of folders into distinct sections.
## UI Structure
### Main Page Layout
The main page will be divided into two distinct sections:
1. **Folders to Tidy Section**
- Contains folders with `folder_type = 'tidy'`
- Typically includes the inbox and other source folders
- Shows pending and processed email counts
- Has actions for viewing and processing emails
2. **Destination Folders Section**
- Contains folders with `folder_type = 'destination'`
- Contains target folders for email organization
- Shows count of emails moved to each folder
- Focuses on folder management and viewing contents
### Visual Design
#### Section Headers
Each section will have a clear header with:
- Section title
- Brief description of the section's purpose
- Action button (e.g., "Add Folder" for destination folders)
#### Section Styling
- Different background colors or borders to visually distinguish sections
- Consistent spacing and layout within each section
- Responsive grid layout that adapts to screen size
## Folder Card Components
### Tidy Folder Card
For folders with `folder_type = 'tidy'`, the card will display:
#### Header
- Folder name (prominent)
- Edit and delete buttons
#### Count Badges
- Total emails count
- Pending emails count (with warning styling if > 0)
- Processed emails count (calculated as total - pending)
#### Actions
- "View Pending" button (if pending count > 0)
- Organize toggle switch
- Priority badge
#### Content
- Rule text description
- Recent email previews (if available)
### Destination Folder Card
For folders with `folder_type = 'destination'`, the card will display:
#### Header
- Folder name (prominent)
- Edit and delete buttons
#### Count Badge
- Emails count (number of emails moved to this folder)
#### Content
- Rule text description (if applicable)
#### Actions
- Basic folder management options
- No organize toggle (as it's not applicable)
## UI Components
### Section Templates
#### Folders to Tidy Section
```html
<div class="folders-to-tidy-section">
<div class="section-header">
<h2>Folders to Tidy</h2>
<p>Folders containing emails that need to be processed</p>
<button class="btn btn-primary" hx-get="/api/folders/new">
<i class="fas fa-plus"></i> Add Tidy Folder
</button>
</div>
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
{% for folder in tidy_folders %}
{% include 'partials/folder_card_tidy.html' %}
{% endfor %}
</div>
</div>
```
#### Destination Folders Section
```html
<div class="destination-folders-section">
<div class="section-header">
<h2>Destination Folders</h2>
<p>Folders where emails are organized and stored</p>
<button class="btn btn-primary" hx-get="/api/folders/new">
<i class="fas fa-plus"></i> Add Destination Folder
</button>
</div>
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
{% for folder in destination_folders %}
{% include 'partials/folder_card_destination.html' %}
{% endfor %}
</div>
</div>
```
### Folder Card Templates
#### Tidy Folder Card
```html
<div class="card bg-base-100 shadow-xl border border-base-300">
<div class="card-body">
<div class="flex justify-between items-start mb-2">
<h3 class="text-xl font-bold truncate">{{ folder.name }}</h3>
<div class="flex space-x-2">
<button class="btn btn-sm btn-outline" hx-get="/api/folders/{{ folder.id }}/edit">
<i class="fas fa-edit"></i>
</button>
<button class="btn btn-sm btn-outline btn-error" hx-delete="/api/folders/{{ folder.id }}">
<i class="fas fa-trash"></i>
</button>
</div>
</div>
<div class="flex space-x-2 mb-2">
<span class="badge badge-outline">{{ folder.total_count }} total</span>
{% if folder.pending_count > 0 %}
<button class="badge badge-warning" hx-get="/api/folders/{{ folder.id }}/pending-emails">
{{ folder.pending_count }} pending
</button>
{% else %}
<span class="badge badge-secondary">{{ folder.pending_count }} pending</span>
{% endif %}
<span class="badge badge-success">{{ folder.total_count - folder.pending_count }} processed</span>
</div>
<div class="bg-base-200 rounded-box p-3 mb-3">
<p class="text-sm">{{ folder.rule_text }}</p>
</div>
<div class="flex justify-between items-center">
<div class="flex items-center space-x-2">
<span class="text-xs">Organize:</span>
<input type="checkbox" class="toggle toggle-sm"
{% if folder.organize_enabled %}checked{% endif %}
hx-put="/api/folders/{{ folder.id }}/toggle">
</div>
{% if folder.priority == 1 %}
<span class="badge badge-error text-xs">High Priority</span>
{% endif %}
</div>
</div>
</div>
```
#### Destination Folder Card
```html
<div class="card bg-base-100 shadow-xl border border-base-300">
<div class="card-body">
<div class="flex justify-between items-start mb-2">
<h3 class="text-xl font-bold truncate">{{ folder.name }}</h3>
<div class="flex space-x-2">
<button class="btn btn-sm btn-outline" hx-get="/api/folders/{{ folder.id }}/edit">
<i class="fas fa-edit"></i>
</button>
<button class="btn btn-sm btn-outline btn-error" hx-delete="/api/folders/{{ folder.id }}">
<i class="fas fa-trash"></i>
</button>
</div>
</div>
<div class="flex space-x-2 mb-2">
<span class="badge badge-primary">{{ folder.emails_count }} emails</span>
</div>
<div class="bg-base-200 rounded-box p-3 mb-3">
<p class="text-sm">{{ folder.rule_text }}</p>
</div>
</div>
</div>
```
## Styling Guidelines
### Color Scheme
- **Tidy Folders**: Use warning colors (yellow/orange) for pending emails
- **Destination Folders**: Use primary colors (blue) for email counts
- Consistent use of DaisyUI badge classes for different states
### Responsive Design
- Grid layout adjusts from 1 column on mobile to 3 columns on desktop
- Cards should have consistent width across all screen sizes
- Section headers should remain readable on mobile
### Interactive Elements
- Buttons should have hover states
- Toggle switches should be clearly labeled
- Pending email count should be clickable to open dialog
## Implementation Notes
### Template Organization
- Create separate partial templates for each folder type
- Main index template will include both sections
- Use HTMX for dynamic updates without full page reloads
### Data Loading
- Fetch folders filtered by type from the backend
- Pass filtered lists to appropriate section templates
- Maintain existing functionality for folder operations
### Accessibility
- Ensure all interactive elements have proper ARIA labels
- Use semantic HTML5 elements for section structure
- Maintain keyboard navigation support

261
docs/design/processed-emails-spec.md Normal file
View File

@@ -0,0 +1,261 @@
# Processed Emails Feature Specification
## Overview
This document outlines the specification for implementing a feature to persistently track which emails have been processed by the Email Organizer system. The goal is to maintain a record of email processing status to avoid reprocessing the same emails during synchronization and provide accurate pending email counts.
## Requirements
### 1. Email Tracking Requirements
- **Unique Email Identification**: Track emails using a unique identifier (UID) provided by the IMAP server, along with the folder name and user ID
- **Processing Status**: Mark emails as either "pending" (unprocessed) or "processed"
- **Minimal Data Storage**: Store only essential information - email UID, folder, user, and processing status - not email content, subjects, or bodies
- **Persistence**: Maintain processing status across application restarts and synchronization cycles
- **Efficient Lookup**: Quickly determine which emails in a folder are pending processing
### 2. Synchronization Requirements
- **Initial Sync**: During first synchronization of a folder, all emails should be marked as "pending"
- **Incremental Sync**: On subsequent syncs, only emails that haven't been processed should be identified as pending
- **Status Update**: When an email is processed, update its status from "pending" to "processed"
- **Cleanup**: Remove records for emails that no longer exist on the IMAP server (optional for future enhancement)
### 3. Performance Requirements
- **Efficient Storage**: Use appropriate database indexing for fast lookups
- **Minimal Memory Usage**: Store only essential data to keep memory footprint low
- **Batch Processing**: Support batch operations for processing multiple emails efficiently
## Data Model Design
### ProcessedEmails Table
```mermaid
erDiagram
USER {
int id PK "Primary Key"
string email "Unique, Not Null"
string first_name "Not Null"
string last_name "Not Null"
string password_hash "Not Null"
json imap_config "JSON Configuration"
datetime created_at "Default: UTC Now"
datetime updated_at "Default: UTC Now, On Update"
}
FOLDER {
int id PK "Primary Key"
int user_id FK "Foreign Key to User"
string name "Not Null"
text rule_text "Natural Language Rule"
int priority "Processing Order"
boolean organize_enabled "Default: True"
int total_count "Default: 0"
int pending_count "Default: 0"
json recent_emails "JSON Array"
datetime created_at "Default: UTC Now"
datetime updated_at "Default: UTC Now, On Update"
}
PROCESSED_EMAIL {
int id PK "Primary Key"
int user_id FK "Foreign Key to User"
int folder_id FK "Foreign Key to Folder"
string email_uid "Not Null" "IMAP Email UID"
string folder_name "Not Null" "IMAP Folder Name"
boolean is_processed "Default: False" "Processing Status"
datetime first_seen_at "Default: UTC Now" "First seen during sync"
datetime processed_at "Nullable" "When email was processed"
datetime created_at "Default: UTC Now"
datetime updated_at "Default: UTC Now, On Update"
}
USER ||--o{ FOLDER : "has"
USER ||--o{ PROCESSED_EMAIL : "has"
FOLDER ||--o{ PROCESSED_EMAIL : "has"
```
### Column Specifications
| Table | Column | Data Type | Constraints | Description |
|-------|--------|-----------|--------------|-------------|
| PROCESSED_EMAIL | id | Integer | Primary Key, Autoincrement | Unique identifier for each processed email record |
| PROCESSED_EMAIL | user_id | Integer | Foreign Key to User, Not Null | Reference to the user who owns this email |
| PROCESSED_EMAIL | folder_id | Integer | Foreign Key to Folder, Not Null | Reference to the folder this email belongs to |
| PROCESSED_EMAIL | email_uid | String(255) | Not Null | Unique ID of the email from IMAP server |
| PROCESSED_EMAIL | folder_name | String(255) | Not Null | Name of the IMAP folder (for redundancy) |
| PROCESSED_EMAIL | is_processed | Boolean | Default: False | Processing status (false=pending, true=processed) |
| PROCESSED_EMAIL | first_seen_at | DateTime | Default: datetime.utcnow | First time this email was detected during sync |
| PROCESSED_EMAIL | processed_at | DateTime | Nullable | When the email was marked as processed |
| PROCESSED_EMAIL | created_at | DateTime | Default: datetime.utcnow | Record creation timestamp |
| PROCESSED_EMAIL | updated_at | DateTime | Default: datetime.utcnow, On Update | Record update timestamp |
### Relationships
- **User to ProcessedEmail**: One-to-many relationship - each user can have multiple processed email records
- **Folder to ProcessedEmail**: One-to-many relationship - each folder can have multiple processed email records
- **Composite Key**: The combination of (user_id, folder_name, email_uid) should be unique to prevent duplicate records
### Database Indexes
- Primary key index on `id`
- Foreign key indexes on `user_id` and `folder_id`
- Composite unique index on `(user_id, folder_name, email_uid)`
- Index on `folder_name` for faster folder-based queries
- Index on `is_processed` for filtering pending emails
- Index on `first_seen_at` for tracking recently added emails
## Service Design
### ProcessedEmailsService
A new service class will be responsible for managing processed email records:
```python
class ProcessedEmailsService:
def __init__(self, user: User):
self.user = user
def get_pending_emails(self, folder_name: str) -> List[str]:
"""Get list of email UIDs that are pending processing in a folder."""
def mark_email_processed(self, folder_name: str, email_uid: str) -> bool:
"""Mark an email as processed."""
def mark_emails_processed(self, folder_name: str, email_uids: List[str]) -> int:
"""Mark multiple emails as processed in bulk."""
def sync_folder_emails(self, folder_name: str, email_uids: List[str]) -> int:
"""Sync email UIDs for a folder, adding new ones as pending."""
def get_pending_count(self, folder_name: str) -> int:
"""Get count of pending emails for a folder."""
def cleanup_old_records(self, folder_name: str, current_uids: List[str]) -> int:
"""Remove records for emails that no longer exist in the folder."""
```
### IMAPService Integration
The existing IMAP service will be enhanced to use the ProcessedEmailsService:
```python
class IMAPService:
def __init__(self, user: User):
self.user = user
self.config = user.imap_config or {}
self.connection = None
self.processed_emails_service = ProcessedEmailsService(user)
def get_folder_email_count(self, folder_name: str) -> int:
"""Get the count of emails in a specific folder, considering processed status."""
def get_pending_emails(self, folder_name: str) -> List[str]:
"""Get email UIDs that are pending processing."""
def sync_folders(self) -> Tuple[bool, str]:
"""Sync IMAP folders with local database, tracking email processing status."""
```
## API Endpoints
### New HTMX Endpoints for Processed Email Management
1. **Get Pending Emails for a Folder**
- Method: GET
- Path: `/api/folders/<folder_id>/pending-emails`
- Response: An Dialog List of email metadata for pending emails (subject, date, UID), a button to preview the email (fetch it from the imap server)
2. **Mark Email as Processed**
- Method: POST
- Path: `/api/folders/<folder_id>/emails/<email_uid>/process`
- Action: Mark a specific email as processed
- Response: Updated dialog body.
## Workflow Integration
### Email Processing Flow
```mermaid
sequenceDiagram
participant U as User
participant B as Browser
participant M as Main Blueprint
participant I as IMAP Service
participant P as ProcessedEmails Service
participant DB as Database
U->>B: Click "Sync Folders"
B->>M: POST /api/imap/sync
M->>I: Sync folders with processed email tracking
I->>I: Connect to IMAP server
I->>I: Get list of email UIDs for folder
I->>P: sync_folder_emails(folder_name, email_uids)
P->>DB: Create pending email records
P->>I: Return list of pending email UIDs
I->>M: Return sync results
M->>B: Update UI with pending counts
```
### Email Processing Status Update
```mermaid
sequenceDiagram
participant U as User
participant B as Browser
participant M as Main Blueprint
participant P as ProcessedEmails Service
participant DB as Database
U->>B: Trigger email processing
B->>M: POST /api/folders/<folder_id>/process-emails
M->>P: mark_emails_processed(folder_name, email_uids)
P->>DB: Update email processing status
P->>M: Return success count
M->>B: Update UI with new counts
```
## Migration Strategy
### Phase 1: Data Model Implementation
1. Create the `processed_emails` table with appropriate indexes
2. Implement the `ProcessedEmailsService` class
3. Add basic CRUD operations for email processing records
### Phase 2: IMAP Service Integration
1. Update `IMAPService` to use `ProcessedEmailsService`
2. Modify folder synchronization to track email UIDs
3. Update email count methods to consider processing status
### Phase 3: API and UI Integration
1. Add API endpoints for processed email management
2. Update UI to display accurate pending counts
3. Add bulk processing capabilities
### Phase 4: Optimization and Cleanup
1. Implement batch processing for performance
2. Add periodic cleanup of orphaned records
3. Optimize database queries for large datasets
## Security Considerations
1. **Access Control**: Ensure users can only access their own email processing records
2. **Data Validation**: Validate all email UIDs and folder names to prevent injection attacks
3. **Rate Limiting**: Implement rate limiting for email processing endpoints to prevent abuse
4. **Data Privacy**: Ensure no sensitive email content is stored in the database
## Performance Considerations
1. **Database Indexing**: Proper indexing on frequently queried fields
2. **Batch Operations**: Use batch operations for processing multiple emails
3. **Memory Management**: Process emails in batches to avoid memory issues with large mailboxes
4. **Caching**: Consider caching frequently accessed email processing status
## Future Enhancements
1. **Email Movement Tracking**: Track when emails are moved between folders
2. **Processing History**: Maintain a history of email processing actions
3. **Email Deduplication**: Handle duplicate emails across folders
4. **Automated Cleanup**: Periodic cleanup of old or orphaned processing records
5. **Analytics**: Provide insights into email processing patterns and efficiency