ai-game-2/tools/ora_editor/ORA_EDITOR.md

# ORA Editor - Browser-Based Image Layer Editor

## Overview

A Flask + Alpine.js web application for editing OpenRaster (ORA) files and PNG images with layer management, polygon-based mask extraction hints, and Krita integration.

**Location:** `tools/ora_editor/`

**Tech Stack:**
- Backend: Flask (Python 3)
- Frontend: Alpine.js + Tailwind CSS (CDN)
- Image Processing: Pillow
- Custom JS: Minimal (~50 lines for polygon canvas)

---

## Project Structure

```
tools/ora_editor/
├── app.py                 # Flask application
├── ora_ops.py             # ORA operations wrapper (functions from ora_edit.py)
├── templates/
│   └── editor.html       # Single-page Alpine.js UI
├── requirements.txt       # Python dependencies
├── temp/                  # Krita temp files (gitignored)
└── .gitignore
```

---

## Dependencies

### Python Packages (`requirements.txt`)
```
flask>=3.0.0
pillow>=10.0.0
```

### External Requirements
- **ComfyUI** running at configurable address (default: `127.0.0.1:8188`)
- **Krita** desktop application (for "Open in Krita" feature)

---

## Technology Choices

### Why Flask + Alpine.js?
- Flask: Minimal Python web framework, mirrors the CLI tool workflow
- Alpine.js: Reactive UI with zero custom JS, perfect for this use case
- Tailwind CSS (CDN): Rapid styling without build step

### Why Minimal Custom JS?
The polygon drawing requires capturing mouse clicks on a canvas. This is the only interaction that Alpine.js cannot handle natively. Everything else (layer toggles, modals, forms) is pure Alpine.js.

---

## File Operations

### Root Directory
All file paths are relative to the project root: `/home/noti/dev/ai-game-2`

### Supported Formats
- **ORA** (OpenRaster): Native format with layer support
- **PNG**: Auto-converted to ORA on open

### Open Flow
1. User clicks "Browse" button to open file browser dialog
2. Backend lists directories and files (PNG, ORA) from project root
3. User navigates directories, selects file
4. Backend checks if file exists
5. If PNG: auto-create ORA with single `base` layer
6. Parse ORA structure, extract layers
7. Return layer list + image data URLs

---

## Browse Dialog

### Purpose
Allows users to visually navigate the project directory and select PNG/ORA files without manually typing paths.

### UI Layout
```
┌─────────────────────────────────────────────────────────┐
│                      OPEN FILE                          │
│  ─────────────────────────────────────────────────────  │
│  📁 ..                                                  │
│  📁 scenes/                                             │
│  📁 tools/                                              │
│  🖼️ pic_010.ora                                        │
│  🖼️ pic_011.png                                        │
│                                                         │
│  Current: scenes/kq4_010/                               │
│                                                         │
│              [Cancel]  [Open Selected]                  │
└─────────────────────────────────────────────────────────┘
```

### Behavior
- **Double-click folder**: Navigate into folder
- **Double-click file**: Select and open immediately
- **Single-click + Enter**: Select and open
- ** ".." entry**: Navigate to parent directory
- **File types**: Only show `.png` and `.ora` files
- **Hidden files**: Excluded (dot-prefixed)

### Backend Implementation

#### `GET /api/browse`
List directory contents for browsing.

**Query params:**
- `path`: Directory path relative to project root (default: "")

**Response:**
```json
{
    "current_path": "scenes/kq4_010",
    "parent_path": "scenes",
    "directories": [
        {"name": "subdir1", "path": "scenes/kq4_010/subdir1"}
    ],
    "files": [
        {"name": "pic.png", "path": "scenes/kq4_010/pic.png", "type": "png"},
        {"name": "scene.ora", "path": "scenes/kq4_010/scene.ora", "type": "ora"}
    ]
}
```

### Frontend State
```javascript
// Added to main store
browseModal: false,
browsePath: '',           // Current directory being browsed
browseDirectories: [],    // List of {name, path}
browseFiles: [],          // List of {name, path, type}
browseSelected: null,     // Currently selected file path
```

### Entry Point
- Button next to path input: `[Browse...]`
- Or clicking into the path input field (if empty)

---

## Layer System

### Layer Data Structure
```python
{
    "name": "door_0",           # Layer identifier
    "is_group": False,          # False for regular layers
    "visibility": True,        # Visible by default
    "src": "data/door_0.png",  # Path inside ORA zip
    "opacity": 1.0,            # Opacity 0.0-1.0
}
```

### ORA Structure Created
```
📁 Group: <entity_name> (e.g., "door")
   └─ <entity_name>_0         (e.g., "door_0")
📁 Group: <another_entity>
   └─ <another_entity>_0
🖼️  Base: base
```

### Layer Operations

| Operation | Description |
|-----------|-------------|
| **Toggle Visibility** | Show/hide specific layer image on canvas |
| **Tint Red Preview** | Apply red tint to layer for mask verification (client-only) |
| **Rename** | Change layer/entity name |
| **Delete** | Remove layer from ORA |
| **Reorder** | Move layer up/down in stack (changes z-index) |
| **Add Masked** | Add new layer with alpha mask |

---

## Canvas Display

### Image Rendering  
- Individual layers rendered as stacked `<img>` elements with CSS positioning
- Layers stacked in order from list (visual z-index matches layer order)
- Visibility togglecheckbox hides/shows each layer's image on canvas
- Tint checkbox applies semi-transparent red overlay for mask verification (client-only, visual aid)
- No server-side compositing for preview

### Layer DOM Structure
```html
<div class="relative w-full h-full">
    <!-- Individual layers rendered in order (x-show controls visibility) -->
    <template x-for="layer in layers">
        <img 
            x-show="layer.visible"
            :src="'/api/image/layer/' + layer.name" 
            class="absolute inset-0"
            :style="layer.tintRed ? 'mix-blend-multiply; opacity: 0.6;' : ''"
        >
    </template>
    <!-- Polygon overlay when active -->
    <canvas id="polygon-canvas" class="absolute inset-0 pointer-events-auto"></canvas>
</div>
```

---

## Polygon Drawing

### Purpose
The polygon is **temporary** - it's a visual guide to help the AI mask extraction understand what area to focus on. It does NOT affect the final ORA file directly.

### Coordinate System
- Points stored as percentages (0.0-1.0) of image dimensions
- Sent to backend for mask extraction hint
- Drawn on a canvas overlay

### User Interaction
1. Click "Start Drawing" button
2. Click on image to add points (shown as small circles)
3. Right-click or double-click to close polygon
4. Polygon renders with selected color and line width
5. "Clear" removes polygon

### Backend Storage
Polygon points stored in memory on server (per-request, not session). Used only when submitting mask extraction with `use_polygon: true`.

### Implementation (Minimal JS)
```javascript
// Canvas click handler (only custom JS needed)
canvas.addEventListener('click', (e) => {
    const rect = canvas.getBoundingClientRect();
    const x = (e.clientX - rect.left) / rect.width;
    const y = (e.clientY - rect.top) / rect.height;
    alpineStore.addPolygonPoint(x, y);
});
```

---

## Mask Extraction (ComfyUI)

### Workflow
1. User describes subject (e.g., "the wooden door")
2. Optionally enable "Use polygon hint"
3. Click "Extract Mask"

### API: `POST /api/mask/extract`
```json
{
    "subject": "the wooden door",
    "use_polygon": true
}
```

### Backend Process
1. If `use_polygon: true`, fetch stored polygon points
2. Draw polygon overlay on input image (using `draw_polygon.py` logic)
3. Build prompt: "Create a black and white alpha mask of {subject}"
4. Queue workflow to ComfyUI at configured server URL
5. Poll for completion (timeout: 4 minutes)
6. Download resulting mask to `temp/` directory
7. Return mask path

### ComfyUI Integration
- Server address stored in localStorage (client-side setting)
- Uses existing `tools/image_mask_extraction.json` workflow
- Same extraction logic as `extract_mask.py`

### Mask Preview Modal
When mask is ready:
1. Full-screen modal appears
2. Shows base image with semi-transparent red overlay where the mask is white (selected area)
   - Uses CSS `mask-image` property to use grayscale values as alpha channel
   - White pixels = fully opaque red tint, black/dark pixels = transparent (no tint)
   - Dark/gray areas of the mask remain invisible so you can clearly see the mask boundary
3. Three buttons:
   - **Re-roll**: Re-run extraction with same params  
   - **Use This Mask**: Add masked layer to ORA, close modal
   - **Cancel**: Discard mask, close modal

### Modal Display
```
┌─────────────────────────────────────────────────────────┐
│                                                         │
│     [Base image with mask applied as tint overlay]     │
│                                                         │
│        [Re-roll]  [Use This Mask]  [Cancel]            │
│                                                         │
└─────────────────────────────────────────────────────────┘
```

---

## Krita Integration

### Flow
1. User selects a layer and clicks "Open in Krita"
2. Backend extracts layer PNG to `temp/{layer_name}_{timestamp}.png`
3. Returns `file://` URL
4. Browser opens URL (Krita launches with file)
5. User edits in Krita, saves

### File Watching
Since browser cannot detect when Krita saves:
- **On tab visibility change**: Check file mtime, auto-reload if changed
- **Manual "Refresh" button**: Force reload from disk
- **No automatic save back**: User explicitly saves in Krita, then refreshes

### Temp File Management
- Temp files stored in `tools/ora_editor/temp/`
- Named: `{layer_name}_{unix_timestamp}.png`
- Not automatically deleted (low priority)

---

## API Reference

### File Browser

#### `GET /api/browse`
List directory contents for file browsing.

**Query params:**
- `path`: Directory path relative to project root (default: "")

**Response:**
```json
{
    "current_path": "scenes/kq4_010",
    "parent_path": "scenes",
    "directories": [
        {"name": "subdir1", "path": "scenes/kq4_010/subdir1"}
    ],
    "files": [
        {"name": "pic.png", "path": "scenes/kq4_010/pic.png", "type": "png"},
        {"name": "scene.ora", "path": "scenes/kq4_010/scene.ora", "type": "ora"}
    ]
}
```

### File Operations

#### `POST /api/open`
Open a PNG or ORA file.

**Request:**
```json
{"path": "scenes/kq4_010/pic.png"}
```

**Response:**
```json
{
    "success": true,
    "file_path": "/home/noti/dev/ai-game-2/scenes/kq4_010/pic.ora",
    "ora_path": "/home/noti/dev/ai-game-2/scenes/kq4_010/pic.ora",
    "layers": [
        {"name": "base", "visibility": true, "src": "data/base.png"},
        {"name": "door_0", "visibility": true, "src": "data/door/door_0.png"}
    ]
}
```

#### `POST /api/save`
Persist current layer state to ORA file.

**Request:**
```json
{"ora_path": "scenes/kq4_010/pic.ora"}
```

**Response:**
```json
{"success": true}
```

---

### Layer Operations

#### `GET /api/layers`
Get current layer structure.

**Response:**
```json
{
    "layers": [
        {"name": "base", "visibility": true, "src": "data/base.png"},
        {"name": "door_0", "visibility": true, "src": "data/door/door_0.png"}
    ]
}
```

#### `POST /api/layer/add`
Add a new masked layer.

**Request:**
```json
{
    "entity_name": "door",
    "mask_path": "/tmp/mask.png",
    "source_layer": "base"
}
```

**Response:**
```json
{"success": true, "layer_name": "door_1"}
```

#### `POST /api/layer/rename`
Rename a layer.

**Request:**
```json
{"old_name": "door_0", "new_name": "wooden_door_0"}
```

**Response:**
```json
{"success": true}
```

#### `POST /api/layer/delete`
Delete a layer.

**Request:**
```json
{"layer_name": "door_0"}
```

**Response:**
```json
{"success": true}
```

#### `POST /api/layer/reorder`
Move layer up or down.

**Request:**
```json
{"layer_name": "door_0", "direction": "up"}
```

**Response:**
```json
{"success": true}
```

---

### Image Serving

#### `GET /api/image/<layer_name>`
Serve a layer PNG. Returns image data.

#### `GET /api/image/base`
Serve base/merged image.

#### `GET /api/image/layer/<layer_name>`
Serve a specific layer as image.

**Query params:**
- `ora_path`: Path to ORA file

**Response:** PNG image data or 404 if layer not found.

#### `GET /api/image/polygon`
Serve polygon overlay image (for drawing mode).

---

### Polygon

#### `POST /api/polygon`
Draw polygon overlay on current image.

**Request:**
```json
{
    "points": [{"x": 0.1, "y": 0.2}, {"x": 0.5, "y": 0.8}],
    "color": "#FF0000",
    "width": 2
}
```

**Response:**
```json
{"success": true, "overlay_url": "/api/image/polygon"}
```

#### `POST /api/polygon/points`
Store polygon points for mask extraction.

**Request:**
```json
{"points": [{"x": 0.1, "y": 0.2}, {"x": 0.5, "y": 0.8}]}
```

**Response:**
```json
{"success": true}
```

#### `POST /api/polygon/clear`
Clear polygon overlay.

**Response:**
```json
{"success": true}
```

---

### Mask Extraction

#### `POST /api/mask/extract`
Extract mask using ComfyUI.

**Request:**
```json
{
    "subject": "the wooden door",
    "use_polygon": true
}
```

**Response:**
```json
{
    "success": true,
    "mask_path": "/home/noti/dev/ai-game-2/tools/ora_editor/temp/mask.png"
}
```

---

### Krita

#### `POST /api/krita/open`
Copy layer to temp and return file URL.

**Request:**
```json
{"layer_name": "base"}
```

**Response:**
```json
{
    "success": true,
    "file_url": "file:///home/noti/dev/ai-game-2/tools/ora_editor/temp/base_1234567890.png",
    "temp_path": "/home/noti/dev/ai-game-2/tools/ora_editor/temp/base_1234567890.png"
}
```

#### `GET /api/krita/status/<layer_name>`
Check if temp file was modified.

**Response:**
```json
{
    "modified": true,
    "mtime": 1234567890
}
```

---

## Frontend State (Alpine.js)

### Main Store (`x-data`)
```javascript
{
    // File state
    filePath: '',
    oraPath: '',
    layers: [],
    baseImage: '',

    // UI state
    selectedLayer: null,
    isDrawingPolygon: false,
    polygonPoints: [],
    polygonColor: '#FF0000',
    polygonWidth: 2,

    // Mask extraction
    maskSubject: '',
    usePolygonHint: true,
    isExtracting: false,
    showMaskModal: false,
    tempMaskPath: null,

    // Krita
    kritaLayer: null,
    kritaMtime: null,

    // Settings
    comfyUrl: localStorage.getItem('comfy_url') || '127.0.0.1:8188',
}
```

### Layer Object
```javascript
{
    name: 'door_0',
    visibility: true,
    src: 'data/door/door_0.png'
}
```

---

## UI Layout

```
┌──────────────────────────────────────────────────────────────────────┐
│ [Open: ________________________] [📁 Browse] [🌀 Open]   [Settings ⚙] │
├───────────────────┬──────────────────────────────────────────────────┤
│ LAYERS            │                                                   │
│ ☑  □ base         │                                                   │
│ ☑  □ door_0      │           [Image Canvas with                      │
│ ☐  ☒ chest_0     │            stacked layers - visibility toggles    │
│                  │             show/hide individual layer images]    │
│                   │                                                   │
│ [Rename] [Delete] │                                                   │
│ [▲ Up] [▼ Down]  │                                                   │
│                   │                                                   │
│ ─────────────────│                                                   │
│ POLYGON TOOL     │                                                   │
│ Color: [#FF0000] │                                                   │
│ Width: [2____]   │                                                   │
│ [Start Drawing]  │                                                   │
│ Points: 5        │                                                   │
│ [Clear]          │                                                   │
│                   │                                                   │
│ ─────────────────│                                                   │
│ MASK EXTRACTION  │                                                   │
│ Subject: [______]│                                                   │
│ ☑ Use polygon    │                                                   │
│ [🌀 Extract Mask]│ (spinner shown when extracting)                    │
│                   │                                                   │
│ ─────────────────│                                                   │
│ [Open in Krita]  │                                                   │
│                   │                                                   │
├───────────────────┴──────────────────────────────────────────────────┤
│ [Save]                                         [Refresh from Krita]  │
└──────────────────────────────────────────────────────────────────────┘
```

Legend: ☑ = visible checkbox, ☒ = tint red checkbox (red when checked)

---

## Mask Preview Modal

```
┌─────────────────────────────────────────────────────────┐
│                      EXTRACTED MASK                     │
│  ─────────────────────────────────────────────────────  │
│                                                         │
│     [Base image with RED mask overlay]                 │
│         (mask shown in semi-transparent red)           │
│                                                         │
│        [Re-roll]  [Use This Mask]  [Cancel]            │
│                                                         │
└─────────────────────────────────────────────────────────┘
```

- **Re-roll**: Re-runs extraction with same params
- **Use This Mask**: Calls `POST /api/layer/add`, closes modal
- **Cancel**: Closes modal, mask discarded
- **Red tint**: Uses CSS filter to render grayscale mask as semi-transparent red

---

## Browse Modal

```
┌─────────────────────────────────────────────────────────┐
│                      OPEN FILE                          │
│  ─────────────────────────────────────────────────────  │
│  📁 ..                    (parent directory)           │
│  📁 scenes/                                             │
│  📁 tools/                                              │
│  🖼️ pic_010.ora                                        │
│  🖼️ pic_011.png                                        │
│                                                         │
│  Path: scenes/kq4_010/                                  │
│                                                         │
│                      [Cancel]  [Open Selected]          │
└─────────────────────────────────────────────────────────┘
```

- Double-click folder to navigate
- Double-click file to open immediately
- Single-click to select, then click "Open Selected"
- Only shows `.png` and `.ora` files

---

## Settings Modal

```
┌─────────────────────────────────────────────────────────┐
│                      SETTINGS                          │
│  ─────────────────────────────────────────────────────  │
│                                                         │
│  ComfyUI Server: [127.0.0.1:8188____________]          │
│                                                         │
│                         [Save]  [Cancel]                │
│                                                         │
└─────────────────────────────────────────────────────────┘
```

- Settings stored in `localStorage`
- ComfyURL stored as `comfy_url` key

---

## Implementation Tasks

### Phase 1: Project Setup
- [ ] Create directory structure (`templates/`, `temp/`)
- [ ] Create `requirements.txt`
- [ ] Create `.gitignore`
- [ ] Test virtualenv/dependencies

### Phase 2: Backend Core
- [ ] Create `ora_ops.py` with wrapper functions:
  - [ ] `load_ora(path)` - Parse ORA, return layers + images
  - [ ] `save_ora(ora_path)` - Rebuild ORA from current state
  - [ ] `add_masked_layer(ora_path, entity, mask_path, source)` - Apply mask
  - [ ] `rename_layer(ora_path, old, new)` - Update layer name
  - [ ] `delete_layer(ora_path, name)` - Remove layer
  - [ ] `reorder_layer(ora_path, name, direction)` - Move up/down
  - [ ] `get_layer_image(ora_path, layer_name)` - Extract PNG
  - [ ] `get_base_image(ora_path)` - Get merged/composited base
- [ ] Create `app.py` with Flask routes:
  - [ ] File browser (`/api/browse`)
  - [ ] File operations (`/api/open`, `/api/save`)
  - [ ] Layer operations (all `/api/layer/*`)
  - [ ] Image serving (`/api/image/*`)
  - [ ] Polygon (`/api/polygon*`)
  - [ ] Mask extraction (`/api/mask/extract`)
  - [ ] Krita (`/api/krita/*`)

### Phase 3: Frontend UI
- [ ] Create `editor.html` with Tailwind + Alpine.js
- [ ] File open input and handler
- [ ] Browse modal for file selection
- [ ] Layer list with visibility toggles
- [ ] Canvas area with stacked layer images
- [ ] Layer operations (rename, delete, reorder buttons)

### Phase 4: Polygon Drawing
- [ ] Polygon tool UI (color, width, start/clear)
- [ ] Canvas overlay for polygon rendering
- [ ] Minimal JS for click-to-add-points
- [ ] Backend polygon drawing endpoint

### Phase 5: Mask Extraction
- [ ] Mask extraction form UI
- [ ] Backend ComfyUI integration
- [ ] Mask preview modal
- [ ] Re-roll / Use This Mask / Cancel logic

### Phase 6: Krita Integration
- [ ] "Open in Krita" button and endpoint
- [ ] File URL generation
- [ ] Refresh button and logic
- [ ] Visibility-change auto-reload (JS)

### Phase 7: Polish
- [ ] Settings modal for ComfyUI URL
- [ ] Error handling and user feedback
- [ ] Loading states
- [ ] Empty states

---

## Key Implementation Details

### ORA File Structure
ORA is a ZIP file containing:
```
mimetype                    # "image/openraster"
stack.xml                   # Layer structure XML
mergedimage.png             # Composited full image
Thumbnails/thumbnail.png    # Preview thumbnail
data/
  base.png                  # Base layer
  <entity>/
    <entity>_0.png          # Masked layer
```

### stack.xml Format
```xml
<image version="0.0.3" w="800" h="600">
  <stack>
    <layer name="base" src="data/base.png" opacity="1.0"/>
    <stack name="door">
      <layer name="door_0" src="data/door/door_0.png" opacity="1.0"/>
    </stack>
  </stack>
</image>
```

### Image Compositing for Preview
- Frontend receives individual layer PNG URLs
- Layers stacked with CSS `position: absolute`
- Visibility toggled via CSS `opacity`
- No server-side compositing needed

### Polygon Overlay Rendering
1. Backend receives points as `[{x: 0.1, y: 0.2}, ...]`
2. Convert to absolute pixel coordinates
3. Use Pillow `ImageDraw.polygon()` to draw
4. Return overlay as PNG

### ComfyUI Workflow
Uses existing `tools/image_mask_extraction.json`:
1. Load input image
2. If polygon hint: composite polygon overlay
3. Set prompt text
4. Queue prompt to ComfyUI
5. Poll for completion
6. Download output mask

---

## Color Reference

### Default Polygon Colors
| Name | Hex |
|------|-----|
| Red | #FF0000 |
| Green | #00FF00 |
| Blue | #0000FF |
| Yellow | #FFFF00 |

### Mask Tint Colors (Preview)
Default: #00FF0080 (semi-transparent green)

---

## Error Handling

| Error | User Feedback |
|-------|---------------|
| File not found | "File not found: {path}" |
| Invalid ORA | "Invalid ORA file format" |
| ComfyUI offline | "Cannot connect to ComfyUI at {url}" |
| ComfyUI timeout | "Mask extraction timed out (4 min)" |
| Layer not found | "Layer '{name}' not found" |

---

## Browser Compatibility

Tested/supported:
- Modern browsers with ES6+ support
- Chrome/Edge (primary)
- Firefox
- Safari

Required features:
- ES6 modules
- CSS Grid/Flexbox
- Canvas API (for polygon)
- localStorage (for settings)

---

## Running the Application

```bash
cd tools/ora_editor
source ../venv/bin/activate  # or your Python env
pip install -r requirements.txt
python app.py
# Opens at http://localhost:5000
```

---

## Future Considerations (Out of Scope)

- Multi-select layers
- Drag-and-drop reordering
- Undo/redo history
- Collaborative editing
- Additional export formats
- Layer groups in UI (show hierarchy)