notid/ai-game-2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
319c2565c6204d42b76b9f1d1c4c8683f98d6baa
ai-game-2/tools/ora_editor/ORA_EDITOR.md
Bryce 319c2565c6 Add ora_ops module for ORA file operations 
- Implements core ORA reading/writing functions
- Layer add, rename, delete, reorder, visibility operations
- Full test suite with 8 passing tests
2026-03-27 08:46:49 -07:00

754 lines
21 KiB
Markdown
Raw Blame History

# ORA Editor - Browser-Based Image Layer Editor
## Overview
A Flask + Alpine.js web application for editing OpenRaster (ORA) files and PNG images with layer management, polygon-based mask extraction hints, and Krita integration.
**Location:** `tools/ora_editor/`
**Tech Stack:**
- Backend: Flask (Python 3)
- Frontend: Alpine.js + Tailwind CSS (CDN)
- Image Processing: Pillow
- Custom JS: Minimal (~50 lines for polygon canvas)
---
## Project Structure
```
tools/ora_editor/
├── app.py # Flask application
├── ora_ops.py # ORA operations wrapper (functions from ora_edit.py)
├── templates/
│ └── editor.html # Single-page Alpine.js UI
├── requirements.txt # Python dependencies
├── temp/ # Krita temp files (gitignored)
└── .gitignore
```
---
## Dependencies
### Python Packages (`requirements.txt`)
```
flask>=3.0.0
pillow>=10.0.0
```
### External Requirements
- **ComfyUI** running at configurable address (default: `127.0.0.1:8188`)
- **Krita** desktop application (for "Open in Krita" feature)
---
## Technology Choices
### Why Flask + Alpine.js?
- Flask: Minimal Python web framework, mirrors the CLI tool workflow
- Alpine.js: Reactive UI with zero custom JS, perfect for this use case
- Tailwind CSS (CDN): Rapid styling without build step
### Why Minimal Custom JS?
The polygon drawing requires capturing mouse clicks on a canvas. This is the only interaction that Alpine.js cannot handle natively. Everything else (layer toggles, modals, forms) is pure Alpine.js.
---
## File Operations
### Root Directory
All file paths are relative to the project root: `/home/noti/dev/ai-game-2`
### Supported Formats
- **ORA** (OpenRaster): Native format with layer support
- **PNG**: Auto-converted to ORA on open
### Open Flow
1. User enters path relative to project root (e.g., `scenes/kq4_010/pic.png`)
2. Backend checks if file exists
3. If PNG: auto-create ORA with single `base` layer
4. Parse ORA structure, extract layers
5. Return layer list + image data URLs
---
## Layer System
### Layer Data Structure
```python
{
"name": "door_0", # Layer identifier
"is_group": False, # False for regular layers
"visibility": True, # Visible by default
"src": "data/door_0.png", # Path inside ORA zip
"opacity": 1.0, # Opacity 0.0-1.0
}
```
### ORA Structure Created
```
📁 Group: <entity_name> (e.g., "door")
└─ <entity_name>_0 (e.g., "door_0")
📁 Group: <another_entity>
└─ <another_entity>_0
🖼️ Base: base
```
### Layer Operations
| Operation | Description |
|-----------|-------------|
| **Toggle Visibility** | Show/hide layer on canvas |
| **Rename** | Change layer/entity name |
| **Delete** | Remove layer from ORA |
| **Reorder** | Move layer up/down in stack (changes z-index) |
| **Add Masked** | Add new layer with alpha mask |
---
## Canvas Display
### Image Rendering
- Layers rendered as stacked `<img>` elements with CSS positioning
- Base layer at bottom, entity layers stacked above
- Visibility controlled by `opacity: 0` or `opacity: 1`
- No server-side compositing for preview
### Layer DOM Structure
```html
<div class="relative w-full h-full">
<img src="/api/image/base" class="absolute inset-0">
<img src="/api/image/door_0" class="absolute inset-0" style="opacity: 1">
<img src="/api/image/chest_0" class="absolute inset-0" style="opacity: 0">
<!-- Polygon overlay when active -->
<canvas id="polygon-canvas" class="absolute inset-0 pointer-events-auto"></canvas>
</div>
```
---
## Polygon Drawing
### Purpose
The polygon is **temporary** - it's a visual guide to help the AI mask extraction understand what area to focus on. It does NOT affect the final ORA file directly.
### Coordinate System
- Points stored as percentages (0.0-1.0) of image dimensions
- Sent to backend for mask extraction hint
- Drawn on a canvas overlay
### User Interaction
1. Click "Start Drawing" button
2. Click on image to add points (shown as small circles)
3. Right-click or double-click to close polygon
4. Polygon renders with selected color and line width
5. "Clear" removes polygon
### Backend Storage
Polygon points stored in memory on server (per-request, not session). Used only when submitting mask extraction with `use_polygon: true`.
### Implementation (Minimal JS)
```javascript
// Canvas click handler (only custom JS needed)
canvas.addEventListener('click', (e) => {
const rect = canvas.getBoundingClientRect();
const x = (e.clientX - rect.left) / rect.width;
const y = (e.clientY - rect.top) / rect.height;
alpineStore.addPolygonPoint(x, y);
});
```
---
## Mask Extraction (ComfyUI)
### Workflow
1. User describes subject (e.g., "the wooden door")
2. Optionally enable "Use polygon hint"
3. Click "Extract Mask"
### API: `POST /api/mask/extract`
```json
{
"subject": "the wooden door",
"use_polygon": true
}
```
### Backend Process
1. If `use_polygon: true`, fetch stored polygon points
2. Draw polygon overlay on input image (using `draw_polygon.py` logic)
3. Build prompt: "Create a black and white alpha mask of {subject}"
4. Queue workflow to ComfyUI at configured server URL
5. Poll for completion (timeout: 4 minutes)
6. Download resulting mask to `temp/` directory
7. Return mask path
### ComfyUI Integration
- Server address stored in localStorage (client-side setting)
- Uses existing `tools/image_mask_extraction.json` workflow
- Same extraction logic as `extract_mask.py`
### Mask Preview Modal
When mask is ready:
1. Full-screen modal appears
2. Shows base image with mask applied as colored tint overlay
3. Three buttons:
- **Re-roll**: Re-run extraction with same params
- **Use This Mask**: Add masked layer to ORA, close modal
- **Cancel**: Discard mask, close modal
### Modal Display
```
┌─────────────────────────────────────────────────────────┐
│ │
│ [Base image with mask applied as tint overlay] │
│ │
│ [Re-roll] [Use This Mask] [Cancel] │
│ │
└─────────────────────────────────────────────────────────┘
```
---
## Krita Integration
### Flow
1. User selects a layer and clicks "Open in Krita"
2. Backend extracts layer PNG to `temp/{layer_name}_{timestamp}.png`
3. Returns `file://` URL
4. Browser opens URL (Krita launches with file)
5. User edits in Krita, saves
### File Watching
Since browser cannot detect when Krita saves:
- **On tab visibility change**: Check file mtime, auto-reload if changed
- **Manual "Refresh" button**: Force reload from disk
- **No automatic save back**: User explicitly saves in Krita, then refreshes
### Temp File Management
- Temp files stored in `tools/ora_editor/temp/`
- Named: `{layer_name}_{unix_timestamp}.png`
- Not automatically deleted (low priority)
---
## API Reference
### File Operations
#### `POST /api/open`
Open a PNG or ORA file.
**Request:**
```json
{"path": "scenes/kq4_010/pic.png"}
```
**Response:**
```json
{
"success": true,
"file_path": "/home/noti/dev/ai-game-2/scenes/kq4_010/pic.ora",
"ora_path": "/home/noti/dev/ai-game-2/scenes/kq4_010/pic.ora",
"layers": [
{"name": "base", "visibility": true, "src": "data/base.png"},
{"name": "door_0", "visibility": true, "src": "data/door/door_0.png"}
]
}
```
#### `POST /api/save`
Persist current layer state to ORA file.
**Request:**
```json
{"ora_path": "scenes/kq4_010/pic.ora"}
```
**Response:**
```json
{"success": true}
```
---
### Layer Operations
#### `GET /api/layers`
Get current layer structure.
**Response:**
```json
{
"layers": [
{"name": "base", "visibility": true, "src": "data/base.png"},
{"name": "door_0", "visibility": true, "src": "data/door/door_0.png"}
]
}
```
#### `POST /api/layer/add`
Add a new masked layer.
**Request:**
```json
{
"entity_name": "door",
"mask_path": "/tmp/mask.png",
"source_layer": "base"
}
```
**Response:**
```json
{"success": true, "layer_name": "door_1"}
```
#### `POST /api/layer/rename`
Rename a layer.
**Request:**
```json
{"old_name": "door_0", "new_name": "wooden_door_0"}
```
**Response:**
```json
{"success": true}
```
#### `POST /api/layer/delete`
Delete a layer.
**Request:**
```json
{"layer_name": "door_0"}
```
**Response:**
```json
{"success": true}
```
#### `POST /api/layer/reorder`
Move layer up or down.
**Request:**
```json
{"layer_name": "door_0", "direction": "up"}
```
**Response:**
```json
{"success": true}
```
---
### Image Serving
#### `GET /api/image/<layer_name>`
Serve a layer PNG. Returns image data.
#### `GET /api/image/base`
Serve base/merged image.
#### `GET /api/image/polygon`
Serve polygon overlay image (for drawing mode).
---
### Polygon
#### `POST /api/polygon`
Draw polygon overlay on current image.
**Request:**
```json
{
"points": [{"x": 0.1, "y": 0.2}, {"x": 0.5, "y": 0.8}],
"color": "#FF0000",
"width": 2
}
```
**Response:**
```json
{"success": true, "overlay_url": "/api/image/polygon"}
```
#### `POST /api/polygon/points`
Store polygon points for mask extraction.
**Request:**
```json
{"points": [{"x": 0.1, "y": 0.2}, {"x": 0.5, "y": 0.8}]}
```
**Response:**
```json
{"success": true}
```
#### `POST /api/polygon/clear`
Clear polygon overlay.
**Response:**
```json
{"success": true}
```
---
### Mask Extraction
#### `POST /api/mask/extract`
Extract mask using ComfyUI.
**Request:**
```json
{
"subject": "the wooden door",
"use_polygon": true
}
```
**Response:**
```json
{
"success": true,
"mask_path": "/home/noti/dev/ai-game-2/tools/ora_editor/temp/mask.png"
}
```
---
### Krita
#### `POST /api/krita/open`
Copy layer to temp and return file URL.
**Request:**
```json
{"layer_name": "base"}
```
**Response:**
```json
{
"success": true,
"file_url": "file:///home/noti/dev/ai-game-2/tools/ora_editor/temp/base_1234567890.png",
"temp_path": "/home/noti/dev/ai-game-2/tools/ora_editor/temp/base_1234567890.png"
}
```
#### `GET /api/krita/status/<layer_name>`
Check if temp file was modified.
**Response:**
```json
{
"modified": true,
"mtime": 1234567890
}
```
---
## Frontend State (Alpine.js)
### Main Store (`x-data`)
```javascript
{
// File state
filePath: '',
oraPath: '',
layers: [],
baseImage: '',
// UI state
selectedLayer: null,
isDrawingPolygon: false,
polygonPoints: [],
polygonColor: '#FF0000',
polygonWidth: 2,
// Mask extraction
maskSubject: '',
usePolygonHint: true,
isExtracting: false,
showMaskModal: false,
tempMaskPath: null,
// Krita
kritaLayer: null,
kritaMtime: null,
// Settings
comfyUrl: localStorage.getItem('comfy_url') || '127.0.0.1:8188',
}
```
### Layer Object
```javascript
{
name: 'door_0',
visibility: true,
src: 'data/door/door_0.png'
}
```
---
## UI Layout
```
┌──────────────────────────────────────────────────────────────────────┐
│ [Open: ________________________] [Open File] [Settings ⚙] │
├───────────────────┬──────────────────────────────────────────────────┤
│ LAYERS │ │
│ ☑ base │ │
│ ☑ door_0 │ [Image Canvas with │
│ ☐ chest_0 │ stacked layers] │
│ │ │
│ [Rename] [Delete] │ │
│ [▲ Up] [▼ Down] │ │
│ │ │
│ ─────────────────│ │
│ POLYGON TOOL │ │
│ Color: [#FF0000] │ │
│ Width: [2____] │ │
│ [Start Drawing] │ │
│ Points: 5 │ │
│ [Clear] │ │
│ │ │
│ ─────────────────│ │
│ MASK EXTRACTION │ │
│ Subject: [______]│ │
│ ☑ Use polygon │ │
│ [Extract Mask] │ │
│ │ │
│ ─────────────────│ │
│ [Open in Krita] │ │
│ │ │
├───────────────────┴──────────────────────────────────────────────────┤
│ [Save] [Refresh from Krita] │
└──────────────────────────────────────────────────────────────────────┘
```
---
## Mask Preview Modal
```
┌─────────────────────────────────────────────────────────┐
│ EXTRACTED MASK │
│ ───────────────────────────────────────────────────── │
│ │
│ [Base image with mask applied as tinted overlay] │
│ │
│ [Re-roll] [Use This Mask] [Cancel] │
│ │
└─────────────────────────────────────────────────────────┘
```
- **Re-roll**: Re-runs extraction with same params
- **Use This Mask**: Calls `POST /api/layer/add`, closes modal
- **Cancel**: Closes modal, mask discarded
---
## Settings Modal
```
┌─────────────────────────────────────────────────────────┐
│ SETTINGS │
│ ───────────────────────────────────────────────────── │
│ │
│ ComfyUI Server: [127.0.0.1:8188____________] │
│ │
│ [Save] [Cancel] │
│ │
└─────────────────────────────────────────────────────────┘
```
- Settings stored in `localStorage`
- ComfyURL stored as `comfy_url` key
---
## Implementation Tasks
### Phase 1: Project Setup
- [ ] Create directory structure (`templates/`, `temp/`)
- [ ] Create `requirements.txt`
- [ ] Create `.gitignore`
- [ ] Test virtualenv/dependencies
### Phase 2: Backend Core
- [ ] Create `ora_ops.py` with wrapper functions:
- [ ] `load_ora(path)` - Parse ORA, return layers + images
- [ ] `save_ora(ora_path)` - Rebuild ORA from current state
- [ ] `add_masked_layer(ora_path, entity, mask_path, source)` - Apply mask
- [ ] `rename_layer(ora_path, old, new)` - Update layer name
- [ ] `delete_layer(ora_path, name)` - Remove layer
- [ ] `reorder_layer(ora_path, name, direction)` - Move up/down
- [ ] `get_layer_image(ora_path, layer_name)` - Extract PNG
- [ ] `get_base_image(ora_path)` - Get merged/composited base
- [ ] Create `app.py` with Flask routes:
- [ ] File operations (`/api/open`, `/api/save`)
- [ ] Layer operations (all `/api/layer/*`)
- [ ] Image serving (`/api/image/*`)
- [ ] Polygon (`/api/polygon*`)
- [ ] Mask extraction (`/api/mask/extract`)
- [ ] Krita (`/api/krita/*`)
### Phase 3: Frontend UI
- [ ] Create `editor.html` with Tailwind + Alpine.js
- [ ] File open input and handler
- [ ] Layer list with visibility toggles
- [ ] Canvas area with stacked layer images
- [ ] Layer operations (rename, delete, reorder buttons)
### Phase 4: Polygon Drawing
- [ ] Polygon tool UI (color, width, start/clear)
- [ ] Canvas overlay for polygon rendering
- [ ] Minimal JS for click-to-add-points
- [ ] Backend polygon drawing endpoint
### Phase 5: Mask Extraction
- [ ] Mask extraction form UI
- [ ] Backend ComfyUI integration
- [ ] Mask preview modal
- [ ] Re-roll / Use This Mask / Cancel logic
### Phase 6: Krita Integration
- [ ] "Open in Krita" button and endpoint
- [ ] File URL generation
- [ ] Refresh button and logic
- [ ] Visibility-change auto-reload (JS)
### Phase 7: Polish
- [ ] Settings modal for ComfyUI URL
- [ ] Error handling and user feedback
- [ ] Loading states
- [ ] Empty states
---
## Key Implementation Details
### ORA File Structure
ORA is a ZIP file containing:
```
mimetype # "image/openraster"
stack.xml # Layer structure XML
mergedimage.png # Composited full image
Thumbnails/thumbnail.png # Preview thumbnail
data/
base.png # Base layer
<entity>/
<entity>_0.png # Masked layer
```
### stack.xml Format
```xml
<image version="0.0.3" w="800" h="600">
<stack>
<layer name="base" src="data/base.png" opacity="1.0"/>
<stack name="door">
<layer name="door_0" src="data/door/door_0.png" opacity="1.0"/>
</stack>
</stack>
</image>
```
### Image Compositing for Preview
- Frontend receives individual layer PNG URLs
- Layers stacked with CSS `position: absolute`
- Visibility toggled via CSS `opacity`
- No server-side compositing needed
### Polygon Overlay Rendering
1. Backend receives points as `[{x: 0.1, y: 0.2}, ...]`
2. Convert to absolute pixel coordinates
3. Use Pillow `ImageDraw.polygon()` to draw
4. Return overlay as PNG
### ComfyUI Workflow
Uses existing `tools/image_mask_extraction.json`:
1. Load input image
2. If polygon hint: composite polygon overlay
3. Set prompt text
4. Queue prompt to ComfyUI
5. Poll for completion
6. Download output mask
---
## Color Reference
### Default Polygon Colors
| Name | Hex |
|------|-----|
| Red | #FF0000 |
| Green | #00FF00 |
| Blue | #0000FF |
| Yellow | #FFFF00 |
### Mask Tint Colors (Preview)
Default: #00FF0080 (semi-transparent green)
---
## Error Handling
| Error | User Feedback |
|-------|---------------|
| File not found | "File not found: {path}" |
| Invalid ORA | "Invalid ORA file format" |
| ComfyUI offline | "Cannot connect to ComfyUI at {url}" |
| ComfyUI timeout | "Mask extraction timed out (4 min)" |
| Layer not found | "Layer '{name}' not found" |
---
## Browser Compatibility
Tested/supported:
- Modern browsers with ES6+ support
- Chrome/Edge (primary)
- Firefox
- Safari
Required features:
- ES6 modules
- CSS Grid/Flexbox
- Canvas API (for polygon)
- localStorage (for settings)
---
## Running the Application
```bash
cd tools/ora_editor
source ../venv/bin/activate # or your Python env
pip install -r requirements.txt
python app.py
# Opens at http://localhost:5000
```
---
## Future Considerations (Out of Scope)
- Multi-select layers
- Drag-and-drop reordering
- Undo/redo history
- Collaborative editing
- Additional export formats
- Layer groups in UI (show hierarchy)
Reference in New Issue View Git Blame Copy Permalink