Skip to content

feat: AI Generate — OpenAI-backed agent loop for widget creation#11

Open
everettjf wants to merge 8 commits intomainfrom
feature/ai-generate
Open

feat: AI Generate — OpenAI-backed agent loop for widget creation#11
everettjf wants to merge 8 commits intomainfrom
feature/ai-generate

Conversation

@everettjf
Copy link
Copy Markdown
Owner

@everettjf everettjf commented Apr 21, 2026 •
edited
Loading

Summary

Adds the AI Generate feature end-to-end: users configure an OpenAI (or compatible) API key in settings, describe a widget in natural language, and an agent loop iterates generate → run in the JSX runtime → feed errors back → regenerate until the widget renders cleanly. User then reviews a live preview, optionally refines it with another prompt, and saves into the Scripts library.

Design doc is in docs/ai-generate.md (first commit on this branch).

What's included

Shared layer (Shared/ScriptWidgetRuntime/AI/)

File Purpose
AISettings.swift UserDefaults-backed store for API key / baseURL / model / maxIterations / temperature, on the existing group.everettjf.scriptwidget app group.
AIReferenceSnapshot.swift Samples Script.bundle/component/* and api/* at startup so the system prompt always teaches the current DSL surface.
PromptBuilder.swift AIWidgetSize enum + system prompt rules + first-turn / fix-turn / refine-turn user prompt templates + fenced-code stripper.
AIClient.swift Actor wrapping SwiftOpenAI. Uses overrideBaseURL for custom endpoints so Azure / DeepSeek / Ollama work.
AgentRuntimeBridge.swift Serial queue that persists generated JSX into a temp ScriptWidgetPackage and drives executeJSXSyncForWidget, capturing console logs + typed errors.
AgentLoop.swift The fresh / refine / fix loop. Re-uses only [system, firstUser, lastAssistant, lastUserFix] per turn so context stays bounded.
AIGenerateSession.swift @MainActor ObservableObject state machine: .idle / .thinking / .running / .fixing / .done / .exhausted / .failed / .cancelled.
AIGenerateProgressView.swift Cross-platform progress + iteration history card.

iOS — SettingAIView (under Settings → new "AI" group), new AIGenerate/ directory with AIGenerateView + AIReviewView, and a "✨ Generate with AI" entry at the top of CreateGuideView.

macOS — new Settings/SettingAIView hosted in the standard Settings {} scene (⌘,), new AIGenerate/AIGenerateWindowView with split-view layout (prompt + progress + preview + refine), and a sparkles toolbar button in SidebarView.

Xcode projects — SwiftOpenAI SPM dep added to both projects' main app targets only (widget / share extensions are not linked). New files registered in Sources build phases; file references grouped under a new logical AI group pointing at ../Shared/ScriptWidgetRuntime/AI.

Behavior

  • Defaults: gpt-4o-mini, max 20 iterations (user-adjustable 5–100), https://api.openai.com, temp 0.7.
  • Custom endpoints: user sets baseURL; trailing /v1 or / is normalized off.
  • Success detection: runtime error is nil, element is not a #UI Not Found# / #Failed# / #Loading# fallback, and logs contain no [error] / uncaught markers.
  • Failure replay: the next turn re-sends the last code + typed error summary + last 10 log lines.
  • Save flow: review page uses sharedScriptManager.createScript(...) to commit into the real Scripts library and posts scriptCreateNotification so the home list refreshes.

Build status

  • iOS: ScriptWidget, ScriptWidgetWidget, ScriptWidgetShare all build cleanly on iphonesimulator with the current Xcode.
  • macOS: ScriptWidgetMac hits a pre-existing dependency-graph failure (swift-nio / MultipartKit unsendable-userInfo errors) that also reproduces on the unpatched main tree under the same Xcode version. Root cause is Vapor's open version range pulling packages that no longer compile under current Swift strict-concurrency. That is orthogonal to this feature and should be addressed in a separate PR (e.g. pin Vapor to a newer release or replace the Monaco editor web service with a non-Vapor static file server).

Test plan

  • open iOS/ScriptWidget.xcodeproj → pick ScriptWidget scheme → build + run.
  • Settings → AI → paste key → "Test Connection" returns OK.
  • "+" (new widget) → "Generate with AI" → describe a widget → watch progress iterate → preview renders → "Save Widget" creates it and shows up in the Scripts list.
  • Ask a refine in the review page; confirm the agent iterates again on top of the current code.
  • Confirm widget / share extensions still run (they must not link SwiftOpenAI).
  • For macOS: once the Vapor dep issue is resolved in a separate PR, retest ⌘, → AI settings and the sparkles toolbar button.

🤖 Generated with Claude Code

everettjf and others added 3 commits April 21, 2026 10:28
@everettjf @claude
docs: add AI Generate feature design
1d53c30 
Plan for a settings-configured OpenAI key plus an AI-driven widget
creation flow that runs an agent loop (generate → run in JSX runtime →
feed errors back → regenerate) before handing the result to the user
for review, preview, and save.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@everettjf @claude
feat(ai): add shared AI layer and iOS/macOS UI for widget generation
15f1d22 
- Shared: AISettings store, SwiftOpenAI-backed AIClient, reference
  snapshot builder (samples Script.bundle for prompt context),
  PromptBuilder, AgentRuntimeBridge, AgentLoop, AIGenerateSession,
  cross-platform AIGenerateProgressView.
- iOS: SettingAIView, AIGenerateView and AIReviewView; wired into
  SettingsView (new AI group) and CreateGuideView (new "Generate with
  AI" entry).
- macOS: SettingAIView (hosted in Settings scene, Cmd+,),
  AIGenerateWindowView (split-view sheet with prompt, live progress,
  preview, refine, save), and a sparkles toolbar button in SidebarView.

Project file (.pbxproj) registration and SwiftOpenAI SPM dependency
follow in the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@everettjf @claude
feat(ai): register AI files + SwiftOpenAI SPM dep in iOS/macOS projects
c0a401f 
Wires the new AI layer into both Xcode projects:

- Adds the SwiftOpenAI Swift Package (from: 4.4.9) to the
  ScriptWidget and ScriptWidgetMac main-app targets. Widget and share
  extensions intentionally do NOT link it.
- Creates a new "AI" group under Shared/ScriptWidgetRuntime pointing
  at ../Shared/ScriptWidgetRuntime/AI and registers the 8 shared
  Swift files in both projects.
- iOS: registers SettingAIView, plus a new "AIGenerate" group with
  AIGenerateView and AIReviewView.
- macOS: registers new "Settings" and "AIGenerate" groups holding
  SettingAIView and AIGenerateWindowView respectively.
- Adjusts AIClient to the current SwiftOpenAI API: the custom-baseURL
  path uses overrideBaseURL (String-based apiKey), and response
  `choices`/`message` are now optionals.

iOS: ScriptWidget, ScriptWidgetWidget, and ScriptWidgetShare all
build cleanly.
macOS: pre-existing Vapor/MultipartKit/swift-nio incompatibility
with the current Xcode SDK blocks building ScriptWidgetMac for
reasons orthogonal to this feature (the unpatched project.pbxproj
fails with the same errors on a clean build). Resolving that is
outside the scope of this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@everettjf everettjf changed the title docs: design for AI Generate feature feat: AI Generate — OpenAI-backed agent loop for widget creation Apr 21, 2026
everettjf and others added 5 commits April 21, 2026 21:58
@everettjf @claude
feat(ai): starter example prompts on the generate screen
e4fdf89 
Adds AIExamplePrompts with 8 curated starters (Weather, Clock,
Countdown, Crypto Price, Battery Ring, Quote, Steps, Habit Grid)
that each describe concrete colors, data sources, and layout so the
agent loop converges quickly.

Surfaced as a horizontal chip row on both iOS AIGenerateView and
macOS AIGenerateWindowView; tapping a chip fills the prompt field
and sets the recommended widget size.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@everettjf @claude
feat(ai,macos): parity polish — exhausted banner, default save name, …
ff2c779 
…menu command

Closes remaining parity gaps with the iOS flow:

- Shows a "Did not fully converge" banner above the preview when the
  agent loop exhausts its iteration budget, matching AIReviewView on
  iOS.
- Prefills the save-name field with "AI yyyy-MM-dd HHmm" on open and
  whenever the generated JSX changes (only if still empty), so the
  Save button is reachable without typing.
- Adds File → "Generate Widget with AI..." with the ⌘⇧N shortcut,
  routed through a notification so SidebarView handles the
  not-configured case consistently with the toolbar button.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@everettjf @claude
fix(macos): drop Vapor, serve Monaco editor via WKURLSchemeHandler
c5e3865 
Vapor's only job on macOS was to spin up a localhost HTTP server on
port 23355 that served Monaco editor static files from
Editor.bundle/static. That came with a large transitive dep graph
(swift-nio, MultipartKit, WebSocketKit, etc.), and the version
combination compatible with Vapor's open `from: 4.0.0` range no
longer builds against the current Xcode SDK under Swift 6 strict
concurrency — MultipartKit's retroactive Sendable conformance on
FormData types is rejected by the compiler, and older swift-nio
pins fail CNIODarwin module imports.

This replaces the whole HTTP server with a tiny WKURLSchemeHandler:

- Adds EditorSchemeHandler that serves the `scriptwidget-editor://`
  scheme directly from Editor.bundle/static (with a path-traversal
  guard and MIME-type mapping).
- Registers the handler on WKWebViewConfiguration in
  EditorInternalWebView.init().
- editorWebServiceUrl() now returns a scriptwidget-editor:// URL.
- AppDelegate no longer starts a background HTTP server.
- Removes Vapor from the project's SPM graph (XCRemoteSwiftPackage
  Reference, product dependency, packageProductDependencies entry,
  and Frameworks build-file entry).

Result: ScriptWidgetMac and ScriptWidgetMacWidget both BUILD
SUCCEEDED under the current Xcode, and 20+ transitive packages
(Vapor, MultipartKit, WebSocketKit, RoutingKit, ConsoleKit,
AsyncKit, swift-crypto, swift-certificates, swift-asn1) drop out of
the dep graph entirely.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@everettjf @claude
feat(ai,macos): surface AI generate inside the "+" create sheet
e07d9f0 
The sparkles toolbar button was easy to miss. Bring the AI entry
point into the same flow iOS uses — first thing users see when they
open the create sheet.

- macOS CreateGuideView now opens with a prominent "Generate with
  AI" card on top of the existing blank-widget name input. Tapping
  it dismisses the sheet and posts the open-request notification so
  SidebarView handles the not-configured alert path uniformly.
- Promotes the sidebar toolbar button: now uses `wand.and.stars` with
  a Label ("Generate with AI" + icon) so it shows a text affordance
  instead of a bare symbol, placed before the "+" button. Adds ⌘⇧N
  hint in the help tooltip.
- Relabels the existing blank-create button to "Create Blank" so the
  choice between blank vs AI is obvious.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@everettjf @claude
feat(ios,macos): template gallery, onboarding, and remix (P0 UX)
b0f4445 
Lower the ramp-up cost for new users by packaging the 45 bundled
templates as a browsable gallery with categories, a first-run guide,
and a one-tap way to fork existing widgets.

- Add ScriptMetadata (category / tags / difficulty / icon / featured)
  loaded from per-template meta.json; 44 meta files added.
- Replace flat "Create from template" list with a searchable grid +
  category chips on both iOS and macOS. macOS gains a full gallery
  (was AI + Blank only).
- Replace empty-state placeholders with onboarding: hero, how-it-works,
  featured templates.
- Add "Remix" (duplicate) — iOS swipe action, macOS context menu —
  backed by ScriptManager.duplicateScript.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@everettjf