Skip to content

feat: cursorless UI automation — no-cursor click/type on macOS and Windows#43

Open
richard-devbot wants to merge 29 commits intoCursorTouch:mainfrom
richard-devbot:richardson/cursorless-automation
Open

feat: cursorless UI automation — no-cursor click/type on macOS and Windows#43
richard-devbot wants to merge 29 commits intoCursorTouch:mainfrom
richard-devbot:richardson/cursorless-automation

Conversation

@richard-devbot
Copy link
Copy Markdown

@richard-devbot richard-devbot commented Apr 13, 2026

Summary

Implements cursorless (zero mouse/keyboard event) UI automation for macOS and Windows as the foundation for agent workspace isolation (#29).

macOS

  • Click: tries AXUIElementPerformAction(kAXPressAction) via Accessibility API before falling back to CGEventPost
  • Type: tries AXUIElementSetAttributeValue(kAXValueAttribute, text) before falling back to click + TypeText

Windows

  • Click: tries IUIAutomationInvokePattern::Invoke() before falling back to SendInput
  • Type: tries IUIAutomationValuePattern::SetValue() before falling back to click + SendKeys

Silent fallback for: right/double/middle click, drag, non-idle caret, unsupported elements.

References

Richardson Gunde and others added 29 commits April 13, 2026 17:56
@claude
Add security hardening design doc and roadmap
8fba86c 
Comprehensive design document covering 5 phases (76 issues):
- Phase 0: CI/CD, test infrastructure, AI principles framework
- Phase 1: Critical security fixes (path traversal, JS injection, terminal, auth)
- Phase 2: AI guardrails & responsible AI (prompt injection, content filtering, ethics)
- Phase 3: Performance benchmarks & optimization
- Phase 4: Comprehensive QA (unit, e2e, adversarial, fuzzing, CI hardening)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@claude
Replace os.system() with subprocess.run() for safe process spawning
ec47ad8 
Addresses CWE-78 (OS Command Injection). Both occurrences in
control_center.py and tui.py now use subprocess.run() with
shell=True, check=False instead of os.system().

Closes CursorTouch#19

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Add bandit, gitleaks, pip-audit, coverage to CI pipeline
7f82d31 
- Add bandit SAST scan step to test job (closes CursorTouch#3)
- Add gitleaks secret detection as parallel secrets job (closes CursorTouch#4)
- Add pip-audit dependency scanning as parallel audit job (closes CursorTouch#5)
- Add pytest-cov coverage reporting with codecov upload (closes CursorTouch#6)
- Add CI badge to README.md (closes CursorTouch#2)
- Add bandit, pip-audit, pytest-cov to dev dependencies

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Add AI_PRINCIPLES.md and operator_use/guardrails/ module
f196f6f 
Implements GitHub issues CursorTouch#11 and CursorTouch#12:

- AI_PRINCIPLES.md: documents 6 core safety principles (least privilege,
  human oversight, transparency, containment, privacy by default, fail safe)
  with a development checklist for pre-merge security review.

- operator_use/guardrails/: new module providing ActionPolicy, ContentFilter,
  PolicyEngine, and RiskLevel abstractions. Includes DefaultPolicy for
  built-in tool risk classification and CredentialFilter for masking API
  keys in logs and LLM context.

Closes CursorTouch#11, closes CursorTouch#12

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Add security and e2e test suite scaffolds
60d8f63 
Scaffold test directories for issues CursorTouch#7 and CursorTouch#10:
- tests/security/: path traversal, terminal command, gateway auth tests
  with helpers for traversal/injection payloads (all skipped pending fixes)
- tests/e2e/: message pipeline tests (all skipped pending full stack)

12 tests collected, 0 errors. All skipped with tracking references.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Update uv.lock for new dev dependencies (bandit, pip-audit, pytest-cov)
7340538 
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Fix Qodo review findings: browser classification, path containment, C…
0ada8f9 
…I cache

- policies.py: Fix browser tool misclassification — tool is 'browser' with
  action arg, not 'browser_script'/'browser_navigate'. script/download => DANGEROUS,
  navigation/interaction => REVIEW (CWE-78 + CWE-22)
- helpers.py + SECURITY_ROADMAP.md: Replace startswith() with is_relative_to()
  for path containment checks — startswith has prefix-collision vulnerability
  where /workspace_evil passes startswith(/workspace)
- ci.yml: Add enable-cache: true to both test and audit setup-uv steps

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Fix CI failures: bandit, gitleaks org license, pip-audit CVEs
255066a 
bandit:
- Remove shell=True from control_center.py and tui.py — pass ["cls"]/["clear"]
  as list args, no shell needed (resolves B602 HIGH)
- Add [tool.bandit] config to pyproject.toml: skip B104 (0.0.0.0 is intentional
  LAN server binding), exclude generated vendored dirs
- Add nosec B324 to restart.py MD5 (filename only, not security)
- Add nosec B310 to fal/openai/together image providers (HTTPS API URLs only)
- Pass -c pyproject.toml in CI so config is loaded

gitleaks:
- Replace gitleaks-action@v2 (requires paid org license for orgs) with free
  gitleaks CLI v8.24.3 downloaded at runtime

pip-audit:
- Upgrade cryptography → 46.0.6, pyasn1 → 0.6.3, requests → 2.33.1,
  tornado → 6.5.5 (all have CVE fixes available)
- Add --ignore-vuln CVE-2026-4539 for pygments (ReDoS, no fix released yet)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Fix graceful_fn not passed to _do_restart on restart
1958817 
The control_center function was hardcoding graceful_fn=None when calling
_do_restart(), ignoring the _graceful_restart_fn kwarg injected by start.py.
This caused test_restart_calls_graceful_fn_not_os_exit to fail and meant
graceful shutdown was never used even when wired.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Fix path traversal in resolve() — use is_relative_to() not startswith…
a468468 
…() [CursorTouch#14]

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Sanitize download filename and URL scheme [CursorTouch#15]
b6cebdc 
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Fix XPath injection — escape all dangerous chars in JS string context […
46efa05 
…CursorTouch#16]

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Update test_utils to match secure resolve() behavior [CursorTouch#14]
8f8a4c3 
Split test_resolve_absolute_path into two cases:
- inside base (should succeed)
- outside base (should raise PermissionError)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Block sensitive JS API access in browser script action [CursorTouch#18]
4179723 
Add _check_script_safety() that blocks scripts accessing document.cookie,
localStorage, sessionStorage, XMLHttpRequest, navigator.credentials, and
other APIs that could exfiltrate auth tokens or stored credentials (CWE-94).

The check runs before execute_script() — blocked scripts return an error
explaining which API was flagged.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Fix download to use sanitized filename and enforce 100MB size limit [C…
25926ff 
…ursorTouch#15]

- Write to safe_name (os.path.basename) not original filename
- Preflight Content-Length check before download
- Post-download body size check

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Add XPath syntax validation before escaping [CursorTouch#16]
e30524b 
_validate_xpath() checks for empty, null bytes, and excessive length.
Called at all 4 _escape_xpath() call sites before interpolation.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Add symlink, unicode, null byte traversal vectors to security tests [C…
79f4aa9 
…ursorTouch#14]

- helpers.py: extend make_traversal_attempts() with null bytes, unicode
  dots, Windows-style separators
- test_path_traversal.py: add test_resolve_blocks_symlink_escape (symlink
  pointing outside workspace) and test_resolve_handles_null_bytes
- All 5 tests pass including symlink escape detection

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
Remove unused tempfile import [CursorTouch#14]
8cb0472 
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
docs: cursorless UI automation design doc and implementation plan
ee8b348
@claude
feat(macos): cursorless click via InvokePattern with coordinate fallback
d4e5813 
Left single clicks now attempt AX InvokePattern before falling back to
coordinate-based CGEvent injection. Right clicks, double clicks, and
middle clicks always use coordinates (context menus and selection need
screen position). Silent fallback on any AX failure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@claude
fix(macos): add debug logging and False-invoke fallback to cursorless…
aabbbb3 
… click

- Add logger.debug on exception fallback so failures are traceable
- Log explicitly when InvokePattern.Invoke() returns False before
  falling back to coordinate click
- Stub CoreFoundation in test sys.modules so tests pass in CI
- Add test for Invoke() -> False path

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@claude
feat(macos): cursorless type via ValuePattern.SetValue with coordinat…
2b1dcef 
…e fallback

When caret_position is idle and the element at (x,y) supports ValuePattern
and is not ReadOnly, set the value directly via SetValue -- no click, no
hardware keystrokes, no focus steal. Falls back to Click+TypeText for
ReadOnly fields, non-idle caret positions, or any exception.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@claude
feat(windows): cursorless click via InvokePattern with coordinate fal…
659acd8 
…lback

Mirror the macOS cursorless click for Windows. For left single clicks,
attempt uia.ControlFromPoint -> GetPattern(InvokePattern) -> Invoke()
before falling back to coordinate-based uia.Click. Right-click,
double-click, and middle-click always use coordinates. All exceptions
in the cursorless path fall back silently with debug logging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@claude
feat(windows): cursorless type via ValuePattern.SetValue with coordin…
7b38793 
…ate fallback

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
fix: ruff lint fixes and format for cursorless automation PR
cce6b67 
- Fix F821: add TYPE_CHECKING import for MCPManager in cli/start.py
- Fix E702: split semicolon statements in macos/desktop/service.py
- Fix F841: remove unused variables in tests/test_mcp_manager.py
- Run ruff format across entire codebase

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
fix: restore plugin hook registration and SYSTEM_PROMPT XML sections …
9276126 
…[ci]

- BrowserPlugin.SYSTEM_PROMPT: add <perception>, <tool_use>, <execution_principles> sections
- BrowserPlugin.register_hooks: actually register _state_hook on BEFORE_LLM_CALL when enabled
- BrowserPlugin.unregister_hooks: unregister _state_hook from BEFORE_LLM_CALL
- BrowserPlugin.unregister_tools: call unset_extension for "browser" and "_browser"
- BrowserPlugin.enable/disable: wire hook register/unregister through lifecycle
- ComputerPlugin.SYSTEM_PROMPT: add <perception>, <tool_use>, <execution_principles> sections
- ComputerPlugin.register_hooks: register _state_hook + _wait_for_ui_hook when enabled
- ComputerPlugin.unregister_hooks: unregister both hooks
- ComputerPlugin.enable/disable: wire hook register/unregister through lifecycle
- control_center: pass kwargs._graceful_restart_fn through to _do_restart(graceful_fn=...)
- ToolRegistry.get: also check _extensions so registry.get("browser") finds the browser instance
- ruff format: reformat entire codebase to resolve style violations

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
fix: bump aiohttp/cryptography for CVEs, suppress bandit B310 [ci]
029f2a6 
- aiohttp>=3.13.4: fixes CVE-2026-34513 through CVE-2026-34525 (DoS, header
  injection, credential leakage, NTLMv2 exposure)
- cryptography>=46.0.7: fixes CVE-2026-39892 (non-contiguous buffer OOB read)
- bandit skips B310: urllib.request.urlretrieve in openai/image.py only
  downloads URLs from OpenAI's HTTPS API — not user-supplied input

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
fix: enforce path traversal guard in utils.helper.resolve [ci]
aa693e5 
The previous resolve() silently allowed absolute paths and ../.. traversal.
This commit makes it raise PermissionError whenever the resolved path
escapes the base workspace directory, covering:
- absolute path injection (/etc/passwd)
- parent-directory traversal (../../secret)
- symlink escapes (symlink pointing outside the workspace)

Fixes the three failing tests in tests/security/test_path_traversal.py.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@claude
fix: update test_utils to expect PermissionError for absolute paths [ci]
01224c2 
test_resolve_absolute_path previously validated the old (insecure) behavior
where resolve() silently passed absolute paths through. Now that resolve()
enforces workspace containment, update the test to confirm it raises
PermissionError as intended.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@richard-devbot