Browser: POC of click_coordinates tool

[jruales]

No description provided.

[Copilot]

Pull request overview

Adds a new internal browser chat tool that can click at viewport-relative coordinates, and wires it into the existing browser toolset contribution so it becomes available alongside the other Playwright-backed tools.

Changes:

• Introduces click_coordinates tool implementation using Playwright page.mouse.click(x, y, …).
• Adds localized invocation + past-tense messages for the new tool.
• Registers the tool in browserTools.contribution.ts (tool registry + toolset).

Show a summary per file

File	Description
src/vs/workbench/contrib/browserView/electron-browser/tools/clickCoordinateBrowserTool.ts	New coordinate-based click tool (schema, messaging, Playwright invocation).
src/vs/workbench/contrib/browserView/electron-browser/tools/browserTools.contribution.ts	Registers the new tool with the browser tool registry and toolset.

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 2

[Copilot]

The invocation/past-tense messages prioritize button over dblClick. If a caller passes both dblClick: true and button: 'right'|'middle', invoke() will perform a double-click (via clickCount: 2) but the UI message will still say “Right-clicking/Middle-clicking…”, which is misleading. Consider either (1) giving dblClick precedence in the message, (2) emitting combined wording (e.g. “Right double-clicking”), or (3) rejecting that parameter combination up front.

Suggested change

	async prepareToolInvocation(_context: IToolInvocationPreparationContext, _token: CancellationToken): Promise<IPreparedToolInvocation | undefined> {
	const params = _context.parameters as IClickCoordinateBrowserToolParams;
	const link = createBrowserPageLink(params.pageId);
	const coordinateLabel = `(${params.x}, ${params.y})`;
	return {
	invocationMessage: params.button === 'right'
	? new MarkdownString(localize('browser.clickCoordinates.invocation.right', 'Right-clicking {0} in {1}', coordinateLabel, link))
	: params.button === 'middle'
	? new MarkdownString(localize('browser.clickCoordinates.invocation.middle', 'Middle-clicking {0} in {1}', coordinateLabel, link))
	: params.dblClick
	? new MarkdownString(localize('browser.clickCoordinates.invocation.double', 'Double-clicking {0} in {1}', coordinateLabel, link))
	: new MarkdownString(localize('browser.clickCoordinates.invocation', 'Clicking {0} in {1}', coordinateLabel, link)),
	pastTenseMessage: params.button === 'right'
	? new MarkdownString(localize('browser.clickCoordinates.past.right', 'Right-clicked {0} in {1}', coordinateLabel, link))
	: params.button === 'middle'
	? new MarkdownString(localize('browser.clickCoordinates.past.middle', 'Middle-clicked {0} in {1}', coordinateLabel, link))
	: params.dblClick
	? new MarkdownString(localize('browser.clickCoordinates.past.double', 'Double-clicked {0} in {1}', coordinateLabel, link))
	: new MarkdownString(localize('browser.clickCoordinates.past', 'Clicked {0} in {1}', coordinateLabel, link)),
	private getClickCoordinateMessage(params: IClickCoordinateBrowserToolParams, coordinateLabel: string, link: string, pastTense: boolean): MarkdownString {
	if (params.button === 'right' && params.dblClick) {
	return new MarkdownString(localize(
	pastTense ? 'browser.clickCoordinates.past.rightDouble' : 'browser.clickCoordinates.invocation.rightDouble',
	pastTense ? 'Right double-clicked {0} in {1}' : 'Right double-clicking {0} in {1}',
	coordinateLabel,
	link
	));
	}
	if (params.button === 'middle' && params.dblClick) {
	return new MarkdownString(localize(
	pastTense ? 'browser.clickCoordinates.past.middleDouble' : 'browser.clickCoordinates.invocation.middleDouble',
	pastTense ? 'Middle double-clicked {0} in {1}' : 'Middle double-clicking {0} in {1}',
	coordinateLabel,
	link
	));
	}
	if (params.button === 'right') {
	return new MarkdownString(localize(
	pastTense ? 'browser.clickCoordinates.past.right' : 'browser.clickCoordinates.invocation.right',
	pastTense ? 'Right-clicked {0} in {1}' : 'Right-clicking {0} in {1}',
	coordinateLabel,
	link
	));
	}
	if (params.button === 'middle') {
	return new MarkdownString(localize(
	pastTense ? 'browser.clickCoordinates.past.middle' : 'browser.clickCoordinates.invocation.middle',
	pastTense ? 'Middle-clicked {0} in {1}' : 'Middle-clicking {0} in {1}',
	coordinateLabel,
	link
	));
	}
	if (params.dblClick) {
	return new MarkdownString(localize(
	pastTense ? 'browser.clickCoordinates.past.double' : 'browser.clickCoordinates.invocation.double',
	pastTense ? 'Double-clicked {0} in {1}' : 'Double-clicking {0} in {1}',
	coordinateLabel,
	link
	));
	}
	return new MarkdownString(localize(
	pastTense ? 'browser.clickCoordinates.past' : 'browser.clickCoordinates.invocation',
	pastTense ? 'Clicked {0} in {1}' : 'Clicking {0} in {1}',
	coordinateLabel,
	link
	));
	}
	async prepareToolInvocation(_context: IToolInvocationPreparationContext, _token: CancellationToken): Promise<IPreparedToolInvocation | undefined> {
	const params = _context.parameters as IClickCoordinateBrowserToolParams;
	const link = createBrowserPageLink(params.pageId);
	const coordinateLabel = `(${params.x}, ${params.y})`;
	return {
	invocationMessage: this.getClickCoordinateMessage(params, coordinateLabel, link, false),
	pastTenseMessage: this.getClickCoordinateMessage(params, coordinateLabel, link, true),

[Copilot]

prepareToolInvocation interpolates params.x/params.y into a MarkdownString without any runtime validation/escaping. Even though the schema says these are numbers, tool inputs are untrusted at runtime; a non-numeric value could lead to confusing output or markdown injection. Consider validating Number.isFinite here as well (or escaping the formatted label) and falling back to a safe placeholder when invalid.