improvement(polling): fix correctness and efficiency across all polling handlers

[waleedlatif1]

Summary

• Gmail: paginate History API with `do...while` loop, add `historyTypes=messageAdded` filter, throw on 403/429 instead of silently falling back, fetch fresh `historyId` from profile API when fallback search returns 0 results (breaks 404 retry loop)
• Outlook: follow `@odata.nextLink` pagination so inboxes >25 new emails aren't silently truncated, decouple `$top` page size (50) from total cap so pagination actually fires, use `fetchWithRetry` for all Graph API calls (handles 429 + Retry-After), skip folder filter on partial well-known folder resolution failure instead of producing wrong results, remove `Content-Type` from GET requests (RFC 9110)
• RSS: add conditional GET (`ETag`/`If-None-Match`, `Last-Modified`/`If-Modified-Since`) to eliminate redundant fetches, handle 304 per RFC 9111, raise GUID dedup cap 100→500, align GUID tracking fallback key with idempotency key for guid-less items, skip items with no identifiable GUID to avoid idempotency key collisions
• IMAP: reuse single `ImapFlow` connection across fetch and mark-as-read (was two connections per poll), track `UIDVALIDITY` per mailbox to detect mailbox recreation, advance UID inside fetch loop per received message, fix `messageFlagsAdd` range type, remove cross-mailbox legacy UID fallback, preserve fresh UIDs after UIDVALIDITY reset in state merge
• Polling route: remove trigger.dev dispatch path (cron → lambda → trigger.dev → poll), revert to direct Redis-locked synchronous polling; delete `background/provider-polling.ts`
• Schedules route: convert dynamic `await import()` inside `.map()` to static import, fixing a flaky concurrent-import deadlock in tests

Type of Change

• Bug fix
• Improvement

Testing

Tested manually

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Apr 9, 2026 5:53pm

[cursor]

PR Summary

Medium Risk
Changes core polling logic for Gmail/Outlook/IMAP/RSS and the polling API route, which can affect event delivery semantics and external API interactions (pagination, retries, dedupe). Risk is mitigated by mostly additive safeguards and caps, but regressions could cause missed or duplicated events.

Overview
Improves polling correctness and efficiency across providers. Gmail polling now paginates the History API (filtered to messageAdded), avoids silent fallback on 403/429, and can refresh an invalid historyId via the profile endpoint; Outlook polling now paginates via @odata.nextLink up to a capped total, uses fetchWithRetry for Graph calls, and avoids incorrect folder filtering when well-known folder resolution is partial.

RSS polling adds conditional GET support (ETag/Last-Modified) with explicit 304 handling, increases GUID dedupe retention, and tightens GUID generation/skip behavior to avoid idempotency collisions; IMAP polling reuses a single ImapFlow connection, tracks UIDVALIDITY per mailbox to reset state safely, and fixes UID/flagging handling during fetch/mark-as-read. Separately, the polling route now guarantees lock release via an inner try/finally, and schedule execution replaces a per-item dynamic import with a static getWorkflowById import to avoid flaky concurrent-import behavior.

^{Reviewed by Cursor Bugbot for commit f82c7f7. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: RSS GUID fallback diverges between tracking and idempotency
  • Added the same truthiness guard (item.title && item.pubDate) to the idempotency key computation at line 295-298 to match the tracking and filtering logic.

Or push these changes by commenting:

@cursor push 79c99ddceb

Preview (79c99ddceb)

diff --git a/apps/sim/lib/webhooks/polling/rss.ts b/apps/sim/lib/webhooks/polling/rss.ts
--- a/apps/sim/lib/webhooks/polling/rss.ts
+++ b/apps/sim/lib/webhooks/polling/rss.ts
@@ -292,7 +292,10 @@
 
   for (const item of items) {
     try {
-      const itemGuid = item.guid || item.link || `${item.title}-${item.pubDate}`
+      const itemGuid =
+        item.guid ||
+        item.link ||
+        (item.title && item.pubDate ? `${item.title}-${item.pubDate}` : '')
 
       await pollingIdempotency.executeWithIdempotency(
         'rss',

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

[greptile-apps]

Greptile Summary

This PR overhauls polling correctness and efficiency across Gmail, Outlook, IMAP, and RSS handlers, and removes the trigger.dev dispatch path in favor of direct Redis-locked synchronous polling. Changes include History API pagination for Gmail, @odata.nextLink pagination for Outlook, conditional GET for RSS, single-connection IMAP fetch+mark-as-read, and a static-import fix in the schedules route.

• P1 (Gmail): A non-fatal History API error on page 2+ of pagination discards all message IDs collected from prior pages and falls back to searchEmails, which uses a different time-based window — emails already found can be silently missed.
• P2 (RSS): Items with no identifiable GUID and no isoDate pass both the GUID dedup and date filters in fetchNewRssItems, consuming slots in the 25-item cap every poll cycle without ever being processed in processRssItems.

Confidence Score: 4/5

Safe to merge for Outlook, IMAP, RSS, and the schedule/route fixes; the Gmail pagination fallback needs a small guard before merging for high-volume inboxes.

One P1 issue remains in gmail.ts: a transient server error on History API page 2+ silently discards all messages collected from earlier pages and substitutes a time-windowed search that may not overlap, causing missed emails. All other providers and the scheduling fix look correct.

apps/sim/lib/webhooks/polling/gmail.ts — mid-pagination error handling in the do...while loop (lines 181–193)

Vulnerabilities

No security concerns identified. The IMAP host continues to be validated via validateDatabaseHost before use. RSS feed URLs remain validated through validateUrlWithDNS and fetched via secureFetchWithPinnedIP. The Redis distributed lock uses a Lua-script ownership check in releaseLock, so the finally-block release on the skip path is a confirmed no-op rather than a lock-hijack vector. OAuth tokens and credentials are handled through existing resolveOAuthCredential utilities without change.

Important Files Changed

Filename	Overview
apps/sim/app/api/webhooks/poll/[provider]/route.ts	Restructures lock management into a nested try/finally so the lock is always released even on poll errors; removes the legacy trigger.dev dispatch path. releaseLock correctly checks ownership via a Lua script, so the call in the skip-case finally is a confirmed no-op.
apps/sim/lib/webhooks/polling/gmail.ts	Adds History API pagination via do...while loop, historyTypes filter, 403/429 hard-throw, and fresh-historyId recovery on empty search fallback. A P1 bug exists: a non-fatal error on page 2+ of pagination discards all previously collected message IDs and falls back to search, risking silent message loss.
apps/sim/lib/webhooks/polling/imap.ts	Consolidates to a single ImapFlow connection per poll, adds UIDVALIDITY tracking per mailbox, advances UID in the fetch loop, and fixes messageFlagsAdd range arguments. Clean and correct.
apps/sim/lib/webhooks/polling/outlook.ts	Adds @odata.nextLink pagination, decouples page size from total cap, adopts fetchWithRetry for all Graph calls, skips folder filter on partial resolution failure, and removes Content-Type from GET requests. Correct behavior.
apps/sim/lib/webhooks/polling/rss.ts	Adds conditional GET (ETag/Last-Modified), 304 handling, raises GUID cap to 500, and skips GUID-less items in processRssItems. A P2 gap: GUID-less items with no isoDate still pass the newItems filter and consume the 25-item cap every poll without being processed.
apps/sim/app/api/schedules/execute/route.ts	Converts dynamic await import() inside .map() to a static top-level import, fixing a flaky concurrent-import deadlock. Clean change with no issues.

_{Reviews (7): Last reviewed commit: "fix(polling): decouple outlook page size..." | Re-trigger Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7c7c95d. Configure here.}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit f82c7f7. Configure here.}