verification log · checked 2026-10-28

We checked our own claims. Here's what was wrong.

We started the five curated guides from memory, search summaries and third-party posts. Then we checked 20 of those claims against each provider's own docs (fetched and read) or a live HTTP probe. The mistakes we found are why the live agent now reads every source itself and checks every link it cites.

11corrected
6confirmed
3still open
sources fromGitHubNotionSlackLinkedInGmail (Google)
corrected: What we believed was wrong or stale. The guides use what the source says.
confirmed: Matches the provider's own docs or a live probe.
still open: No primary source. The guides label these as estimates.
read the primary doc· 11 checks
live HTTP probe· 3 checks
tried it ourselves· 6 checks

GitHub

Developer tools

Notion

Docs / knowledge
Corrected
believedCreate a 'public integration' at notion.so/my-integrations (most tutorials).
foundNotion now calls them 'connections' (internal, public, plus personal access tokens). Public connections choose an installation scope at creation, and only 'any workspace' is Marketplace-eligible.
read the primary docdevelopers.notion.com/…/get-started/overview2026-10-28

Slack

Communication
Corrected
believedMarketplace review takes 'a few weeks'.
foundPreliminary review takes up to 10 business days, functional review up to 10 weeks for new apps, and the queue resets on each resubmission. You also can't submit until you have 10+ active workspace installs.
read the primary docdocs.slack.dev/…/slack-marketplace/slack-marketplace-review-guide2026-10-28
Corrected
believedDistribution docs live at docs.slack.dev/distribution.
foundThat URL renders Slack's 404 page. Current docs are at /app-management/distribution/. The live agent checks every link it cites for this reason.
live HTTP probedocs.slack.dev/…/app-management/distribution2026-10-28

LinkedIn

Social / marketing

Gmail (Google)

Email / Google Workspace
Corrected
believedgmail.readonly is a 'sensitive' scope (third-party verification guide).
foundGoogle's Gmail scopes page lists gmail.readonly, and even gmail.metadata, as RESTRICTED, which means CASA. Only gmail.send / add-on scopes are sensitive.
read the primary docdevelopers.google.com/…/auth/scopes2026-10-28
Confirmed
believedTesting-mode apps get 7-day refresh tokens and a 100-user cap.
foundBoth stated by Google (OAuth 2.0 overview; 'When is verification not needed').
read the primary docdevelopers.google.com/…/protocols/oauth22026-10-28
Still open
believedCASA assessment takes 1–6 weeks and costs money.
statusGoogle says 'several weeks' with no figure. One vendor (Nylas) reported about 2 weeks with SOC 2. No authoritative price found.
read the primary doc2026-10-28

Across providers

claims that cover several apps at once
ConfirmedAll five
believedAuthorize endpoints in the guides are real.
foundProbed with a dummy client_id. Google fails fast ('invalid_client'). GitHub and Notion redirect to login/install UIs. Slack and LinkedIn render HTML. Google, Slack and LinkedIn serve OIDC discovery.
live HTTP probe2026-10-28
Still openNotion · LinkedIn
believedReview timelines.
statusNeither publishes an SLA. Our ranges are labeled as estimates in the guides.
read the primary doc2026-10-28
Still openAll
believedSteps match the current console UI.
statusCreating real apps needs signed-in provider accounts, which an automated agent shouldn't create. A human click-through is the next verification step.
tried it ourselves2026-10-28

Building the agent

what broke while we built the live agent, and what we changed
Corrected
believedThe free Gemini key can do Google Search grounding for discovery.
foundReturned 429 RESOURCE_EXHAUSTED on the free key. The agent now proposes candidate URLs, then fetches and verifies each one itself, so every cited source is one it actually read.
tried it ourselves2026-10-28
Corrected
believedThe Gemini free tier is plenty for a live demo.
foundgemini-3-flash allows 20 requests/day per project on the free tier, and our test runs used that up. Cheap steps now run on a lite model, writing falls back down a model chain, and finished guides are cached for 7 days.
tried it ourselves2026-10-28
Corrected
believedAsking the model to label evidence in the prompt is enough.
foundA fallback model put claim text in the evidence field. Labels are now enforced twice: a JSON schema with enums at generation time, then code that downgrades any claim citing a source the agent never read.
tried it ourselves2026-10-28
Corrected
believedA valid citation id means the claim is supported.
foundNot necessarily: a model can cite a real page for something it doesn't say. The agent now has to return a verbatim supporting sentence, and code checks that sentence is in the page. On test runs, 28/31 (HubSpot) and 36/38 (Calendly) passed. The rest were downgraded to inferred.
tried it ourselves2026-10-28
Corrected
believedAsking the model for a few URLs is enough to find the right docs.
foundGuessed doc URLs are often stale (3 of 8 were 404 for Calendly and Dropbox). Many doc sites publish llms.txt: Zoom indexed 818 links, Calendly 106. Reading it led the agent to the actual app-review and rate-limit pages.
tried it ourselves2026-10-28

Not yet verified by anyone: the click-by-click console steps. That needs signed-in provider accounts, which the agent deliberately never creates. A human click-through is the next check before a guide gets a verified stamp.