recipes
Serp Monitoring

Recipe: SERP Monitoring

Track how a keyword's Google search result page changes - top organic listing, AI Overview, featured snippet - and get notified the moment any of them shift.

curl -X POST https://api.verid.dev/v1/monitors \
  -H "Authorization: Bearer $VERID_API_KEY" \
  -d '{
    "name": "SERP - best invoicing software",
    "url": "https://www.google.com/search?q=best+invoicing+software&hl=en&gl=us&pws=0",
    "fetch_mode": "browser",
    "schedule_interval_seconds": 1800,
    "extract_config": {
      "method": "xpath",
      "fields": {
        "top_result_title":  "(//div[@id=\"search\"]//h3)[1]",
        "top_result_url":    "(//div[@id=\"search\"]//a[h3])[1]/@href",
        "ai_overview":       "//div[contains(@aria-label,\"AI Overview\")]",
        "featured_snippet":  "//div[@data-attrid=\"wa:/description\"]"
      }
    },
    "diff_predicate": { "type": "any_field_changes" },
    "deliveries": [
      { "type": "webhook", "url": "https://your-app.com/hooks/serp-shift" }
    ]
  }'

Notes:

  • Google's results are JavaScript-rendered, so a plain fetch can never see them. Verid detects search-results URLs and renders them in a browser even on fetch_mode: "auto"; setting "browser" explicitly is still the clearest way to express the intent.
  • Pin hl, gl, and pws=0 in the URL so personalization can't introduce noise.
  • Google removed &num=100 in September 2025. A results page now holds 10 entries, so tracking deeper positions means separate monitors on &start=10, &start=20, and so on.
  • Avoid &udm=14: it renders, but result links come back wrapped in google.com/goto?url=…, so any @href field extracts an opaque blob instead of the destination.
  • When Google answers with its "unusual traffic" page instead of results, Verid records the check as blocked. It never stores the block page, so a blocked check can't fire a false change alert.
  • A refusal is not retried, because retrying it does not work: in testing, once Google had flagged an IP, 16 of 16 further attempts were refused over 70 minutes regardless of how the request was made. Instead Verid stops querying that engine for 15 minutes, doubling per consecutive refusal up to 6 hours, and checks during that window fail immediately with the reason. The first healthy check clears it.
  • Google is the hardest target here by a wide margin. If you need results you can depend on rather than best-effort, monitor Bing, DuckDuckGo or Yahoo - none of them gate the results document on JavaScript, and none of them refuse the way Google does.
  • Selectors evolve when Google ships layout changes; expect to update them occasionally.
  • For high-frequency or geo-specific tracking, configure the residential proxy layer so layer-3 fallback can route requests from the right region.

Related use case

For ranking-shift strategy, proxy/geo notes, and the full payload, see the SERP Monitoring (opens in a new tab) use case.

Price Drop AlertSitemap Monitor