AI Crawlers 
RDAP rate limits and access tiers
RDAP rate limits vary wildly by registry, from 10 requests per 10 seconds to 300 a day, none of it standardized. Real per-registry numbers and what ICANN's RDRS access-tier system actually is in 2026.
Handcrafted, fresh content on all things web.
All-in-one website monitoring platform.
Welcome to the WebPixie Blog, your go-to source for insights, tips, and updates on web development, security, and uptime.
Try WebPixie Free →-
-
DNS propagation is a myth
DNS propagation isn't a transport mechanism, it's cache expiry. Negative caching, not TTL alone, explains why a fresh record can lag. Here's how to actually verify a change, and why that verification never really stops.
-
How to read an RDAP JSON response
An RDAP JSON response reads in four groups: identity, events, entities, and status, plus a few blocks you can skip. Walked through with one real response for example.com, field by field, in the order you actually need them.
-
Why Cloudflare can make your uptime monitor lie to you
A monitor that says your site is down is not always right. Cloudflare can block your monitoring bot specifically, or have its own outage, and neither one means your server is broken.
-
What Cloudflare’s 5xx error codes actually mean
Cloudflare’s 521 through 526 errors all mean the same broad thing: Cloudflare reached the internet fine, but something went wrong between Cloudflare and your own server. Here is what each code means, and what actually catches each one.
-
Why your sitemap and your crawl disagree
A sitemap is a snapshot of what your site looked like when it was generated. A crawl reflects your site right now. The two drift apart continuously, not just after a big migration.
-
Broken link audits: how often, and what to fix first
A weekly link crawl catches most broken links before they cost you traffic. When you find them, internal 404s on your highest-traffic pages come first, then redirect chains, then outbound links last.
-
WHOIS vs RDAP: what changed and why it matters
WHOIS and RDAP look up the same domain registration record, but only one of them is still the protocol ICANN actually requires. Here is the real difference and the verified 2019 to 2025 timeline that made RDAP the default.
-
ACME Renewal Information: the monitoring blind spot
ACME Renewal Information (RFC 9773) lets a CA tell your client to renew earlier than the raw expiry date suggests. Every guide treats it as an ACME-client feature. None treats it as a second signal an external monitor could watch.
-
Certificate inventory: skip the spreadsheet
Building a certificate inventory from scratch is unnecessary work. Certificate Transparency logs already record every certificate issued for your domain, in public, for free. Here is how to read that record yourself.
-
Let's Encrypt stopped sending expiration emails
Let's Encrypt ended its certificate expiration emails in June 2025. Here is why its own recommended fix does not solve the real problem, and what actually catches a renewal that fails silently.
-
Is your website agent-ready? A practical checklist
AI agents now crawl, read, and act on websites the way search engines used to. Here is a practical, standards-graded checklist for making your site agent-ready, and what to skip for now.
-
Web Bot Auth explained: what RFC 9421 signing proves
Web Bot Auth lets a crawler cryptographically sign its requests instead of relying on a spoofable User-Agent string. Here is how the RFC 9421 mechanics actually work, who really signs requests today, and what the spec still leaves unresolved.
-
DNS-AID explained: what it is and why it isn't settled yet
DNS-AID proposes publishing AI agents through DNS records instead of well-known URIs. Here is what the draft actually specifies, what the Linux Foundation launch means, and why the record shape is still inconsistent across every implementation we checked.
-
Domain monitoring vs DNS monitoring: what each catches
Domain monitoring and DNS monitoring watch two different layers of the same name: registration versus resolution. Here is what each one catches, where they overlap at the nameserver, and which layer catches which outage, so you know why you need both.
-
How to detect domain hijacking early
Domain hijacking shows up in the registration record before you lose the name: a transfer lock removed, a pendingTransfer flag, a nameserver or registrant rewrite. Here is how to detect domain hijacking early by watching WHOIS and RDAP, with the perl.com case as the lesson.
-
Domain expiry monitoring: catch it before the outage
Domain expiry monitoring reads your domain's WHOIS or RDAP record on a schedule and warns you weeks ahead, so a missed renewal never takes your site, email, and certificates offline at the same time. Here is the expiry lifecycle, how to check it, and how to alert in time.
-
How to detect DNS record changes
Detecting DNS record changes is a snapshot-and-diff loop on top of dig and the SOA serial. The hard part is not the detection, it is filtering the false positives from GeoDNS, round-robin, and CDN IP rotation so you only alert on changes that matter.
-
What DNS monitoring catches (and what it can't)
DNS monitoring catches the silent failures an uptime check never shows: records pointing at the wrong server, nameservers out of sync, expired DNSSEC signatures, broken email records. Here is exactly what it catches, what it cannot, and how to start with dig.
-
How to check SSL certificate expiration
Checking an SSL certificate's expiration takes ten seconds with a browser or openssl. The real problem: certificate lifespans have already dropped to 200 days and reach 47 by 2029, so a once-a-year manual check no longer protects you. Here is how to check, and how to automate it.
-
llms.txt and AI crawler indexability: does it work?
llms.txt still does almost nothing for AI search citations, and a 137,000-site study found 97% of files never get fetched. But in 2026 Google moved it into a new agentic browsing standard alongside WebMCP. Here is what changed, and when llms.txt is still worth adding.
-
GEO vs SEO: What actually changes in your content
GEO is SEO with one extra job: getting cited inside AI assistant responses. The 2023 paper that coined the term tested 9 content interventions. Four worked strongly, three gave moderate gains, and two flatlined or backfired, including classic keyword stuffing.
-
Should you block AI crawlers? Honest answer for 2026
Most sites should allow AI training crawlers in 2026: invisibility in AI assistant answers now costs more than uncrawled-for-training saves. Here is the per-provider breakdown (GPTBot, ClaudeBot, Google-Extended) and the robots.txt for each realistic decision.
-
Website downtime cost: deconstructing the famous numbers
The famous downtime cost numbers (Gartner's $5,600/min, IDC's $1M/hour) come from enterprise IT surveys that rarely apply to most sites. Here is where each one comes from, why it overstates the cost for most businesses, and a calculation framework that does apply.
-
How often should you check uptime: the honest answer
Picking an uptime check interval is not a question of "as fast as your plan allows." It is a question of which failure mode you are catching and how fast the world needs to know it is broken. Here is how to pick by impact, plus the alert latency math vendors rarely show.
-
robots.txt vs llms.txt vs sitemap.xml: what each is for
Three small files with three different jobs. robots.txt controls who crawls, sitemap.xml maps what to crawl, and llms.txt curates what AI assistants read first. Get the mental model right with minimum-viable examples and the mistakes that confuse them.