I run Rivale.io, a competitive-intelligence tool for WordPress plugin founders and developers. Part of what it does is watch competitors’ marketing pages and tell you when something moves: a pricing change, a new add-on, a repositioned homepage.
One morning I asked what should have been an accounting question. My daily ops email showed the crawler’s credit burn jumping around day to day, 2 one day and 6 the next, with nothing new added and the same short list of monitored pages.
I wanted to know why I was paying for a crawl of pages that hadn’t changed.
The answer turned out to be two separate bugs sitting on top of each other, and the expensive one was the less serious one. Underneath the cost problem was a correctness problem. My system had sent users 46 “competitor changed” alerts for changes that never happened, including the same fake discovery announced ten separate times since June. Nothing errored and no test failed, because the fake alerts were structurally identical to real ones.
This guide covers the phantom events first, then the redirect that was disabling free HTTP caching.
Architecture at a glance
The component that caused the damage is the first box, the “what did we see last time?” lookup. It fed both the cache pre-check and the change classifier from a single query, and those two consumers need different answers.
Part 1: The classifier that described instead of comparing
The symptom
The events feed carried repeated announcements of the same non-event. One competitor’s “Add-ons marketplace introduced” fired on 13 June, 16 June, 30 June, 15 July, 18 July, 26 July, 3 August, 6 August, 12 August, and 15 August.
That company did not launch an add-ons marketplace ten times.
In the 30 days before the fix, the events table held 29 events built from only 6 distinct titles. And 46 change events immediately followed a row whose crawl status was failed or skipped_head.
The root cause
The worker read its previous state with a single query for the newest crawl row, regardless of that row’s status:
// The read that caused it: newest row of ANY status
const { data: lastResult } = await db
.from('<crawl_results>')
.select('content_hash, etag, last_modified, diff_excerpt, content_markdown')
.eq('url', url)
.order('created_at', { ascending: false })
.limit(1)
.maybeSingle();Two kinds of row carry no content_hash: a failed crawl, and a skipped_head row (the cheap “nothing changed” path). Both are legitimate rows. Both are frequently the newest.
So after any failure, lastHash came back null. That made the hash gate impossible to satisfy, which forced a scrape and then an AI call. And the AI was handed prevContent = null:
const prevContent = lastResult?.content_markdown ?? lastResult?.diff_excerpt ?? null;The classifier’s prompt has a legitimate branch for “no previous version”. It describes the page instead of comparing two versions. That description comes back as a label, changeDetected flips to true, and an event fires at the user.
Warning: This is not a model failure. The prompt behaved exactly as designed. The defect is upstream, in a data path that hands the classifier a null prior state after a failure, rather than only on a genuine first sighting. A guard did exist for first sightings, but it tested a first-sighting flag, which post-failure runs do not set.
One failed fetch therefore manufactures a fake change on the next successful run. At discovery, over a third of the monitored URLs had a hash-less newest row, including every one of the flaky ones.
The fix
Split one query into two, because they answer different questions. The shipped code documents the reasoning inline:
// ─── STEP 3: Fetch last crawl state — TWO reads, two different questions ──
// These must not be one query. "What are the freshest cache validators?" and
// "what was the last content we actually saw?" have different correct answers:
//
// • etag/last_modified — want the NEWEST row of ANY status. Failure rows do
// carry validators, and they are fresher. Feeding a stale ETag to
// headCheckUrl makes it report changed:true and forces a full paid scrape.
// • content_hash/markdown — want the newest row that actually HAS a hash.Validators want the freshest row whatever its status. Content wants the freshest row that actually holds content. Reading both from one query means one of them is always wrong.
Verification
After deploying, all three URLs that had been returning NULL recovered a real hash, while the one site with cache headers kept its fresh ETag from the newest row. Both halves of the split behaved as designed against live data.
Part 2: The trailing slash that turned off free caching
The symptom
Credit burn on every run, against a fixed list of pages that mostly did not change. Over 30 days, 128 crawl runs produced only 28 AI calls, so the hash gate was doing its job at 78%. The money was going out earlier in the pipeline, because the free pre-check almost never fired.
What did not work
Two plausible theories were tested and discarded before the real one.
Theory 1: the sites are blocking us, so stop retrying. Disproven by reading 30 days of failures. There were zero 403s or blocks. Every failure was HTTP 408, HTTP 502, or an aborted signal, all timeout-class. The block-retry path never fired at all.
Theory 2: normalize the rotating asset tokens. Pages carried ?ver= query strings on assets that changed between fetches, defeating a naive content hash. Stripping them with a regex worked mechanically, and was abandoned anyway:
Note: Every new site adds another pattern to strip, forever, and over-stripping silently hides a real change. A missed skip costs one credit, an over-strip hides a change you needed to know about, and that asymmetry settles the question. Let the origin server answer it instead.
The root cause
The orchestrator stripped the trailing slash from each stored URL to build a dedup key, then crawled that stripped URL. On WordPress-style sites, the slashless form does not serve the page. It answers 301.
The HEAD pre-check pins redirect: 'manual' as an SSRF guard, and treats any 3xx as inconclusive. Inconclusive means “fall through to a full paid scrape.” So the free path was unreachable for every URL that redirected.
Four decisions, each correct on its own, chained into a bug:
The dotted path is the one the servers were offering the whole time.
The probe that settled it, run against the exact URLs in production:
for u in "https://example-a.com/" "https://example-a.com/add-ons/" "https://example-a.com/add-ons/premium/"; do
hdr=$(curl -sIL --max-time 20 "$u")
etag=$(printf '%s' "$hdr" | grep -i '^etag:' | tail -1 | sed 's/^[Ee][Tt][Aa][Gg]: *//' | tr -d '\r')
echo "$u ETag='$etag'"
if [ -n "$etag" ]; then
code=$(curl -s -o /dev/null -w '%{http_code}' -L --max-time 20 -H "If-None-Match: $etag" "$u")
echo " conditional GET -> HTTP $code"
fi
doneEvery one of those URLs returned a free 304 on the trailing-slash form. The same pages, slash stripped, returned 301 and were treated as unknowable.
The measurement recorded in the shipped code comment is that the single URL that reached its ETag got 60% free skips, and the three that redirected got 0%.
Important: This is not limited to the two competitors above. Every
/pricingpage tested on the other monitored sites answered 301 without its trailing slash. If you normalize URLs anywhere upstream of an HTTP cache check, you have probably disabled that check.
The fix
Dedup on a slash-insensitive key, but crawl the URL exactly as the user stored it. The key and the fetched URL are different things and were being conflated.
Adding a slash everywhere was rejected as a guess that breaks query-string and file-extension URLs. The worker follows the redirect once instead, resolving canonically where SSRF validation already lives.
A second layer shipped alongside it, because several of the monitored sites serve no cache headers at all: a raw-hash pre-check that does a free GET over our own egress, hashes the body, and skips the paid scrape on a match. The redirect fix helps cooperative servers, the raw hash covers the rest.
A dead sentinel
A failure-sentinel constant was read in the hash comparison but never written anywhere. Zero rows carried it. The ordering hazard is that repairing the read path is exactly what would have made that dead branch live. Dead code is not automatically safe to fix.
Troubleshooting reference
| Symptom | Root cause | Fix |
|---|---|---|
| Same “change” announced repeatedly, weeks apart | Prior-state lookup returns a failure row with no content hash, so the classifier gets null and describes instead of comparing | Split the lookup: newest-any-status for validators, newest-with-content for the hash |
| Change events cluster right after failed crawls | A failure row is the newest row, poisoning the next run’s gate | Filter the content read on a non-null hash |
| Cache pre-check never returns 304 despite server sending ETags | URL normalization produces a redirect, and the guard treats 3xx as inconclusive | Dedup on a key, fetch the URL as stored |
| A guard branch never fires | The sentinel it tests is read but never written | Confirm with a count before trusting the branch, and expect fixing the read to make it reachable |
Verification checklist
| Component | Test action | Expected result |
|---|---|---|
| Prior-state split | Run the worker on a URL whose newest row is a failure | A real content hash is recovered, not null |
| Cache validators | Same run, on a site that serves ETags | Fresh ETag still read from the newest row of any status |
| Conditional GET | curl -H "If-None-Match: <etag>" on the stored URL form | HTTP 304 |
| URL canonicalization | curl -sI the stripped form and the stored form | Stripped form 301s, stored form 200s |
What two weeks of live data said
The phantom events stopped. In the 30 days before the fix, 24 of 81 successful crawls produced a change event. In the two weeks after, 1 of 16 did, and that one was a real hero-section rewrite with a distinct content hash.
The cost prediction did not land the way I expected. Free skips ran at 4 of 31 runs after the fix versus 18 of 125 before, which is no measurable improvement yet. The redirect fix and the raw-hash layer are both in place and both fire in the logs, but the sample is small and a third of the runs since still ended in a timeout before any header came back. The 60% versus 0% measurement in the code comment is the evidence that the fix is right. I will report the burn line once there is enough post-fix data to measure it.
What to take from this
- A classifier handed no prior state will describe rather than compare, and a description reads as a change. The guard belongs on the data path, because the prompt behaved as designed.
- URL normalization upstream of an HTTP cache check can silently disable it, and the only symptom is a bill that nothing errors on.
- Both bugs produced correct-looking output, and they surfaced only because I checked the burn number against the crawl table.
Raw draft by Claude from session transcripts. Edited, fact-checked, and finalized by Matteo Duò.