delver

A crackhead StumbleUpon, in three flavors:

• web -- watches the Certificate Transparency log firehose (every HTTPS cert issued anywhere, in real time), filters out well-known and infra/CDN/SaaS domains, gives each survivor one polite front-page GET, and scores what comes back.
• youtube -- searches YouTube for old, low-view videos and scores them by obscurity (few views, old upload, near-zero engagement).
• timecop -- enrichment over what the web delver already found: asks the Internet Archive's Wayback Machine how far back each domain's history goes, and surfaces the ones with real pre-2010 roots.

All three feed into one dashboard with a review queue -- keep/skip whatever it finds, browse what you've kept.

Setup (one time)

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The web delver needs no API keys (CT logs + the Tranco popularity list are both open) and already has its CT log server vendored/built at bin-certstream-server-go (see "rebuilding" below if it's missing).

The YouTube delver needs a free Google API key -- open the dashboard and go to Setup, which walks you through getting one and saves it for you. No manual .env editing needed.

Run everything

./start.sh

Starts the CertStream server + web delver in the background and the dashboard in the foreground at http://localhost:8787. Ctrl+C stops all three. (YouTube delver is not auto-started -- see below.)

Or run pieces individually from the Runner tab in the dashboard itself (python dashboard.py), which can start/stop each process and shows live logs.

The dashboard

Page	What it's for
/ (web digest)	Everything the web delver currently has active, most recent first
/queue	Web delver's unreviewed finds, highest-scored first -- keep/skip/absolutely not
/kept	Web domains you've kept
/nope	Domains you rejected with "absolutely not" -- a harder no than plain skip
/youtube	YouTube delver's unreviewed finds -- keep/skip
/youtube/kept	Videos you've kept
/timecop	Known domains with Internet Archive history before a cutoff year (default 2010), oldest first
/runner	Start/stop each pipeline, watch live logs
/setup	First-time YouTube API key setup

Pause button (top right, always visible): the web delver runs off a live, effectively infinite CT log stream, so it'll keep piling new finds into the queue while you're still reviewing the old ones. Pause stops processing new discoveries within a second or two -- but it's not a durable spool: certificates issued while paused are read off the live stream and discarded, not queued or replayed, so they're effectively missed rather than caught up on resume. Hit resume when you're ready for more; you'll just pick back up with whatever's on the stream at that point.

The YouTube delver doesn't need a pause button -- each run is a bounded batch (--searches, default 8) that stops on its own.

Why YouTube runs differently than web

The web delver is a perpetual stream you start once and leave running. The YouTube delver burns real API quota per search (100 units, 10k/day free tier), so it's designed to be triggered a few times a day from the Runner tab rather than left running. Tune a batch with:

.venv/bin/python youtube/main.py --searches 15 --max-views 2000

How timecop works

Rather than a separate discovery pipeline, timecop enriches domains the web delver already found: for each one it hasn't checked yet, it asks the Wayback Machine's CDX API for the earliest snapshot on record (timecop/cdx.py, matchType=host catches other paths and the www-variant of the domain's front page without reaching into unrelated subdomains; collapse=timestamp:4 keeps each lookup to at most one row per archived year -- cheap regardless of how much history a domain has). A domain that's alive today but has a 2003 capture is a much stronger "interesting" signal than anything visible from one live GET.

Bounded/on-demand like the YouTube delver -- trigger a batch from the Runner tab, or run it directly:

.venv/bin/python timecop/main.py --batch 200 --rate 1

--rate caps requests/second to the Archive's API (default 1 -- be polite, same spirit as the web delver's --concurrency/--rate). Each run only looks up domains it hasn't checked before, so it's safe to re-run anytime; pass --retry-errors to also re-check ones whose previous lookup failed. Results show up at /timecop, oldest history first.

A standalone discovery mode (seeding old host platforms like GeoCities/ Angelfire directly via CDX, rather than only enriching what web already found) is planned next -- see TODO.md.

Wayback replay is not a sandbox. Confirmed in practice: browsing an archived page and clicking something on it can silently leave web.archive.org entirely and land you on whatever's currently live at that URL -- no warning, address bar just changes. The Internet Archive's own help docs say as much: when an archived page references something that wasn't itself archived, Wayback can fall back to fetching it from the live web. Old pages can also carry live client-side redirects that fire the instant you open them. So the same rule this project already applies to unvetted live domains (/queue, /digest, /flagged, /clusters -- copy the URL, don't click it) applies just as much to a /timecop snapshot link: treat it as a deliberate, isolated visit, and watch the address bar the whole time you're in there.

How it stays friendly (web delver)

• Discovery is domain-name-based (CT logs), never raw IP-space scanning.
• One logical front-page probe per discovered domain, no crawling and no linked-resource fetching -- ever. (Not literally one raw HTTP request: an HTTPS attempt can fall back to plain HTTP, and a redirect chain is followed up to MAX_REDIRECTS, each hop re-validated by the SSRF guard. What's guaranteed is no second visit, no following links past the front page, and no retrying a failed attempt.)
• Identifiable User-Agent string (see web/fetcher.py).
• Concurrency and request-start rate are both capped (--concurrency, --rate flags on web/main.py).
• The Tranco top-1M list and a hardcoded infra-domain blocklist (web/filters.py) keep it away from well-known sites and CDN/cloud noise.

Running this securely

delver's whole job is fetching pages from domains it has zero reason to trust -- freshly-issued certs are disproportionately phishing/malware/scam infrastructure -- and then showing you what it found. The fetcher itself is read-only and never executes anything it downloads (see "How it stays friendly" above), but you clicking through to a live result in a real browser is a real risk the code can't fully close for you. Two separate things to get right: where you run delver, and how you look at what it finds.

Quick / local setup

Running it on your own daily-driver machine (as in "Setup" above) is fine for casual use -- the dashboard binds to 127.0.0.1 only and has CSRF defenses (dashboard.py's TrustedHostMiddleware + same-origin guard), so nothing on your LAN or a random web page can reach or drive it. The residual risk at this tier is entirely the fetcher reaching somewhere it shouldn't (SSRF) or you opening a bad result -- see the next two sections.

Secure server setup (recommended if this box has anything else to protect)

The fetcher's SSRF guard (web/fetcher.py) validates every hostname -- initial request and every redirect hop -- against private/loopback/ link-local/reserved ranges before connecting, but it's a check-then-connect check: a DNS answer that changes between the validation and the actual connect (DNS rebinding) isn't caught at the app level. The fix is a real network boundary, not more application code: run delver on a dedicated, network-isolated host so that even a total failure of the app-level guard has nowhere to go.

deploy/README.md is the full walkthrough -- Proxmox VM on its own VLAN, nftables default-deny-outbound with an explicit RFC1918/loopback/link-local drop (deploy/nftables.conf), systemd units with sandboxing directives (deploy/systemd/), a dedicated non-privileged user, and reaching the dashboard over SSH tunnel/Tailscale instead of exposing port 8787 to anything. The short version:

• Isolate the network path (VM on its own VLAN, or at minimum a firewalled network namespace) -- this is what actually closes the DNS-rebinding gap, at the network layer instead of the app layer.
• Default-deny outbound, explicitly drop private/loopback/link-local/ reserved ranges, allow only 80/443 out.
• Run as a dedicated non-root user; chmod 700 the data/ directory (holds delver.db and the YouTube API key).
• Never expose :8787 to your LAN or the internet -- reach it via SSH tunnel or a WireGuard/Tailscale network instead.

Viewing what you find, safely

Unreviewed domains on /queue, /digest, /clusters, and /timecop are not clickable links -- clicking a domain (or, on /timecop, a Wayback snapshot) copies its URL to your clipboard instead of opening it, specifically so that reviewing a batch of unvetted sites is never a string of reflex clicks in the browser you also use for everything else. /timecop's snapshot links get the same treatment even though they resolve to web.archive.org rather than the live domain -- the Archive's playback still renders old, attacker-uncontrolled-at-capture-time markup, and this project's posture is "unvetted content stays a deliberate paste, not a reflex click," full stop. Paste the copied URL somewhere isolated:

• A separate, disposable browser profile (not your daily one -- no saved logins/cookies to leak), or
• A VM/sandbox you don't mind burning, or
• Nothing at all -- the title, meta description, and 400-byte text snippet already shown in the queue are usually enough to decide keep/skip without ever opening the live page.

/flagged (pages that tripped a static risk-signal check -- see web/risk_signals.py) never links to the live site at all; the suspicious HTML is shown as inert, escaped text so you don't need to visit it to see what tripped the flag. Once you've kept something, /kept and /archive do show it as a real link -- by then it's something you've already looked at and decided to trust, a much smaller risk than the raw queue.

If you'd rather use the public CertStream

web/main.py --certstream-url wss://certstream.calidog.io/ switches back to it. It was completely silent (connects, zero messages, indefinitely) when this was tested in August 2026, but it may recover -- worth an occasional retry since it saves you running a second process.

Rebuilding the CertStream server

If bin-certstream-server-go is missing (it's gitignored as a build artifact) or you want a newer version, and you have Go installed:

git clone --depth 1 https://github.com/d-Rickyy-b/certstream-server-go.git vendor-certstream-server-go
cd vendor-certstream-server-go && go build -o ../bin-certstream-server-go . && cd ..

certstream-config.yaml deliberately sets recovery.enabled: false -- without that, a server with no prior index backfills each CT log from entry 0, meaning it downloads a historical backlog of millions of certs per log instead of just tailing new ones live. Leave that alone.

Tuning what counts as "interesting"

Web scoring heuristics live in web/scoring.py (page size, retro HTML tags, personal-homepage language, unusual server headers, CMS detection, etc.). YouTube's obscurity scoring lives in youtube/main.py (obscurity_score). Both are deliberately simple rule-based scorers -- easy to extend as you see what the queues turn up. A natural v2 for either is swapping/augmenting the rules with an LLM call that reads the title+snippet and rates it directly.

Running tests

pip install -r requirements-dev.txt
pytest

tests/ is a golden-case regression corpus for the parts of this project that are easy to quietly break while tuning heuristics: scoring.py's classify_content()/score(), risk_signals.py's scan(), domain filtering (filters.py), the SSRF guard (fetcher.py's _host_is_public), and timecop/cdx.py's CDX response parsing. A lot of these cases are checked-in versions of real false positives/bugs found in actual use (see TODO.md) -- losing one silently passing again would mean the bug is back. A couple of tests are deliberately marked as documenting current, known, not-yet-fixed behavior rather than desired behavior (search for "known wart" / xfail) -- when those get fixed, the corresponding test is expected to need updating, not a sign something broke.

requirements-dev.txt (which pulls in pytest) is intentionally separate from requirements.txt -- the deploy/ production VM only needs the latter.

Project layout

delver/
  db.py              shared SQLite (domains, videos, timecop, pause/control state)
  dashboard.py        unified FastAPI app -- queues, runner, setup
  start.sh             one-command startup (certstream + web + dashboard)
  templates/
  web/                 CT-log domain delver
    main.py, certstream_source.py, fetcher.py, filters.py, scoring.py, risk_signals.py, trackers.py
  youtube/             YouTube obscurity delver
    main.py, keywords.py, .env (created by /setup, gitignored)
  timecop/             Internet Archive enrichment
    main.py, cdx.py
  tests/                golden-case regression corpus (pytest)
  requirements-dev.txt  pytest, dev-only, not installed on the deploy/ VM
  bin-certstream-server-go, certstream-config.yaml, vendor-certstream-server-go/
  data/                delver.db, logs/, cached Tranco list (all gitignored)
  deploy/               secure-server setup guide, nftables.conf, systemd units

Ideas not built yet

• Screenshot capture (Playwright) for the web review queue instead of just text.
• Common Crawl host list as a second web discovery source, for older/dormant sites that never triggered a fresh cert.
• LLM-based scoring for either delver instead of hand-rolled heuristics.