TLD checker Wayback checker

Wayback batcher

Downloads the full CDX result set for one host. CDX queries can return millions of rows in a single response, so this streams the data straight to disk in capped-size files instead of holding it all in memory.

Page

TLD

https://web.archive.org/cdx/search?url=example.com

Query options

From (optional)

To (optional)

Collapse

Collapse length (optional)

Exclude 4xx/5xx (filter=!statuscode:[45]..)

Fields to include in output (fl)

Client-side filtering

Applied to each row after it's fetched, since the CDX API's own &filter= is unreliable at scale.

Keep regex (one per line, matched against URL)

Remove regex (one per line, matched against URL)

Keep only these file types (by URL extension; none checked = no filtering)

Output

Rows per file (max 100000)

Output file name

Timestamp polling (enables Continue after Stop)

Idle.

Rows stream in and are written to a new file every time the row-per-file limit is hit, so memory use stays flat no matter how large the result set is. With timestamp polling off, Stop ends the download for good — Continue only becomes available after a Stop if polling is on.