SITE INVENTORY · Workflow idea
For a site owner or researcher with permission to inventory a website, discover its sitemap files, follow valid nested sitemap references, clean page addresses, and save selected URLs for review. Sitemaps can contain restricted or irrelevant locations, and a listed page is not permission to crawl or reuse it. Check robots guidance, terms, page scope, and request limits; exclude private or sensitive paths; preserve source and retrieval details; and approve any follow-on collection separately.
01 / The starting point
Site-owner permission or other valid authority, a defined path scope, and an approved destination
“Prepare a clean, traceable URL inventory without crawling or reusing content beyond the approved purpose.”
02 / The work
Checks candidate sitemap locations, follows valid nested sitemap references, filters selected URLs, and records them in a sheet.
Record who approved the domain, path scope, and intended use.
Exclude restricted paths and follow the site’s stated access rules.
Preserve canonical source links and flag malformed or unexpected entries.
Collect URLs only, avoid unnecessary page content, and stop on access warnings.
Require separate permission before fetching pages, archiving content, or sharing the list.
03 / The result
A deduplicated URL list with source sitemap, retrieval date, and scope notes.
Reviews & approvals · Example outcome
A deduplicated URL list with source sitemap, retrieval date, and scope notes.
A sitemap can expose paths that should not be collected or visited. Exclude disallowed and private locations, respect robots guidance and site terms, and do not treat URL discovery as permission to scrape page content.
04 / Make it yours
A few choices shape the work around your team.
Maintain a reviewed allowlist and clear exclusions.
Use bounded discovery and stop when a site signals restrictions.
Restrict the destination sheet and remove outdated URLs when no longer needed.
Do not use a sitemap as blanket permission to copy or republish pages.