Crafting a Custom XML Sitemap That Skips Useless Pages
A default XML sitemap can quietly hold your site back. Most platforms hand you a sprawling file stuffed with tag archives, internal search results, author pages, and draft previews, and they all show up in the same submission to search engines. Crawlers spend time on those low-value URLs while the pages you actually care about compete with junk for attention. The fix is a tailored XML sitemap that lists only what deserves to be indexed.
For Australian site owners running lean operations from a home office in Brisbane or a coworking desk in Melbourne, this matters even more. Search console crawl budgets are not infinite, and the local market punishes bloated index profiles with slower rankings and thinner snippets. A trimmed sitemap that reflects your real priorities gives crawlers a sharper map of your site, helps new content reach the front page of Google Australia faster, and stops parameter-laden URLs from dragging the whole domain down.
Why a Default Sitemap Often Hurts Australian Sites
Australian websites lean heavily on WordPress, Shopify, and a growing wave of static generators hosted on Vercel or Netlify with local edges in Sydney. Each of these setups produces a sitemap by default, but rarely one aligned with editorial priorities. A typical WordPress installation will include every category, every tag, every author archive, and every paginated comment feed, even when those pages add zero commercial value. For a small business running out of Adelaide or Hobart, that clutter is more damaging than on a sprawling news outlet with thousands of staff writers.
Local SEO amplifies the problem. When you target suburbs like Parramatta, Carlton, or Fortitude Valley, you need your service and product pages to be the ones search engines see first. A messy sitemap sends mixed signals, and Google will quietly deprioritise pages that look thin or duplicated. Pair that with the ACMA-style focus on transparent, accurate indexing practices, and the case for pruning becomes hard to ignore.
Another factor is mobile-first crawling. Most Australian traffic now arrives over the NBN on phones, and Google indexes the mobile version of every page. If your default sitemap lists hundreds of URLs that only exist for desktop layouts or admin previews, you are paying for that waste twice. Cleaning the sitemap is the cleanest way to reclaim crawl efficiency before reaching for fancier structural fixes.
Markup and Required Tags for a Lean Sitemap
Every entry inside a custom sitemap must follow the official schema. At the root, you wrap the document in <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">, and each URL lives inside a <url> block. Inside that block, you place a <loc> element with the absolute URL, and you can optionally add <lastmod>, <changefreq>, and <priority> tags. Keep <lastmod> in the W3C datetime format, because Bing and Sogou both reject sloppy timestamps, which matters when you submit to multiple search engines from a single console.
A common mistake is repeating the homepage dozens of times with trailing slash variations. Pick one canonical form and stick to it. For a site targeting Australian customers, that usually means the https://www. prefix with no trailing slash on the root and trailing slashes on subpaths, matching what your canonical tags already say. Mismatched forms create soft duplicate signals that quietly erode trust.
When you want to exclude a page, you have two clean options. Either omit it from the sitemap entirely, or serve a header that returns the proper non-indexable status. The first option is preferred for utility pages, thank-you screens, and internal search results. The second is better for pages you want users to reach but crawlers to skip, such as legal disclaimers and staging previews.
| Approach | Best For | Maintenance | Skill Level |
|---|---|---|---|
| Manual XML file | Static sites under 100 pages | High manual effort | Beginner |
| WordPress plugin (rank-aware) | Dynamic blogs and small business sites | Low, automated | Intermediate |
| Custom build script | Static generators and headless CMS | Medium, version-controlled | Advanced |
| CMS-native sitemap (Shopify etc) | E-commerce stores with strict needs | Very low | Beginner |
Filtering Pages With Robots Meta and Noindex Tags
Before touching the sitemap itself, audit which pages already carry a noindex meta tag. Crawlers will eventually drop those URLs from the index even if they sit inside the sitemap, but they will still consume a slot in your daily crawl budget. The cleanest workflow is to remove them from the sitemap first and let noindex serve as a backup signal. A useful script can crawl your staging export, parse every <meta name="robots"> value, and emit only the URLs whose content is marked index, follow.
For WordPress users, the Yoast and Rank Math plugins both expose granular controls. In Rank Math, open the Titles and Meta screen and switch off "Add to sitemap" for each post type you want to hide, including attachments, author archives, and product out-of-stock pages. Yoast handles the same task through the Search Appearance settings. For Australian e-commerce stores built on WooCommerce, hiding out-of-stock variations keeps Google from indexing thousands of ghost product URLs that drag down the entire category.
Static sites need a different approach. If you use Eleventy, Astro, or Hugo, the sitemap plugin usually accepts an exclude list. In Astro, the @astrojs/sitemap integration takes a filter function that returns false for any page matching /admin, /preview, or /internal-search. Hugo's template lets you wrap the loop in a conditional that skips pages with a private: true front matter flag. Either way, you keep full control without touching raw XML.
Programmatic filtering also helps when you handle multilingual content. Australian businesses serving New Zealand or the Pacific often run parallel subdirectories, and each locale should ship its own sitemap. Generate one file per locale, list it in a sitemap-index.xml, and submit the index to Google Search Console. That structure scales cleanly and avoids the size limits, currently fifty thousand URLs or fifty megabytes per file, that have tripped up plenty of small operators running solo from a home setup.
Building the Sitemap Programmatically
When your site grows beyond a handful of pages, hand-maintaining XML is asking for trouble. A short Node.js script can read your content folder, build the URL list, and write a fresh sitemap on every build. Pull the slug list with fs.readdirSync, prepend your domain, and wrap each entry in the <url><loc>...</loc></url> template. To exclude unwanted pages, simply skip any slug whose path matches a deny-list array such as ['/admin', '/draft', '/thanks'].
For WordPress developers comfortable with a few lines of PHP, dropping a custom function into functions.php is equally effective. Hook into wp_sitemaps_post_types to unset post types you do not want, then hook into wp_sitemaps_taxonomies to remove tag and category archives that are not part of your editorial strategy. One small tweak removes hundreds of URLs from the output without breaking anything downstream.
Static generators make the cleanest pipeline. With Astro, you can run a small filter against the routes returned by getCollection, then push the surviving slugs into the sitemap integration. The build process becomes self-correcting: if a contributor accidentally publishes a draft to the wrong branch, the sitemap simply skips it, and your index stays clean. Pair this with a CI step that pings Google whenever the sitemap hash changes, and you have a setup that runs itself from a sunny Brisbane studio or a shared office in Perth.
After the sitemap is live, connect the new file to your broader automation stack. Once a fresh post goes live and lands in your clean sitemap, the natural next step is to connect an autoresponder, turning that freshly indexed URL into a subscriber pipeline rather than a one-off visit.
Testing, Submitting and Maintaining Your Sitemap
Never submit a sitemap without first validating it. Google's own Search Console accepts a sitemap file up to fifty megabytes and flags parse errors down to the line. Run your file through the XML Sitemap Validator at xml-sitemaps.com or Screaming Frog before submission, and open the file in a browser to confirm every <loc> resolves to a real, two-hundred response. Broken entries are treated as soft errors and slow down the rest of the index.
Once clean, submit the sitemap inside Search Console under the Sitemaps section for each verified property. Australian domains using a .com.au TLD should also submit through Bing Webmaster Tools, because Bing still drives a measurable slice of Australian search traffic, particularly from desktop users in Canberra and federal government searches. Add the sitemap URL to your robots.txt as a courtesy line at the bottom, and the crawlers will find it even without manual submission.
Maintenance is where most operators drop the ball. Review your sitemap quarterly and compare it against your live URL inventory. If you have deprecated any products, removed any service pages, or merged any blog categories, make sure the stale URLs are also removed from the sitemap and return a proper 410 or 301, no surprises. Treat the sitemap as a living document, not a one-time artefact, and your index will stay sharp through every algorithm update.
Build the habit today. Open your current sitemap, count the URLs you would not want a stranger to see, and prune them. Whether you are running a one-person consultancy from a Newtown terrace or scaling a national Shopify operation out of a Surry Hills warehouse, a custom XML sitemap that excludes the noise is one of the highest-leverage technical SEO moves available to Australian site owners right now. Start with a single deny-list this arvo, regenerate your file, and watch your crawl stats clean up by next week.