Skip to main content

Add Websites

Train your agent on content from your own website — one page, a whole sitemap, or everything the crawler can reach from a starting URL.

How It Works​

When you start an import, Leezy:

  1. Discovers the pages to fetch (from links, a sitemap, or the single URL you gave it)
  2. Fetches each page and extracts the main text, dropping navigation, sidebars, footers, ads, and scripts
  3. Chunks and embeds the text
  4. Makes it retrievable for answers

Import Modes​

Go to your agent's Sources page and choose how to import:

Enter a URL and the crawler follows links from that page to discover more pages on the same site.

  • Best for help centres, documentation, and product sections
  • The exact page count is only known after the run, because new links get discovered along the way and some turn out to be invalid

Sitemap​

Point at a sitemap.xml directly, or at the site root — the sitemap is then discovered automatically.

  • Best for complete, predictable coverage of a site you control
  • Usually the fastest route to broad coverage

Fetch exactly one URL and nothing else.

  • Best for a single FAQ, pricing, or policy page
  • Useful for topping up a knowledge base without re-crawling

Crawl Limits​

Two limits apply, both by plan:

PlanPages per crawlCrawls per day
Free253
Starter5020
Pro500100
Business5,000500

Daily limits reset every 24 hours. The Sources page shows how many imports you have left today.

Your storage quota applies on top: a crawl stops early if importing more pages would exceed it.

Advanced Options (Pro and above)​

The Advanced options panel adds two controls:

Path rules​

Restrict which URLs are imported using include and exclude rules. Each rule takes an operator:

OperatorExample
Starts with/blog
Ends with.html
Containstutorial
Exact match/about
Wildcardhttps://example.com/blog/**

Include only paths limits the crawl to matching URLs. Exclude paths removes matching URLs from an otherwise broad crawl. Excludes are the usual way to keep changelogs, tag archives, and paginated listings out of a knowledge base.

Slow scraping​

Renders each page in a full headless browser instead of fetching raw HTML. Slower, but it is the setting that makes JavaScript-heavy pages and single-page apps import correctly.

:::tip When to Reach for Slow Scraping If a crawl succeeds but the imported pages contain almost no text, the content is probably rendered client-side. Re-run the import with slow scraping enabled. :::

Best Practices​

Choose Good URLs​

Ideal pages for training:

  • FAQ and help centre articles
  • Documentation
  • Product and pricing pages
  • Policy pages (returns, shipping, warranty)

Avoid These URLs​

  • Pages behind a login — the crawler is not authenticated
  • Pages that are mostly images or video
  • Tag, category, and pagination pages, which add noise without adding answers

Keep Content Fresh​

Imported pages are a snapshot taken at crawl time. When the site changes, re-run the import for that source. Crawled pages are grouped by crawl run on the Sources page, so you can delete a stale run and re-import it cleanly.

URL Requirements​

  • Must be publicly accessible without authentication
  • Must return HTML
  • https:// is assumed if you omit the protocol

Troubleshooting​

Crawl Failed​

Common causes: the URL is wrong, the page requires login, the site blocks crawlers, or the server timed out.

Solutions: open the URL in a private browser window to confirm it is public, then try a different entry point on the same site — a sitemap import often succeeds where link crawling does not.

Few or No Pages Imported​

  • Content is rendered by JavaScript — enable Slow scraping
  • Path rules are too narrow — check your include rules
  • The daily crawl limit or storage quota was reached mid-run

Too Many Irrelevant Pages​

Add exclude rules for the noisy sections, delete the crawl run, and re-import.

Next Steps​

Last updated on