Websites
Import a web page or a whole website, choose its pages with rules, preview before you import, and set how often a site syncs.
A single web page
Choose Add a resource, then the Web page tab, and paste one or more addresses. Each address becomes a resource, read the way a visitor sees it.
Lookio tries the cheapest way to read a page first and moves on only if the result is thin:
- It fetches the HTML, removes menus, headers, footers and scripts, and converts the rest to Markdown.
- If the page is an empty shell built with JavaScript, it renders the page in a browser and reads that.
- As a last resort, for pages that still come back empty or that refuse automated reads, it uses an external fetching service that processes pages outside the EU. Only the public address of the page is sent.
Lookio remembers which way worked for a domain, so the next read starts there.
Which addresses work
Only public web pages. Lookio refuses localhost, private IP ranges, internal names and anything that is not http or https.
An address is cleaned up before it is stored: a missing scheme becomes https, the host is lowercased, the fragment (#section) is dropped and tracking parameters (utm_*, gclid, fbclid and a few others) are removed. Two spellings of one page become one resource.
A page that returns a file Lookio supports, such as a PDF, is read as that file.
A whole website
A website is one resource. Its pages are not resources of their own: they do not clutter your lists, and an assistant that reads the site reads all its pages, now and after each sync.
Paste the address
Open the Sitemap tab and paste the site, one section of it (https://docs.acme.com/guide) or its sitemap.
Preview the pages
Lookio looks for the sitemaps of the site (from robots.txt, then /sitemap.xml and /sitemap_index.xml, including nested ones). Without a sitemap it follows links from your address. It shows how many pages it found and groups them by section, so you can see what a rule would keep.
Set the rules
Include or exclude parts of the site. The preview recounts as you type.
Import
The resource is created as queued. Its pages are read in the background.
Lookio respects robots.txt: pages that it forbids to all robots are left out of the preview and the import, and the preview tells you how many.
A site is limited to 10,000 pages. When a site has more, the preview says it was truncated and the resource shows “Limit of 10,000 pages reached. Refine the rules to choose which pages to import.” Narrow the rules.
Rules
Rules apply to a page’s path (with its query string), never to the full address. For https://acme.com/blog/post-1, the path is /blog/post-1.
| Rule | What it does |
|---|---|
| Include | Path patterns to keep. Empty means the whole site. |
| Exclude | Path patterns to leave out. They always win over include. |
| Regex | Advanced. A regular expression on the path. Matching pages are left out. |
| Excluded pages | Single pages left out, by full address. Up to 5,000. |
In a pattern, * matches within one level and ** matches across levels. Patterns are not case sensitive.
| Pattern | Matches |
|---|---|
/docs/** | /docs and everything below it |
/fr/* | /fr/pricing, but not /fr/blog/post |
You can set up to 50 include patterns and 50 exclude patterns. An exclusion regex that does not compile is refused, so fix it or leave it empty.
Keep it in sync
A website has a schedule.
| Schedule | Value | Behaviour |
|---|---|---|
| Manual | manual | Only when you choose Sync now in Manage sitemap (or call reindex). |
| Recurring | daily, weekly, monthly, or every_3_days, every_2_weeks, every_1_month | New pages are added, changed pages are read again and pages that disappeared are removed at that interval. |
Free workspaces use manual sync. Paid plans can choose recurring updates: 1-90 days, 1-52 weeks or 1-12 months. A month is treated as 30 days. On creation, Lookio defaults to manual on Free and weekly on paid plans. When you update rules through the API, CLI or MCP, omit the schedule to keep the current one.
A page is removed only after it was absent from two syncs in a row, so a page that is down for a day does not vanish from your assistants.
Changing the rules or the schedule starts a sync at once. Pages the new rules leave out are removed, and new ones are imported.
Check the pages
Open a website resource and choose Manage sitemap. The Pages tab lists every page with its state, failed pages first with the reason, and a search box once the list is long. It shows up to 5,000 pages. Choose Retry failed pages to read the failed ones again without a full sync, or Sync now to refresh the whole site. The Rules tab holds the include and exclude rules, with the pages they select, and the schedule.
Importing a website costs no credits. Its words count against your word allowance.
From the API, the CLI and MCP
- API: Preview a site, Create a resource, List a site’s pages, Change a site’s rules, Retry failed pages.
- CLI:
lookio sites preview,lookio sites add,lookio sites pages,lookio sites rules,lookio sites retry. - MCP:
preview_site,add_site,list_site_pages,update_site_rules,retry_failed_pages.