--- title: "Websites" description: "Import a web page or a whole website, choose its pages with rules, preview before you import, and set how often a site syncs." canonical: "https://lookio.app/docs/platform/websites" updated: "2026-10-25" --- # Websites ## A single web page Choose **Add a resource**, then the **Web page** tab, and paste one or more addresses. Each address becomes a resource, read the way a visitor sees it. Lookio tries the cheapest way to read a page first and moves on only if the result is thin: 1. It fetches the HTML, removes menus, headers, footers and scripts, and converts the rest to Markdown. 2. If the page is an empty shell built with JavaScript, it renders the page in a browser and reads that. 3. As a last resort, for pages that still come back empty or that refuse automated reads, it uses an external fetching service that processes pages outside the EU. Only the public address of the page is sent. Lookio remembers which way worked for a domain, so the next read starts there. ### Which addresses work Only public web pages. Lookio refuses `localhost`, private IP ranges, internal names and anything that is not `http` or `https`. An address is cleaned up before it is stored: a missing scheme becomes `https`, the host is lowercased, the fragment (`#section`) is dropped and tracking parameters (`utm_*`, `gclid`, `fbclid` and a few others) are removed. Two spellings of one page become one resource. A page that returns a file Lookio supports, such as a PDF, is read as that file. ## A whole website A website is one resource. Its pages are not resources of their own: they do not clutter your lists, and an assistant that reads the site reads all its pages, now and after each sync. 1. **Paste the address** Open the **Sitemap** tab and paste the site, one section of it (`https://docs.acme.com/guide`) or its sitemap. 2. **Preview the pages** Lookio looks for the sitemaps of the site (from `robots.txt`, then `/sitemap.xml` and `/sitemap_index.xml`, including nested ones). Without a sitemap it follows links from your address. It shows how many pages it found and groups them by section, so you can see what a rule would keep. 3. **Set the rules** Include or exclude parts of the site. The preview recounts as you type. 4. **Import** The resource is created as `queued`. Its pages are read in the background. Lookio respects `robots.txt`: pages that it forbids to all robots are left out of the preview and the import, and the preview tells you how many. A site is limited to 10,000 pages. When a site has more, the preview says it was truncated and the resource shows "Limit of 10,000 pages reached. Refine the rules to choose which pages to import." Narrow the rules. ### Rules Rules apply to a page's **path** (with its query string), never to the full address. For `https://acme.com/blog/post-1`, the path is `/blog/post-1`. | Rule | What it does | | ------------------ | ------------------------------------------------------------------------ | | **Include** | Path patterns to keep. Empty means the whole site. | | **Exclude** | Path patterns to leave out. They always win over include. | | **Regex** | Advanced. A regular expression on the path. Matching pages are left out. | | **Excluded pages** | Single pages left out, by full address. Up to 5,000. | In a pattern, `*` matches within one level and `**` matches across levels. Patterns are not case sensitive. | Pattern | Matches | | ---------- | -------------------------------------- | | `/docs/**` | `/docs` and everything below it | | `/fr/*` | `/fr/pricing`, but not `/fr/blog/post` | You can set up to 50 include patterns and 50 exclude patterns. An exclusion regex that does not compile is refused, so fix it or leave it empty. > **Tip: Keep one language** > > To keep a single language of a multilingual site, include only its prefix, for example `/en/**`. Or exclude the others: `/fr/**`, `/de/**`. ### Keep it in sync A website has a schedule. | Schedule | Value | Behaviour | | --------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | Manual | `manual` | Only when you choose **Sync now** in **Manage sitemap** (or call reindex). | | Recurring | `daily`, `weekly`, `monthly`, or `every_3_days`, `every_2_weeks`, `every_1_month` | New pages are added, changed pages are read again and pages that disappeared are removed at that interval. | Free workspaces use manual sync. Paid plans can choose recurring updates: 1-90 days, 1-52 weeks or 1-12 months. A month is treated as 30 days. On creation, Lookio defaults to manual on Free and weekly on paid plans. When you update rules through the API, CLI or MCP, omit the schedule to keep the current one. A page is removed only after it was absent from two syncs in a row, so a page that is down for a day does not vanish from your assistants. Changing the rules or the schedule starts a sync at once. Pages the new rules leave out are removed, and new ones are imported. ### Check the pages Open a website resource and choose **Manage sitemap**. The **Pages** tab lists every page with its state, failed pages first with the reason, and a search box once the list is long. It shows up to 5,000 pages. Choose **Retry failed pages** to read the failed ones again without a full sync, or **Sync now** to refresh the whole site. The **Rules** tab holds the include and exclude rules, with the pages they select, and the schedule. Importing a website costs no credits. Its words count against your [word allowance](/docs/platform/resources#word-allowance). ## From the API, the CLI and MCP - API: [Preview a site](/docs/api/sites/preview-site), [Create a resource](/docs/api/resources/create-resource), [List a site's pages](/docs/api/resources/list-site-pages), [Change a site's rules](/docs/api/resources/update-site-rules), [Retry failed pages](/docs/api/resources/retry-failed-pages). - CLI: `lookio sites preview`, `lookio sites add`, `lookio sites pages`, `lookio sites rules`, `lookio sites retry`. - MCP: `preview_site`, `add_site`, `list_site_pages`, `update_site_rules`, `retry_failed_pages`.