AI Search endpoints get custom domains, Access auth, and discover crawling

AI Search gets you from a data source to a working search endpoint quickly. This release adds what you need to put that endpoint in front of real users: your own domain, authentication, and one endpoint across several instances. It also adds crawling for sites without a complete sitemap, so your index covers everything you want it to find.

Each of the following is a new option. The previous behavior is still the default, so nothing changes until you change it.

Serve search from your own domain

A public endpoint is a URL that a site or app can query directly, with no authentication in front of it. By default that URL is a generated hostname on search.ai.cloudflare.com. You can now serve the same endpoint from a custom domain, a hostname in a zone that you own:

<span class="line"><span class="nb-shiki-wvjl67">https://search.example.com/search</span></span>

Restrict who can query your content

Once your endpoint is on your own domain, you can put Cloudflare Access in front of it. For example, you usually want to give /mcp to specific agents rather than to anyone who finds the URL. Agents authenticate with an Access service token, and people who open the endpoint in a browser sign in through your identity provider.

Search several instances from one URL

A namespace can expose its own public endpoint with /search, /chat/completions, and /mcp paths that fan out across the instances you choose:

<span class="line"><span class="nb-shiki-1t8gfj">curl</span><span class="nb-shiki-mdbnqw"> https://ns-</span><span class="nb-shiki-1itgoe"><</span><span class="nb-shiki-mdbnqw">NAMESPACE_ENDPOINT_I</span><span class="nb-shiki-140thh">D</span><span class="nb-shiki-1itgoe">></span><span class="nb-shiki-mdbnqw">.search.ai.cloudflare.com/search</span><span class="nb-shiki-dzsirb"> \</span></span>
<span class="line"><span class="nb-shiki-dzsirb">  --header</span><span class="nb-shiki-mdbnqw"> "Content-Type: application/json"</span><span class="nb-shiki-dzsirb"> \</span></span>
<span class="line"><span class="nb-shiki-dzsirb">  --data</span><span class="nb-shiki-mdbnqw"> '{</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">    "messages": [{ "content": "How do I configure AI Search?", "role": "user" }],</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">    "ai_search_options": { "instance_ids": ["docs", "support"] }</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">  }'</span></span>

Index your sites without a sitemap

Website data sources support a new discover parse type. It starts at the source URL and collects pages from both your sitemaps and the links it finds while crawling:

<span class="line"><span class="nb-shiki-1t8gfj">curl</span><span class="nb-shiki-dzsirb"> -X</span><span class="nb-shiki-mdbnqw"> POST</span><span class="nb-shiki-mdbnqw"> "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/ai-search/instances"</span><span class="nb-shiki-dzsirb"> \</span></span>
<span class="line"><span class="nb-shiki-dzsirb">  -H</span><span class="nb-shiki-mdbnqw"> "Authorization: Bearer <API_TOKEN>"</span><span class="nb-shiki-dzsirb"> \</span></span>
<span class="line"><span class="nb-shiki-dzsirb">  -H</span><span class="nb-shiki-mdbnqw"> "Content-Type: application/json"</span><span class="nb-shiki-dzsirb"> \</span></span>
<span class="line"><span class="nb-shiki-dzsirb">  -d</span><span class="nb-shiki-mdbnqw"> '{</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">    "id": "my-ai-search",</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">    "type": "web-crawler",</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">    "source": "example.com",</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">    "source_params": {</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">      "web_crawler": {</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">        "parse_type": "discover",</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">        "discover_options": { "source": "links", "limit": 5000, "depth": 3 }</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">      }</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">    }</span></span>
<span class="line"><span class="nb-shiki-mdbnqw">  }'</span></span>

To learn more, refer to the AI Search documentation.