Skip to content
Get a free preview
Get a free preview Sign in

Crawler

ViplorBot

ViplorBot is the crawler used by Viplor to analyze online stores at the request of their owners. How to identify it and how to block it.

What ViplorBot is

ViplorBot is the web crawler of Viplor (https://viplor.app), a service that checks how well an online store's product pages can be understood by AI shopping assistants and search engines. It visits a store only when someone asks Viplor to analyze that store, for example the store's owner or team using the free preview or a paid audit on viplor.app. It is not a general search engine crawler, it does not build a public index, and it does not collect data to sell or to train models.

Its User-Agent looks like this:

ViplorBot/0.1.0 (+https://viplor.app/bot)

The version number changes over time; the name ViplorBot and this page's address stay the same. The free preview on viplor.app can be requested by anyone for any public web address. If ViplorBot visited your site and you did not expect it, please tell us (see Contact below).

What it reads

  • robots.txt of the site, first, and it follows it.
  • The XML sitemap(s) listed in robots.txt, or /sitemap.xml when none is listed and robots.txt allows it.
  • The home page, and a small sample of product pages (at most 20 per collection run).
  • On those pages: the visible text, the page title and the structured product data (JSON-LD, such as name, price, availability, identifiers).

It only uses http and https on the standard ports. It does not sign in, does not send cookies, does not fill in forms, does not run JavaScript, does not add anything to a cart or check out, and does not download images or other media.

How often

Only when an analysis is requested; a store is not crawled on a schedule of our own. Each collection run makes at most 45 requests in total, one at a time, with a pause of at least half a second between two requests to your server, a 10 second timeout per request, and it stops after 2 minutes. A store audit or free preview may use more than one collection run. A product analysis uses one collection run per attempt; retries may add runs. Pages larger than 2 MB are not read.

How to block it or slow it down

ViplorBot obeys robots.txt. To block it from your whole site, add:

User-agent: ViplorBot
Disallow: /

To block only some paths:

User-agent: ViplorBot
Disallow: /private/
Disallow: /cart

If your robots.txt cannot be read (a server error), ViplorBot does not crawl your site at all. If there is no robots.txt (404), it treats the site as open. The Crawl-delay directive is not interpreted; if the default pace is too fast for your server, write to us and we will adjust it for your site.

You can also block the User-Agent ViplorBot in your web server or firewall. We do not publish a fixed list of IP addresses at this time, so please identify ViplorBot by its User-Agent.

AI crawlers and your robots.txt

ViplorBot is Viplor's crawler. Other AI and assistant crawlers are run by other companies. Viplor does not control those bots and does not measure your store's real presence, ranking, recommendations or sales in any assistant. Allowing a named bot does not guarantee citations or shopping recommendations.

Use this checklist on your own robots.txt:

  1. Open https://{your-store}/robots.txt.
  2. For each bot below: is there a specific User-agent: group? Does it Disallow: / (or product paths)? Does a broad User-agent: * with Disallow: / block the site?
  3. Blocking a bot is your right; it may also mean that assistant cannot fetch those pages when that bot is used.
  4. Keep ViplorBot rules separate from the tokens below.

Common robots.txt tokens for OpenAI, Google and Copilot (Microsoft):

  • GPTBot, ChatGPT-User and OAI-SearchBot, OpenAI
  • Googlebot and Google-Extended, Google
  • Bingbot, Microsoft, for Copilot

Example using GPTBot:

User-agent: GPTBot
Allow: /products/
Allow: /collections/
Disallow: /cart
Disallow: /checkout
Disallow: /account

Viplor Agents score methodology: the score uses 19 binary Viplor criteria. B4 is worth 11 points and requires a valid GTIN of 8, 12, 13 or 14 digits with a valid check digit. MPN and SKU do not count. Robots follow four cases. A named ViplorBot opt out means no score and an opt out message. A platform default rule, such as Shopify Disallow for /cart, is Not checked. A merchant added rule for all robots that closes a page useful for buying is Not met, with the rule stored. Anti bot blocking makes C1 Not met; other points are Not checked.

Contact

Questions, unexpected visits or requests to stop: hello@viplor.app

Please include your site's address and, if you have them, the times of the requests. We reply within 2 business days.