De RenoPal-crawler

Kort, in het Nederlands

RenoPal is een Belgische prijsvergelijker voor doe-het-zelf- en bouwmaterialen. Onze crawler haalt publieke productpagina's op om prijzen te vergelijken, en verwijst bezoekers altijd door naar uw eigen winkel.

• We respecteren robots.txt zonder uitzondering.
• We halen maximaal één pagina per 15 seconden op per winkel — en per 20 seconden bij GAMMA. Dat zijn ondergrenzen: trager kan, sneller niet.
• We nemen geen productbeschrijvingen of afbeeldingen over.
• Bij een 403, een challenge of een blokkade stoppen we; we omzeilen niets.
• robots.txt is de snelste weg om ons te vertragen of tegen te houden: die werkt automatisch en u hoeft ons niet te contacteren. Een bericht aan bot@renopal.be wordt door de beheerder zelf gelezen en handmatig doorgevoerd.

What this crawler does

RenoPal (https://renopal.be) is a Belgian price- and specification-comparison site for DIY and building materials. The crawler fetches publicly accessible product pages from Belgian retailers to extract price, EAN/GTIN, product name, brand and specification tables, so a visitor can compare the same product across shops. Every result links out to the retailer; we do not sell or redistribute catalogues, and we never reproduce product descriptions or images.

How it identifies itself

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) RenoPal/0.1 (+renopal.be)
Web Bot Auth (RFC 9421 HTTP Message Signatures): Requests may carry Signature, Signature-Input and Signature-Agent headers signed with Ed25519, using tag="web-bot-auth". Our public key set is published at /.well-known/http-message-signatures-directory, and that document is signed by the same key. Key ID (JWK thumbprint): cs0o8zjtY1XqJ1JUNzfmJZovt_DWb30HZrvl22eATlk
Signature-Agent: https://renopal.be

How it behaves

• robots.txt is fetched and obeyed for every host, with no exceptions. Disallowed paths are never requested.
• One request at a time per domain, with a minimum of 15 seconds between requests to the same retailer (20 for GAMMA), and exponential back-off on errors. Those are floors: an operator can slow a store down, but cannot speed one up past the value published below.
• Product pages are discovered from each retailer's own sitemap, never by querying their on-site search.
• A refusal ends the crawl. On HTTP 403, or a 429 serving a bot-protection challenge, we abandon the host rather than retry or work around it. We do not render or solve challenges, rotate addresses, or impersonate another crawler or verified bot. RenoPal identifies itself as RenoPal — by the named token in its User-Agent, and cryptographically via Web Bot Auth.
• Pages are re-fetched at most every few days, and a page seen recently is skipped entirely.
• We store only non-personal product and price data.

Current request intervals

The minimum delay between two requests to the same retailer. These are floors — an operator can raise them but not lower them below what is shown here.

Retailer	Min. interval
bouwdepot	15s
bouwkampioen	15s
brico	15s
gamma	20s
hubo	15s
klium	15s
lidl	15s
toolstation	15s
vanderschueren	15s
vostools	15s
zelfbouwmarkt	15s

How to control or stop it

1. robots.txt — a Disallow for User-agent: * is honoured immediately, on the next fetch of your robots file. A Crawl-delay is respected too.
2. Email bot@renopal.be — to ask for a slower rate, a narrower set of pages, or removal altogether. To be clear about what this is: there is no automated opt-out behind this address. It is read by the operator and the change is applied by hand. No reason is required, and removal is applied per retailer. If you want an immediate, self-service stop, use robots.txt above — that one needs nobody's attention.
3. Block us — a 403 is treated as final. We stop and do not attempt to get around it.

If anything on this page is inaccurate about traffic you are seeing, we would rather hear about it than have you guess — bot@renopal.be.