Our crawler

PostelixBot

PostelixBot is the web crawler of Postelix. This page says what it reads, how it behaves and how to stop it. Postelix also reads single pages on demand under a second name, PostelixFetcher, and a few public news feeds and SEC filings under a third, PostelixFeeds.

Who we are

Postelix is a sales intelligence service run by Unfair Advantage Ltd, Meleti 2C, 8570 Pegeyia, Cyprus (registration HE 490027). It tells business customers which companies are hiring for which roles, so they can reach the right company at the right time. Company details are in our Impressum.

What PostelixBot reads

  • Public career pages and the public job feeds of hiring systems (Greenhouse, Personio, Workday and similar) that a company links from its own website.
  • A company's homepage and its Impressum or imprint page, to find the career page, the company's LinkedIn page link, its legal name and register number. From the homepage it also keeps a short summary of what the company says about itself: the page title and description, the first headings, its menu labels, links to its product, pricing and about pages, the tools the page loads (for example its CMS or chat widget) and a short excerpt of the visible main text. It does not keep a copy of the homepage itself, beyond the 12-month page copies below.
  • From readable public homepages across the company index, Postelix also uses AI to classify what the organisation itself does and write a short description of what it sells and to whom. It keeps the supporting words from that homepage. These classifications help match companies to customers’ searches.
  • From each job ad: the title, place, date, link and the text of the ad as published.
  • For companies our customers follow, about once a month, up to five pages linked from their website about the company's products, customers, industries, pricing, about, integrations or partners, including up to two public news or announcement pages within that same limit: their title, description, first headings, the visible main text and the company names on customer and partner logos. From product and company pages Postelix writes a short description of what the company sells and to whom, each point with the sentence of the page it comes from. From dated announcements it records company changes, with the source link and supporting words. Team, career, contact and legal pages are not among these five pages.

What it never reads or keeps

  • No people data. It does not read team pages or profiles. Names, email addresses, phone numbers and contact lines are removed from ad text, from the homepage summary and from the product, company and news pages before anything is stored; team, testimonial and contact-person sections are left out.
  • Nothing behind a login, a paywall or a cookie wall, and no form is ever filled in.
  • Copies of the homepage, career, imprint, product and company pages it reads are deleted after 12 months. Job ads are kept as a dated record of what the company published.

How it behaves

  • It obeys robots.txt, including Crawl-delay, and reads it again every day.
  • It makes one request at a time to a company's website, at most one every two seconds, and backs off when a server asks it to (HTTP 429 or 503 with Retry-After).
  • It reads a company's career pages about once a day.
  • It never solves a captcha, never hides what it is and never pretends to be a browser. A blocked page stays blocked.
  • It respects text and data mining reservations: the tdm-reservation header or meta tag, /.well-known/tdmrep.json, and a robots.txt Content-Signal of ai-input=no. On such a site it keeps only the title, date, link and place of each ad and no ad text, and from the homepage only its title, links and the tools it loads, no text. It reads none of its product and company pages, and Postelix writes no description of it.

How to recognise it

User agent:

PostelixBot/1.0 (+https://postelix.com/bot)

It connects only from two fixed address pairs, one per region. Frankfurt: IPv4 209.71.114.177, IPv6 2a09:8280:e612:1:0:1f1:7444:0. United States (New Jersey): IPv4 209.71.73.218, IPv6 2a09:8280:e610:1:0:1f1:7444:0. Each site is crawled from one of them at a time. A request that claims to be PostelixBot from any other address is not ours. The same addresses as a machine-readable list: postelixbot-ips.json.

How to stop it

To keep PostelixBot off your whole site, add this to your robots.txt:

User-agent: PostelixBot
Disallow: /

To let it see that you are hiring but keep your ad text out of any AI reading, add Content-Signal: ai-input=no for PostelixBot (or for *) in your robots.txt.

Or email crawler@postelix.com with your domain. We stop crawling it within 24 hours. This covers the domain and its subdomains.

Contact

Questions, problems with our crawler, or a request to remove data: crawler@postelix.com. A person reads every message. For privacy questions see our privacy policy.

PostelixFetcher

PostelixFetcher is not a crawler. It reads one company's website when a task in Postelix needs that company, for example when a customer enters their own website, or when Postelix prepares a message or a meeting with someone who works there. It is a separate program from PostelixBot and carries its own name, so you can tell the two apart.

What PostelixFetcher reads

  • At most five pages of one company's site per task: the page it was given, the homepage, and pages picked by name such as /pricing, /about or /impressum. It does not follow links it finds on a page.
  • When Postelix knows only a company's name, the homepages of up to 14 likely addresses for it, such as name.com or name.co.uk, to find the right site.
  • A single file, such as a PDF, from a link that someone sent to Postelix in a message.

How PostelixFetcher behaves

  • Each read serves the task at hand. PostelixFetcher does not come back on a schedule, only when a new task needs the same company.
  • Requests are plain GET requests over HTTPS. It never logs in and never fills in a form.
  • It does not read robots.txt. Each read is made for one task, the way a browser opens a page when someone clicks a link, so robots.txt rules, including those for PostelixBot, do not apply to it.
  • It runs on cloud servers without fixed addresses, so there is no address list for it. Recognise it by its user agent. It never calls itself PostelixBot.

How to recognise PostelixFetcher

PostelixFetcher/1.0 (+https://postelix.com/bot#fetcher)

How to stop PostelixFetcher

Refuse requests whose user agent contains PostelixFetcher, for example with a firewall rule that answers 403. It does not retry a refused request. Questions go to crawler@postelix.com.

PostelixFeeds

PostelixFeeds reads public news feeds about company funding rounds and public company filings, once a day, to note company-level events such as a funding round, a new chief executive or an acquisition. It is a separate program from PostelixBot and carries its own name.

What PostelixFeeds reads

  • The RSS or Atom feeds and news sitemaps of a short, fixed list of startup news sites. It does not follow links from them to other pages.
  • The public filing index of the US SEC (EDGAR) and the filings it lists that matter for these events, within the SEC's fair-access limits.
  • Company facts only: the company, the amount and round, the investing firms, the date and the link. It keeps no names of people.

How PostelixFeeds behaves

  • It obeys robots.txt, including Crawl-delay: the rules for PostelixFeeds, and also those for PostelixBot, so a site that keeps PostelixBot out keeps PostelixFeeds out too.
  • It respects text and data mining reservations (tdm-reservation, /.well-known/tdmrep.json, a Content-Signal of ai-input=no or ai-train=no). On such a site it keeps only the headline, date and link.
  • It makes one request at a time, at most one every three seconds to a news site and two a second to the SEC. It never logs in.
  • It runs on cloud servers without fixed addresses. Recognise it by its user agent. It never calls itself PostelixBot.

How to recognise PostelixFeeds

PostelixFeeds/1.0 (+https://postelix.com/bot#feeds; crawler@postelix.com)

How to stop PostelixFeeds

User-agent: PostelixFeeds
Disallow: /

Or email crawler@postelix.com with your domain. We stop reading it within 24 hours.