JOA-bot: the Job Opportunities API crawler
Our crawler identifies itself with the User-Agent JOA-bot/1.1 (+https://jobopportunitiesapi.org/bot; [email protected]).
What it fetches, and why
Public job openings published by employers on their own career pages and recruiting systems (ATS), so that the jobs can be listed here with a link back to the employer. To find an employer's careers page it reads the homepage and follows the careers link the site itself shows. We read robots.txt and, where a site publishes one, the standard /sitemap.xml at the site's root; we don't guess any other addresses. It then re-checks known job pages periodically, so that closed jobs are removed.
What it does not do
No logins, no forms, no personal data collection, no vulnerability scanning.
Where requests come from
We normally fetch from the addresses in our public IP list (https://jobopportunitiesapi.org/bot/ips.json). Occasionally, when needed, some requests may go through third-party proxy or VPN networks; they still identify as JOA-bot. To opt out, write to [email protected] or [email protected].
Politeness
It obeys robots.txt (User-agent: JOA-bot, Disallow rules are respected); at most one request every 5 seconds per server IP address on ordinary hosting, and a per-site limit on large shared platforms. It honours HTTP 429/503 Retry-After, and a site that answers with an error is not contacted again for the rest of the re-check cycle.
Opt out
Add "User-agent: JOA-bot" with "Disallow: /" to your robots.txt, or email [email protected] or [email protected] with the domains or IP ranges. We add them to a deny-list that every crawler component checks, usually the same day.
Contact
Public IP list (machine-readable, with last_updated): /bot/ips.json