Scraping

Josh talks to Gergely Nagy (algernon) about his tool Iocaine. Iocaine creates a maze to trap scraping bots in a world a fake pages they cannot escape. algernon tells us how Iocaine effectively traps bots by serving them endless loops of nonsensical URLs and web pages. It’s an extremely clever tool that’s designed to be completely hidden from normal users, but not hidden to the scrapers. Episode Links Gergely Iocaine algernon on Mastodon Nam-Shub of Enki Iocaine on mastodon Iocaine example site This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player. ...

Josh chats with Xe Iaso, the creator of Anubis the web AI firewall. We discuss how Anubis is tackling bots and scrapers. The discussion around the scrapers is fascinating and challenging, these things are everywhere and don’t behave very nicely. There’s also discussion about running a successful open source project. Xe has a lot of experience to share with us, you’re going to learn something new with this one. Episode Links Xe Anubis Blog Odd number of cores blog Common Crawl This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player. ...