I am a Kagi user and have been for 7 months now. I signed up for the 300 searches per month plan because I felt like that would fit me well enough and it turns out I average 87 searches per month. That’s a lot lower than I thought it would be by quite a bit.

      • azerial@lemmy.dbzer0.comEnglish
        6·
        1 month ago

        The best part of that robots.txt is:

        Reddit believes in an open internet, but not the misuse of public content.

        Sure Jan.

        • l_b_i@yiffit.netEnglish
          2·
          1 month ago

          Reddit believes in an open pay to access internet, but not the misuse of public content our content we didn’t make.

        • l_b_i@yiffit.netEnglish
          3·
          1 month ago

          A few possibilities,

          1. brave has state in the past they use the googlebot user agent, if all reddit does is check the useragnet, it won’t block brave. This does however mean brave is violating the robots.txt file.
          2. I saw some mentions of google fall back, I don’t know if they still do it, but that could be another possibility.
          3. Brave ignores robots.txt files
          4. Brave paid for access

          No matter the reason, well behaving crawlers will no longer crawl reddit, Everything is disallowed in the robots.txt

          User-agent: *
Disallow: /