Hot Potato@lemmy.world to Technology@lemmy.worldEnglish · 3 months agoGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coexternal-linkmessage-square248fedilinkarrow-up1880arrow-down110file-textcross-posted to: planetdyne@fed.dyne.orgtechnology@lemmy.worldtechnology@beehaw.orghackernews@lemmy.smeargle.fans
arrow-up1870arrow-down1external-linkGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coHot Potato@lemmy.world to Technology@lemmy.worldEnglish · 3 months agomessage-square248fedilinkfile-textcross-posted to: planetdyne@fed.dyne.orgtechnology@lemmy.worldtechnology@beehaw.orghackernews@lemmy.smeargle.fans
minus-squareEril@feddit.orglinkfedilinkEnglisharrow-up13·3 months agoActually currently it contains this: User-agent: * Disallow: / Well, that actually is a blanket ban for everyone, so something else must be at play here.
minus-squarestarman@programming.devlinkfedilinkEnglisharrow-up21·3 months agohttps://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy Reddit is serving different file to google
minus-squareRuss@bitforged.spacelinkfedilinkEnglisharrow-up17·3 months ago We believe in the open internet, but we do not believe in the misuse of public content. That’s real rich, coming from Reddit.
Actually currently it contains this:
User-agent: * Disallow: /Well, that actually is a blanket ban for everyone, so something else must be at play here.
https://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy
Reddit is serving different file to google
That’s real rich, coming from Reddit.