Prevent crawlers from accessing deep forum pages

We have got a forum site that has a lot of threads/posts. Thus, if we go to the forum listing page, it has a "lot of" pages (In paging) of threads.

Now some crawlers access these deep forum pages, like 60000th page, 70000th page etc.

"GET /forum/latest-news/page-64099 HTTP/1.1"
"GET /forum/latest-news/page-65000 HTTP/1.1"

This creates a heavy load on the db because MySQL offset queries will be heavy to fetch this kind of content. And eventually, the site goes down in front-end for a...

Read more

Читать далее...