#1

Hi Robots.txt of twitter has something rules.
I want to follow their rules. But I have a problem to understand some rules.

Every bot that might possibly read and respect this file.

User-agent: *
Allow: /?lang=
Allow: /hashtag/?src=
Allow: /search?q=%23
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /?
Disallow: //followers
Disallow: /*/following

Disallow: /account/not_my_account
Disallow: /account/deactivated
Disallow: /settings/deactivated

Disallow: /oauth
Disallow: /1/oauth

Disallow: /i/streams
Disallow: /i/hello

These are the rules of Every bot, and if I use a url like ‘timeline?lang=en&l=&f=tweets&q=user&src=typed&max_position=’, then is it agree with robots.txt?
That url get parameter starts with ‘lang’.

Please help me about this topic.
Thank you

#2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.