According to docs there are multiple IPs of crawler:
199.59.148.209
199.59.148.210
199.59.148.211
199.16.156.124
199.16.156.125
199.16.156.126
first 3 of these have reverse DNS records:
In [18]: socket.gethostbyaddr('199.59.148.209')
Out[18]: ('r-199-59-148-209.twttr.com', [], ['199.59.148.209'])
In [19]: socket.gethostbyaddr('199.59.148.210')
Out[19]: ('r-199-59-148-210.twttr.com', [], ['199.59.148.210'])
In [20]: socket.gethostbyaddr('199.59.148.211')
Out[20]: ('r-199-59-148-211.twttr.com', [], ['199.59.148.211'])
But forward lookups for these domain return different IP address:
In [21]: socket.gethostbyname('r-199-59-148-209.twttr.com')
Out[21]: '23.21.224.150'
In [22]: socket.gethostbyname('r-199-59-148-210.twttr.com')
Out[22]: '23.21.224.150'
In [23]: socket.gethostbyname('r-199-59-148-211.twttr.com')
Out[23]: '23.21.224.150'
It’s not critical because all IP addresses of crawler are listed in docs but it’s common practice to validate bots by forward and reverse DNS lookups:
In [25]: socket.gethostbyaddr('157.55.33.18')
Out[25]: ('msnbot-157-55-33-18.search.msn.com', [], ['157.55.33.18'])
In [26]: socket.gethostbyname('msnbot-157-55-33-18.search.msn.com')
Out[26]: '157.55.33.18'
In [27]: socket.gethostbyaddr('66.249.66.1')
Out[27]: ('crawl-66-249-66-1.googlebot.com', [], ['66.249.66.1'])
In [28]: socket.gethostbyname('crawl-66-249-66-1.googlebot.com')
Out[28]: '66.249.66.1'
Is it possible to fix forward and reverse lookups of twitterbot crawler?
