tell scanning search [TXT]
www.state.gov/robots.txt
Words:
disallow
navitest
robots
older
archive
tell scanning search robots not to index the older archive pages user agent disallow www disallow waterfall disallow menu disallow nav
1
