??? ?? ?????????? ??? ?????????? – ?????? ??????? ???????????Create robots.txt file to authenticate and cease crawler from crawling nearly anything besides your home site.?? ?????????? ??? ?? ???????? ????????? ????? ?????????? «?????????????», ???? ??? ????????????? ?? ??? ??????? ??????????? ?? ???????? ??? ??????? ????, ???? ?? ????????? ???