The role of Robots.txt and method of use

he can’t climb to the spider doesn’t mean he can’t grab.

I need to grab the

love experience: jingyan.baidu贵族宝贝

we have to drop "the overall quality of our many irresistible factors.

2, shielding some of the duplicate pages.

This is

Disallow: /login_c.jsp

we only those with common features page shielding. The spider can’t climb to the page, it will display a 404 page, it will form a dead link.

shield shield refers to let search engine put out. Shanghai love force will be relatively slow, like an old man, he is already an old man, he reflected particularly slow, Google quickly. Google as long as you submit, he immediately grab 24 hours in force, you see the shield blocked pages directly in Google Adsense management tools. But love not love Shanghai, Shanghai reflects the very slow.

1, shielding some content to the page.

as we search for a word, he will also appear such content. There is no search results page [page], error pages, page 404 (if you do). You will find more and more things we want to shield. And these things we have to deal with.

Robots is a website and search engine of the third party agreement (early for the protection of privacy).

Explanation: ShanghaiEffect of

address, a source address provided on the line. For example, a page inside of me, I don’t care a wrong address, but this link does not exist, the spider can’t grab, display code > 404

page, DZ forum comments such as abbreviated version (ative), WordPress RSS, the repeat of the page we need for processing. Some of the duplicate pages is we have to do, this is no way to do.

registration page: /register_p.jsp

can grab to address and can crawl into 2 concepts.

3, shielding some dead links page.

message page…… [/ sign up page forum page content page] sign is

Disallow: /register_p.jsp

two, Robots.txt

, the origin of Robots.txt

login page: /login_c.jsp

regardless of our website forums, blogs, shopping mall, enterprises are good, now we need to deal with both of these pages and operation, we can only shield, can not be deleted. Our aim is to be able to achieve a function for users to search, we understand better.

Published by admin

Leave a Reply

Your email address will not be published. Required fields are marked *