How to Create a Robots.txt file tutorialDefinition of a Robots.txt file:A robot index websites, check link popularity on websites, and judge websites whether it contains relevant content with targeted keywords or not. A robot is also called a Crawler or a Spider, they have the same meaning. A robot is a computer program operated by a search engine, a research organization, a University or an individual. Everytime a search engines robot look into your root domain a file named "robots.txt" usually at the address: 'http://www.mydomain.com/robots.txt'. This robots.txt file tell robots what files or folders they have to index on your website. We call that system the "Robots Exclusion Standard". Here is a list of common mistakes used in a robots.txt file. The robots.txt file looks like that:# mydomain.com robots.txt: The command:User-agent is the name of the robots, spiders, crawlers that visit your website.User-agent: googlebot The rules only apply to googlebot spider the disallow command keep out robots from certain files or folders.Disallow: /content/ Robots cannot index the folder content There is no Allow command as you understood it! If it was so it means you have to write down all the folders and files you want the robots to index. The comments tagYou can also make comments in a robots.txt file like the following examples: User-agent: * Here is another robots.txt example:# go away This example is the best, all robots are disallowed to crawl the website. Let's make some more examples and explain about itUser-agent: inktomislurp + Here the targeted robot is inktomislurp and is not allowed to index the folder cgi-bin and forum and also not allowed to index the file taxes.asp from the folder data. This rules only apply to the robot Inktomislurp. User-agent:* + All robots are disallowed to enter the folder 4gettobuybread and common. And are disallowed to access the files 404.jsp, tips.jsp, index.jsp, site_index.jsp. User-agent:googlebot + It restricts googlebot from indexing all PDF, JEPG and EXE files. Next page: Robots.txt common mistakes |
|
MenuHomeNews Blog Free phpLD Templates Free Web Tools Site Map Add Your Site Resources Spammers List Seo Guide Seo Glossary Web DirectoriesIndexDMOZ Yahoo! Looksmart Best Paid Web Directories Best Web Directories Search EnginesIndexAOL Ask Jeeves Excite Hotbot Altavista Lycos Teoma SEO TutorialsGoogle OptimizationYahoo! Optimization Meta Tags Optimization Keywords Optimization Link Popularity Tutorials Domain Name TLDs Create A Robots.txt file |
Network: Home | Web Tools | Link to Us | Seo Blog | Site Map | Resources | Web Design Services |
Irkawebpromotions 2005 - 2011 © - All Rights Reserved. |