Robots.txt Generator

link directory

main menu

Free Templates

Free Template Stats
Total Templates: 336
Total Categories: 15
Total Downloads: 9275

The Working Girl 2.0 Wordpress
DAJ_Glass phpBB
Ellipse Joomla
Modern World
JS Natura Joomla
Night-star WordPress
SunsetIdea 1.0 WordPress
Game zone template
Wireframe Drupal
Blue Mountain 1.0 WordPress

free resources

Total Links: 1039
Total Categories: 40

1: UploadYourImages
2: FlamingText
3: ZippyImages
4: Blue Sfear
5: TacoNet

Site Reviews

"Site looks great, I have always been able to find the resources to my questions on this site for up to date information, with the latest in Modules, blocks, and themes, I..."

Mugs

Robots.txt Generator Tool

Robots.txt files are used by web owners and webmasters to signify to spider-bots and search engines not to crawl or index certain files and directories. The Robots.txt file is a command file that most creditable search-bots will obey and check for on entry to your website URL. To create a robots.txt file, use the following form and select the fields to either disallow certain bots, disallow directories or deny access from bad-bots. If you have any questions please use our support forums.

Support future development

If you enjoy our services and online resources consider making a donation to support our efforts.

Menu

| Creating A Robots.txt File | List of Bad-Bots |

Default Robots.txt file setting

Default - All Robots are:

Crawl-Delay:

Specific Search Robots

Search Engine	Choose Setting	Bot-Name
Google		googlebot
MSN Search		msnbot
Yahoo		yahoo-slurp
Alexa/Wayback		ia_archiver
Ask/Teoma		teoma
DMOZ		robozilla
GigaBlast		gigabot
Scrub The Web		scrubby

Specific Special Bots:

Search Engine	Choose Setting	Bot-Name
Google Image		googlebot-image
Yahoo MM		yahoo-mmcrawler
MSN Pic-Search		psbot
Yahoo Blogs		yahoo-blogs/v3.9

Extra Exclusion features:

Search Engine	Choose Setting	Bot-Name
Google		googlebot
MSN		msnbot
Yahoo		yahoo-slurp

Disallow Bad Bots:
Select this option to block the most common "Bad-Bots," also known for spam and email harvesting from accessing your domain.
Disallow Bad-Bots:	User-agent: Black Hole User-agent: Titan User-agent: WebStripper User-agent: NetMechanic User-agent: CherryPicker User-agent: EmailCollector User-agent: EmailSiphon User-agent: WebBandit User-agent: EmailWolf User-agent: ExtractorPro User-agent: CopyRightCheck User-agent: Crescent User-agent: Wget User-agent: SiteSnagger User-agent: ProWebWalker User-agent: CheeseBot User-agent: Teleport User-agent: TeleportPro User-agent: MIIxpc User-agent: Telesoft User-agent: Website Quester User-agent: WebZip User-agent: moget/2.1 User-agent: WebZip/4.0 User-agent: WebSauger User-agent: WebCopier User-agent: NetAnts User-agent: Mister PiX User-agent: WebAuto User-agent: TheNomad User-agent: WWW-Collector-E User-agent: RMA User-agent: libWeb/clsHTTP User-agent: asterias User-agent: httplib User-agent: turingos User-agent: spanner User-agent: InfoNaviRobot User-agent: Harvest/1.5 User-agent: Bullseye/1.0 User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 User-agent: CherryPickerSE/1.0 User-agent: CherryPicker /1.0 User-agent: WebBandit/3.50 User-agent: NICErsPRO User-agent: Microsoft URL Control - 5.01.4511 User-agent: DittoSpyder User-agent: Foobot User-agent: WebmasterWorldForumBot User-agent: SpankBot User-agent: BotALot User-agent: lwp-trivial/1.34 User-agent: lwp-trivial User-agent: Wget/1.6 User-agent: BunnySlippers User-agent: Microsoft URL Control - 6.00.8169 User-agent: URLy Warning User-agent: Wget/1.5.3 User-agent: LinkWalker User-agent: cosmos User-agent: moget User-agent: hloader User-agent: humanlinks User-agent: LinkextractorPro User-agent: Offline Explorer User-agent: Mata Hari User-agent: LexiBot User-agent: Web Image Collector User-agent: The Intraformant User-agent: True_Robot/1.0 User-agent: True_Robot User-agent: BlowFish/1.0 User-agent: JennyBot User-agent: MIIxpc/4.2 User-agent: BuiltBotTough User-agent: ProPowerBot/2.14 User-agent: BackDoorBot/1.0 User-agent: toCrawl/UrlDispatcher User-agent: WebEnhancer User-agent: TightTwatBot User-agent: suzuran User-agent: VCI WebViewer VCI WebViewer Win32 User-agent: VCI User-agent: Szukacz/1.4 User-agent: QueryN Metasearch User-agent: Openfind data gathere User-agent: Openfind User-agent: Xenu's Link Sleuth 1.1c User-agent: Xenu's User-agent: Zeus User-agent: RepoMonkey Bait & Tackle/v1.01 User-agent: RepoMonkey User-agent: Zeus 32297 Webster Pro V2.9 Win32 User-agent: Webster Pro User-agent: EroCrawler User-agent: LinkScan/8.1a Unix User-agent: Keyword Density/0.9 User-agent: Kenjin Spider User-agent: Cegbfeieh

Restricted Directories:

The path is relative to root and must contain a trailing "/"

Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"

Create The Robots.txt

Click here to download the robots.txt file!

Now, copy and paste this text above into a blank text file called "robots.txt" and put it in the root directory of your website. Make sure it's permissions are set so that visitors and search engines can read it.

Example: http://www.yoursite.com/robots.txt

Get Your Ad Here

Click Here to Advertise

Affordable Hosting! http://www.invision-graphics.com/images/banners/468X60_VISIONHOSTING.gif

Today: 10,103 Yesterday: 27,683 Total Hits: 41,029,507
Page Rendered in: 0.14s - Total Queries: 26 - MySQL DB: 30.8 mb's - Pages served in past 5 minutes: 198