Do you know the importance of a Robots.txt file? Read to know.

Success of big companies lies in keeping their confidential data a secret, hidden from all. This enables them to execute their future course of action easily and change plans according to the situation. Job of robots.txt file is the same. It can or cannot allow a search engine to visit some or all of your web pages. Of course a human visitor is free to visit these pages. That being the case, for the search engines your website may be different than what a visitor is seeing. If you think one or some of the pages aren’t good enough to be visited by search engines you can do it.

Every search engine has a “robot” (a software program) that does the job of visiting a website. Their purpose is to gather a copy of the site and keep them in their database. So, if your site is not there in their database it never shows up in the search results.

Web Robots are sometimes referred to as Web Crawlers, or Spiders. Therefore the process of a robot visiting your website is called “Spidering” or “Crawling”. When somebody says “the search engines have spidered my website”, it means the search engine robots have visited their website. This robot is known by a name and has an independent IP address. This IP address is of no importance to us, but knowing their names will help since this name will be used when we create a robots.txt file. This is why the file is called “robots.txt.”

Given below is the list of the robots of some of the very popular search engines:

Search Engine	Robot
Alexa.com	ia_archiver
Altavista.com	Scooter
Alltheweb.com	FAST-WebCrawler
Excite.com	ArchitextSpider
Euroseek.net	Arachnoidea
Google.com	Googlebot (http://www.google.com/bot.html)
Hotbot.com (uses Inktomi’s robot)	Slurp
Inktomi.com	Slurp
Infoseek.com	UltraSeek
Looksmart.com	MantraAgent
Lycos.com	Lycos_Spider_(T-Rex)
Nationaldirectory.com	NationalDirectory-SuperSpider
UKSearcher.co.uk	UK Searcher Spider

Writing Robots.txt:

Let’s learn to write robots command. Note that there are two ways to write robots command. One is to include all the commands in a text file called “robots.txt” and another is to write robots command in the meta tag.

We will learn both ways of writing robots command.

Writing robots command in Meta tag:

There are 4 things you can tell a search engine robot when it visits your page:

1) Do not index this page – the search engines will not index the page.

2) Do not follow any links on this page – the search engines will not follow the links included in the page, i.e. they will not index any page that this page links to.

3) Do index this page – the search engines will index the page.

4) Do follow the links – the search engines will index the pages that this page links to.

Note that “index” is different than “spider”. A search engine first spiders a page and then indexes it. Indexing is giving a certain importance to the page on the basis of its content, information, meta tags, link popularity with respect to the searched keyword. All this is decided at run time. When you tell search engines not to index a page, it means they know that “certain” page exists but do not rank them. That is, a no-index page will never be shown in their search results. This in any case does not mean a no-index page will not get visitors, it might get visitors indirectly from a page which links to it. Yes, no direct visitors from the search engines.

Suppose you want the search engines to index and also index (follow) its linked pages then include the following command in the Meta Tag:

<meta name=”robots” content=”index, follow”>

Suppose you want the search engines to index a page but not follow its links then include the following command in the Meta Tag:

<meta name=”robots” content=”index, nofollow”>

Suppose you do not want the search engines to index a page but follow its links then include the following command in the Meta Tag:

<meta name=”robots” content=”noindex, follow”>

Suppose you do not want the search engines to either index or follow links of a particular page then include the following command in the Meta Tag:

<meta name=”robots” content=”noindex, nofollow”>

Note:

Google makes a “Cached” of every file it spiders. It’s a small snap shot of the page. Want to stop Google from doing so? Include the following Meta Tag:

<meta name=”robots” content=”noindex, nofollow, noarchive”>

Like any meta tag the above written tags should be placed in the HEAD section of an HTML page:

<html>
<head>
<title>your title</title>
<meta name=”description” content=”your description.”>
<meta name=”keywords” content=”your keywords”>
<meta name=”robots” content=”index, follow”>
</head>
<body>

Creating robots.txt file:

A robots.txt file is an independent file and should be written in a plain text editor like Notepad. Do not use MS-Word or any other text editor to create robots.txt. The bottom line is this file should have the extension “.txt” else it will be useless.

Let’s begin. Open Notepad (it comes free with Microsoft Windows) and save the file with the name “robots.txt”. Make sure that the extension is .txt.

By the way, did you note we did not use name of any robot in the meta tag! What does it indicate? Simple – by using meta you direct all the search engines to do something or not do something on a page. You do not have control over any one search engine. The solution is robots.txt.

It can always happen you do not want a particular search engine to index a page for certain reasons. In that case using a robots.txt file will help. Even though I do not recommend such a thing. The search engines get you traffic, why hate them. Stop them from doing their job and they hate you. I again repeat keep your pages smart for the search engines and welcome them. Fine, then why take the trouble to learn robots.txt? Why should you include a robots.txt file at all?

Let’s suppose yours is a dynamic database site containing information of your newsletter subscribers, customers, their address, phone numbers etc. All these confidential information is kept in a separate directory called “admin”. (It is recommended to keep such information in a separate directory. Handling data will be easier for you and so will be easy to keep the search engines away. We will just know how.) I am sure you would never want any unauthorized person to visit this area leave alone the search engines. It does not help the search engines either since they have nothing to do with the data or files there. Here comes the role of a robots.txt file.

Write the following in the robots.txt file:

---

User-agent: *
Disallow: /admin/

---

This does not allow the spiders to index anything in the admin directory also including sub-directories if any.

The asterisk (*) mark indicates all the search engines. How do you stop a particular search engine from spidering your files or directory?

Suppose you want to stop Excite from spidering this directory:

---

User-agent: ArchitextSpider
Disallow: /admin/

---

Suppose you want to stop Excite and Google from spidering this directory:

---

User-agent: ArchitextSpider
Disallow: /admin/

User-agent: Googlebot
Disallow: /admin/

---

Files are no different. Suppose you want a file datafile.html not to be spidered by Excite:

---

User-Agent: ArchitextSpider
Disallow: /datafile.html

---

Similarly, you do not want it to be spidered by Google too:

---

User-agent: ArchitextSpider
Disallow: /datafile.html

User-agent: Googlebot
Disallow: /datafile.html

---

Suppose you want two files datafile1.html and datafile2.html not to be spidered by Excite:

---

User-Agent: ArchitextSpider
Disallow: /datafile1.html
Disallow: /datafile2.html

---

Can you guess what does the following mean?

---

User-agent: ArchitextSpider
Disallow: /datafile1.html
Disallow: /datafile2.html

User-agent: Googlebot
Disallow: /datafile1.html

---

Excite will not spider datafile1.html and datafile2.html, but Google will not spider only datafile1.html. It will spider datafile2.html and the rest of the files in the directory.

Imagine you have a file kept in a sub-directory that you wouldn’t like to be spidered. What do you do? Lets suppose the sub-directory is “official” and the file is “confidential.html”.

---

User-agent: *
Disallow: /official/confidential.html

---

If the syntax of your robots.txt file is not written correctly, the search engines will ignore that particular command. Before uploading the robots.txt file double check for any possible errors. You should upload robots.txt file in the ROOT Directory of your server. The search engines look for robots.txt file only in the root directory.

Note:

You should be able to see robots.txt file if you type the following in the address bar of your Internet browser.

http://www.your-domain.com/robots.txt

Here is Google’s Robots.txt file:

http://www.google.com/robots.txt

All search engines follow robots.txt command.

You can look in your web server log files to see what search engine robots have visited. They all leave signatures that can be detected. These signatures are nothing but name of their robots. For instance if Google has spidered your site it will leave a log file called Googlebot. This is how you know which search engine has spidered your pages and when!

We are highly experienced in SEO/SEM/Pay Per Click Management. Contact us regarding any query you may have.

Meta Tags (Keyword and Description) and the Title Tag are an absolute essential for any page to get top ranks in the search engines. In fact this is a very important criterion the search engines take into consideration before deciding to give rank to a page with respect to a keyword.

We discuss them one by one.

The Meta Keyword Tag:

The Meta Keyword tag is written like this:

<META NAME=”Keywords” CONTENT=”keyword1, keyword2, …., keywordX”>

Where keyword1, keyword2, …., keywordX are important keyword phrase with respect to the site.

Note: The Meta Keyword Tag is not displayed to the viewer while browsing. It’s there for certain technical reason. An explanation is beyond the scope of this article, but know that it does help in search engine ranking – therefore we are discussing it here!

Some of the search engines ignore this tag, however it doesn’t take much on your part to include it. Search engines that give importance to the Meta Keyword Tags look for it.

Punching this tag with a lot of Keywords reduces the weight of this Tag. The search engines might even ignore them. Also, repetition of the same keyword more than twice is unnecessary. The best idea would be to write the most important keyword in upper as well as lower case. For example, “search engine optimization, search engine, SEARCH ENGINE OPTIMIZATION”.

Suppose your site deals in online booking of hotel rooms in the US. You have one page dedicated to hotels in Florida. You can write

<META NAME=”Keywords” CONTENT=”hotels in florida, US Hotels, Florida accommodation, US accommodations, HOTELS IN FLORIDA”>

Note that none of the keywords is repeated more than twice and somehow we achieve the target of pushing the most important keywords. There is no need to stuff this tag with a lot of keywords – even the search engines that do recognize the Meta Keyword Tag will ignore them.

You can repeat some very popular and important keywords in different pages of your website. It does make your site stronger for that keyword. Keep in mind not to include this keyword in any page that has absolutely nothing to do with the meaning of the keyword. The index page is the best example. Your index page is the most important page for the search engines. Since the index page contains the general information of your business/site you can include the most popular keywords here. It’s recommended you include the top 10 keywords in the index page. Repeat these keywords in the respective page meant to target that portion of your business. For example the index page could contain the keyword “Hotels in Florida” and a page that is dedicated to online booking/information in hotels in Florida could also contain this keyword.

The Bottom Line – even if not-so-important tag, do include the Keyword Meta Tags in all of your web pages. No need to stuff this tag with a lot of keywords.

We can write Meta Tags for your website for upto 10 pages for $10 only. This is what we will do:

Research your site and come up with highly efficient 10 keywords. These keywords will get you good traffic in the long run.

Write Meta Tags (Description and Keywords tag) and Title Tag for these keywords that you can implement in 10 pages of your website.

For 10 Dollars this service is by far cheaper then any Meta Tag writing service online. Please pay by clicking the button below and do not forget to mention the URL of your website. We will send you the list of 10 keywords and ready Meta Tags to be used in your pages within 1 business day.

The Meta Description Tag:

The Meta Description Tag is written like this:<META NAME=”Description” CONTENT=”Describe your site in a single line here.”>Where in the CONTENT part you write the description of your website in general and the page in particular. Note that your description should not exceed 20 words. Try to make it less than 20 words. Actually you can write a long description but the search engines are too busy to check what’s written approximately after that many words, so anything after 20 words is ignored.

Note: The Meta Description Tag too is not displayed to the viewer, but it does help tremendously in search engine ranking. This tag is displayed by most of the search engines in their search results. The surfers read this sentence before they click on to visit your site. Imagine the importance of this tag – a smart description can force a surfer to click on it and a weak one may drive your traffic away!

These 20 odd words carry some serious weight as far as search engine placement is concerned. Try to target the most important keywords within the first 15 words. Your description should be a grammatically correct sentence with no spell errors. Start your description with an upper case letter (just as you would do to write a sentence) and end it with a period. The more your target keyword is near to the beginning of the sentence, the better.

An example:
Lets take the same travel agent site featuring online reservations in the hotels in USA. The page that features accommodation in Florida could include the following description:

<META NAME=”Description” CONTENT=”Florida accommodation – online reservation available with huge discount.”>

The first two words “Florida accommodation” is our important keyword. The description begins with a popular keyword. Online reservation is a general keyword and helps even if somebody types “Florida accommodation online” or “online Florida accommodation”. Note that there is no other keyword ‘diluting’ the importance of the keyword – Florida accommodation. Since this page we assume is dedicated to Florida accommodation there is no need to push any other keyword. Now, read the sentence. Any surfer interested in visiting Florida would click on it.

The index page can contain all the important keywords in its description, but as said earlier do not exceed 20 words.

The Bottom Line – no more than 20 words in a description; grammatically correct sentence; no spelling mistakes; try to start with an important keyword; only the first letter in upper case, of course nouns should begin with upper case.

We can write Meta Tags for your website for upto 10 pages for $10 only. This is what we will do:

Research your site and come up with highly efficient 10 keywords. These keywords will get you good traffic in the long run.

Write Meta Tags (Description and Keywords tag) and Title Tag for these keywords that you can implement in 10 pages of your website.

For 10 Dollars this service is by far cheaper then any Meta Tag writing service online. Please pay by clicking the button below and do not forget to mention the URL of your website. We will send you the list of 10 keywords and ready Meta Tags to be used in your pages within 1 business day.

ALT Codes:

There are still many people in the world who use very old browser. These were the first generation browsers. Very peculiar browses, they support only text and/or the very first version of HTML (Hyper Text Markup Language). They don’t understand frames neither have any capability to download images and present them to their viewers. The ALT code was probably included for these reasons in the modern browsers. ALT code is a text that is shown in place of an image if the browser cannot display images. Also, there are some people who disable the image-displaying property of their browsers so that the websites download faster. Since they cannot view the images, they see the ALT codes. (If you are using a browser which supports images you can still view the ALT code. To view take your mouse pointer over the image – the ALT text will appear and then disappear after a couple of seconds.)

The search engines had no way out but to support these ALT codes since browsers old or new support it. Note that the search engines DO NOT care about images – they are neither text nor something they can understand but they do care for the text in the ALT codes. So why not to use the ALT code optimally, especially when they can help us to achieve good search engine ranking.

You should use at least 2-3 popular keywords in the ALT codes. The text you use should have some relevance to the image. Just as an example, if the image is a logo of your company the ALT code could be the official name of the company.

Here’s is how to write ALT code:

<IMG SRC=”path of the image to be displayed” WIDTH=”width of the image in percentage or pixels” HEIGHT=”height of the image in percentage or pixels” ALT=”popular keyword 1, popular keyword 2, popular keyword 3″>

Where popular keyword 1, popular keyword 2, popular keyword 3 are your important alt code keywords that will help you to get top rank in the search engines. It is recommended that you define height and width of the images. This will speed up the process of displaying the page. If height and width is defined the browsers understand the space to be provided to the image and therefore go ahead to display the text on the page, else they wait for the image to be downloaded fully, display them and then display the text.

If your site has 15 images and you included 2-3 unique keyword/phrase in each image you end up including 30-45 important words in your site with no extra effort. This will surely boost your ranking. (Please note that there are certain browsers that do not support ALT codes, in that case you may not be able to see the ALT codes. For the best results it is advised that you view this site in a browser that supports ALT code.)

COMMENT Tag:

The COMMENT tag as the name suggests is used for comments in between HTML tags or text in a Web Page. Browsers do not display text written in the comment tag. It was probably made with the point to help the webmasters to know why a particular tag or command had been written or what their functions are. This being the case, most of the search engines too ignore the text written in the comment tag, but a few don’t.

The COMMENT Tag is written like this:

<!– This is a comment –>

The text you are seeing in between <!– and –> is not displayed by the browser.

Include a few important keywords in the comment tag. A big page may contain 4-5 comment tags with important keywords while a smaller one may contain 2-3. Construct a few sentences using your important keywords. Do not insert keywords one after another separated with commas. Convention is that this tag contains sentences to help webmasters. You are advised not to repeat the same sentences in any two comments present in your site although you can repeat the keywords. Try not to exceed 45 words in a comment tag so that it looks like a comment. You can make as many sentences as you want as long as you remember your limit. Not to mention, the sentences you construct in the comment tag should make some sense with what is there in the content of the web page.

This article deals with whether the JavaScript Codes disrupts SEO Optimization of your site or not.

You might find a lot of articles related to using JavaScript Code or not for better SEO Optimization. Some might advise not to use it at all, while some advise to use it within certain restrictions. For example, “don’t use long JavaScript codes”, “use them at the bottom of your page” etc.

First of all make it very clear – do not fear putting the JavaScript codes if they are really required. In no way it’s going to disrupt you in SEO Optimization. The point is WHERE to keep the codes.

This has been a topic of debate since a long time now as far as SEO optimization is concerned. Do the codes really mater? Does the placement of codes really matter?

Lets discuss where we usually keep the JavaScript codes. Most of the webmasters prefer inserting the JavaScript codes at the beginning of the HTML file, between the <HEAD> and </HEAD> section of the file.

It goes like this:

<html>
<head>
<title>The site’s title</title>
<meta name=”description” content=”The site’s description.”>
<meta name=”keywords” content=”The important keywords”>
<SCRIPT LANGUAGE=”JavaScript”>
<!– Begin
function f1()
{
// The codes
}
function f2()
{
// The codes
}
// End –>
</SCRIPT>
</head>
<body>
<p>The Body of the Page!</p>
</body>
</html>

Irrespective of the length of the code it’s always preferred there. The question is WHY? Is it really required to be kept there as a software rule, or the codes can be kept anywhere in the file?

The JavaScript codes can be kept ANYWHERE in the file. They will work as fine. So why do the webmasters prefer keeping it there? It’s probably to maintain the clarity of the page. Moreover if the page gets too long and the codes are scattered throughout, it becomes difficult to find out the exact JavaScript code that has to be debugged or that has to be changed. This is a kind of convention too that is being followed by most of the webmasters. So the placement of the JavaScript codes is just a customary practice rather than a rule and it has nothing to do with SEO optimization.

The search engines have a mammoth task of spidering millions of ever growing web pages. Google has already reached the 3 billion mark. They are really busy and do not want to be loaded with any more work.

Imagine a web page that has just 2 lines of text or information but 120 lines of JavaScript Code. For getting those 2 lines of information the search engines have to pass through all the unnecessary 120 lines of code. Keeping in mind what is written above do you think the search engines will like the page? They are very busy and if for 2 lines of information they have to work so hard then it’s unlikely that this page will ever get a good ranking. How much ever hard you try to optimize it.

Many SEO gurus have come out with an idea to tackle this problem to optimize such a page. According to them putting the JavaScript code at the bottom of the page will solve all the problems since the search engines will not have to go farther down to search the 2 lines of information.

This is what they say:

<html>
<head>
<title>The site’s title</title>
<meta name=”description” content=”The site’s description.”>
<meta name=”keywords” content=”The important keywords”>
</head>
<body>
<p>The Body of the Page!</p>
<SCRIPT LANGUAGE=”JavaScript”>
<!– Begin
function f1()
{
// The codes
}
function f2()
{
// The codes
}
// End –>
</SCRIPT>
</body>
</html>

Note that in this situation the body of the page is above the JavaScript Code. According to the SEO gurus the search engines will now get the content of the page before the code and therefore the importance of the page increases. Right? WRONG!

What if there were 2 more lines of content right at the bottom of the page just after the </SCRIPT> tag? The search engines will have to come to the bottom of the page to get those 2 lines of information. Even if you do not have any information at the bottom of the page the search engines do spider the whole page. They don’t have any idea whatsoever whether you have included some content at the bottom of your page or not. The search engines look for </HTML> tag or end of page whichever comes earlier to end their crawling. So they crawl your whole page anyway irrespective of where you have inserted the JavaScript code and will obviously rank your page on the basis of the content found. This again brings us to square one. Where should you include the JavaScript code if at all?

Those who know this scripting language know very well that a different file can be created to keep all the JavaScript codes separately. This file has an extension of .js. It is a simple file that can be created using notepad. It has nothing except the JavaScript code. NO HTML tags are included. You can place functions, code and style sheets into this external Js file and refer to the file with one line from each web page that wants to use the functions contained in it.

This .js file is kept in the same directory where the file in respect of whom the codes were written resides. Whenever the required codes are called they are taken from the .js file and executed. This file is called only once and then taken from the cache memory of your computer for further execution. It is retrieved only when the cache memory is expired.

This is how an external JavaScript file is included in the page:

<html>
<head>
<title>The site’s title</title>
<meta name=”description” content=”The site’s description.”>
<meta name=”keywords” content=”The important keywords”>
<SCRIPT LANGUAGE=”JavaScript” SRC=”externalfile.js”>
</SCRIPT>
</head>
<body>
<p>The Body of the Page!</p>
</body>
</html>

This process has a few advantages. External JavaScript files saves bandwidth, which can be achieved by caching any JavaScripts and Cascading Style Sheets (CSS) that are repeated often in a web site. Secondly the length of the page reduces giving way to faster download.

You can always instruct a search engine to index a page or not using a robots.txt file. Once the .js file is ready and uploaded to the server you can always tell the search engines NOT to index your external JavaScript file allowing them to save their time. If you don’t know how to write robots.txt file, read article on creating robots.txt file.

This technique solves both the purposes – the search engines do not have to look for the information, they get it without wasting a second and your purpose is solved too – you give search engines the important information fast!

However you need not bother if your page is full of Unique and Valuable information, or if you are providing Quality services. The JavaScript code is a trivial matter if your site is content rich. Secondly, the search engines actually do not spider the JavaScript codes. As soon as they find the <SCRIPT> tag they stop spidering. They resume their job again once they get the </SCRIPT> tag. But they have to go through the codes. So, by including the JavaScript code you are only taking a little bit of their time. Yes, too much of JavaScript code with very less content may irritate them. This is the only reason why a page with a lot of code with comparatively less content does not get a good Google PR (page rank). Unfortunately it’s the JavaScript code that has to take all the blame.

Truly speaking there is an unnecessary confusion being created with the use of JavaScript code vis-à-vis SEO optimization. Give some useful content and great service, your JavaScript codes will then play virtually no role in SEO optimization.

A Very Important Note: Google Site Search isn’t able to index content contained in JavaScript. To make sure it does, is to ensure that all of the text that needs to be indexed is visible in a text-based browser. That is in a lay man’s language – JavaScript is meant for scripting only, if you add text in JavaScript thinking you would get top ranks in organic results you will not get that.