To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually  ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]

When you write a page title, you have a chance right at the beginning of the page to tell Google (and other search engines) if this is a spam site or a quality site – such as – have you repeated the keyword four times or only once? I think title tags, like everything else, should probably be as simple as possible, with the keyword once and perhaps a related term if possible.
One of the things Google looks at when ranking a page is the content on that page. It looks at the words on the page. Now picture this, if every word on, for instance, a blog post about a digital piano is used 2 times, then all words are of equal importance. Google won’t have a clue which of those words are important and which aren’t. The words you’re using are clues for Google; it tells Google and other search engines what the page or post is about. So if you want to make Google understand what your page is about, you need to use it fairly often.
Understanding the balance of terms that might be a little more difficult due to competition, versus those terms that are a little more realistic, will help you maintain a similar balance that the mix of long-tail and head terms allows. Remember, the goal is to end up with a list of keywords that provide some quick wins but also helps you make progress toward bigger, more challenging SEO goals.
I prefer simple SEO techniques and ones that can be measured in some way. I have never just wanted to rank for competitive terms; I have always wanted to understand at least some of the reasons why a page ranked for these key phrases. I try to create a good user experience for humans AND search engines. If you make high-quality text content relevant and suitable for both these audiences, you’ll more than likely find success in organic listings and you might not ever need to get into the technical side of things, like redirects and search engine friendly URLs.
I’m really grateful for your generous post, Brian. I’m definitely going to implement TOC on some of my over 4k words posts, where I’m trying to become the source. 😉 And I will also use the stats on some new posts. Thanks to you, I also researched big keywords, which I’d stayed away from, and found that many of the high CPC and ranking articles are from 2014. Hoping some of my fresh new content helps rank me higher. Love what you do, sir!

Either porn SC or Ads containing porn on non­Porn pages can be very distracting or even upsetting to users. Please refresh the page a few times to see the range of Ads that appear, and use your knowledge of the locale and cultural sensitivities to make your rating. For example, an ad for a model in a revealing bikini is probably acceptable on a site that sells bathing suits. However, an extremely graphic porn ad may warrant a Low (or even Lowest) rating.”

Google is falling into a familiar pattern. First, they offer web publishers increased visibility and SERP display options. Next, they incent participation in specific formats and data structures. Finally, they take that data for themselves, changing the SERPs to favor advertising, their own properties, and/or instant answers that can reduce publisher traffic. For web marketers, it's a prisoner's dilemma. In this presentation, Rand will show data on how Google is being used today, how it's changing, then dive into strategic initiatives and specific examples of how savvy players can build a moat to protect against long-term risk.

Hey Sharon, great post! Re. dwell time – I’ve read conflicting opinions, some saying that Google DOES consider it an ‘important’ ranking signal, and others saying that it doesn’t, because dwell time can sometimes be a misleading indicator of content quality. For example when a user searches for something specific and finds the answer immediately in the recommended page (meaning that the content on the page is actually spot on) so he returns to the SERPs very quickly. I have been unable to locate any definitive statements (written/spoken) from anyone at Google that suggest that dwell time IS still a factor in ranking considerations, but it makes sense (to me, anyway) that it should be. Do you have any ‘proof’ one way or the other re. whether Google definitely considers dwell time or not?

QUOTE: “Some pages load with content created by the webmaster, but have an error message or are missing MC. Pages may lack MC for various reasons. Sometimes, the page is “broken” and the content does not load properly or at all. Sometimes, the content is no longer available and the page displays an error message with this information. Many websites have a few “broken” or non-functioning pages. This is normal, and those individual non-functioning or broken pages on an otherwise maintained site should be rated Low quality. This is true even if other pages on the website are overall High or Highest quality.” Google
Redirecting is the act of sending a user to a different URL than the one initially requested. There are many good reasons to redirect from one URL to another, for example, when a website moves to a new address. However, some redirects are designed to deceive search engines and users. These are a very poor user experience, and users may feel tricked or confused. We will call these “sneaky redirects.” Sneaky redirects are deceptive and should be rated Lowest.
Being ‘relevant’ comes down to keywords & key phrases – in domain names, URLs, Title Elements, the number of times they are repeated in text on the page, text in image alt tags, rich markup and importantly in keyword links to the page in question. If you are relying on manipulating hidden elements on a page to do well in Google, you’ll probably trigger spam filters. If it is ‘hidden’ in on-page elements – beware relying on it too much to improve your rankings.

QUOTE: “Supplementary Content contributes to a good user experience on the page, but does not directly help the page achieve its purpose. SC is created by Webmasters and is an important part of the user experience. One common type of SC is navigation links which allow users to visit other parts of the website. Note that in some cases, content behind tabs may be considered part of the SC of the page.” Google Search Quality Evaluator Guidelines 2017

SEO techniques can be classified into two broad categories: techniques that search engine companies recommend as part of good design ("white hat"), and those techniques of which search engines do not approve ("black hat"). The search engines attempt to minimize the effect of the latter, among them spamdexing. Industry commentators have classified these methods, and the practitioners who employ them, as either white hat SEO, or black hat SEO.[50] White hats tend to produce results that last a long time, whereas black hats anticipate that their sites may eventually be banned either temporarily or permanently once the search engines discover what they are doing.[51]

Sometimes we refer to those root keywords as "vanity keywords," because if you do just one search to see who seems to be winning the space, you are likely to pick the single broadest keyword and see who comes up ranked highly. In nearly every case, however, we have found it to be more successful and deliver a significantly better return on your SEM investment by focusing on the hundreds or even thousands of more specific keywords that more closely match the services, products, brands, and locations that you sell or serve.
Wow! This was so helpful to me. I am new to the blogging world and have been feeling really frustrated and discouraged because I lacked the knowledge of getting my post to rank in search engines. I know there is a lot more I still need to learn but this has layed a foundation for me. I am bookmarking it so I can return and read it again. Thank you for writing!