Understanding Keyword Research and Analysis for SEO.

Search engine optimization (SEO) is the process of structuring sites and content in a way that improves the website's visibility and placement on search engines when users search for certain keywords. A proper keyword research and analysis is essential for SEO and can boost the qualified and converting traffic of a website. When done thoroughly, it can even help in the formation of a brand's content strategy or simply in the creation of a granular structure in an adwords account or website taxonomy

Understanding Search Demand 

Keyword research & analysis is the process through which you identify the relevancy and the popularity of search queries that you need to target through the use of keywords throughout your website's content.
Keyword research usually starts within various tools such as Adwords Keyword Planner, Wordtracker, Keyword Spy and Ubersuggest. It is important to keep in mind that the human search patterns can be illustrated by the search demand curve, an adaptation of a long tail probability distribution in statistics.

The long tail concept appeared adapted for the internet world in 2004 in an article from 'Wired' when the entertainment industries were struggling to devise new business models that would help them to survive the combat against online piracy. According to the article, ''the future of entertainment online lies in the million of niche markets at the shallow end of the bitstream", meaning that is not only about blockbusters and mega hits selling millions, but instead about millions of non-mainstream pieces that collectively can sell more than the few mega hits. The same principle is valid to all content online.

Now back to keyword research, according to the long tail search demand curve, keywords are divided to short tail, mid and long tail depending on their popularity in search queries. 

  • Short tail queries usually include 1-3 keywords that drive millions of monthly searches (e.g. Insurance, credit cards etc.) and are almost impossible to rank for due to hard competition in search results. Typical examples ranked in the top positions for fat head keywords, are websites with high Page Rank and domain authority

Short tail queries competition example
Top results for "insurance" in Google UK

  • Mid tail queries, were once comprising of two-three words, but currently include an average of four word terms (e.g cheap online car insurance) in some markets. This category can be suitable for new websites but requires scanning for each keyword in order to identify competitive intensity of search results. 
  • Long tail queries are usually 4+ word phrases that account for the 70% of the total search volume. (e.g: google plus statistics in 2013, cheap online car insurance in Alburqueque, etc.) These keyphrases are far more relevant to the searcher intent than the board short tail queries and are ideal for new websites that can narrow their content in order to reflect the "niche" markets mentioned above.

keyword research and the long tail curve
source: moz.com

According to a Hubspot survey, the top 10k keywords of the demand curve make up less than 20% of the overall search traffic, while 80% comes from long tail keywords.

Keywords, Rankings & CTRs

When starting your keyword research you need to create an initial keyword list, also called a seed list, with relevant keywords to your website's context. At this point you can start brainstorming words about how searchers might look for your content/product/service online.  

Next you need to use a keyword tool (some mentioned above) that will help you identify the entire search curve by defining the short, mid and long tail keywords of your niche. By extracting, comparing and sorting keyword search volumes, you then attach values to the selected keywords (i.e high search volume=high value).As mentioned earlier short tail queries are extremely difficult to compete for, as they drive the majority of the traffic from the search results. 

Back in 2006, after a major leak of click data from AOL, it was revealed that the top 10 results in SERPs for a selected keyword can heavily affect click through rates (CTRs) and therefore website traffic. The top 3 positions in SERPs were driving 60% of the total traffic, while 75% of the users never scroll past the first SERP!  More recent studies from various companies such as Optify, Catalyst and advanced web ranking illustrate the trends of CTRs in search results depending on various factors such as the appearance of search ads or not and the appearance of local, news or multimedia results.    

In October 2013, the top ranking site for the keyword "dog breeds" in Google Greece, received 5K clicks in organic traffic out of 14,8K local monthly search volume. That is a 33,7% CTR for the top position, which is significantly lower than the AOL figures below and demonstrates a trend that CTRs for top organic rankings will shrink further in the future.

The top 10 rankings and their hierarchy in CTRs

As mentioned earlier, the probability of a new webpage that has completed onpage optimization to rank among the top rankings is limited only to results for long tail queries. In this case, a realistic approach for keyword planning would be a granular distribution of long tail keywords across webpages that cover a niche subject in depth.

The question is, which part of the search demand curve should the webmasters pursue? Although the answer is common sense, the process of selecting the keywords is tricky and the process can be overwhelming. Note this: The ideal keywords or keyphrases to target correspond to the search queries with relatively high search volumes and weak ranking sites in their search results.

It is important to keep in mind that depending on the country you are targeting, search queries volume and number of words forming keywords may vary. For instance, in larger and mature online markets such as the US, monthly search volumes are measured in 10Ks, 100Ks or even millions and short tail keywords usually consist of 2~4 word phrases respectively. In smaller less mature online markets, such as in Greece, monthly search volumes are limited to few thousands and short tail keywords are usually limited to only 1~2 word terms.

Analyzing Keyword Competitiveness

To analyze and determine keyword competition you need to take in to account the numerous ranking factors affected by "keyword specific" metrics such as:
  • the search volumes for selected keywords.
  • the number of competitive webpages that have implemented onpage optimization using the selected keywords or include keywords in domain names.
  • the domain age, page rank and page authority of the competitive webpages.
  • the number and the authority of the pages and domains that link back to competitors.
  • the frequency of keywords appearing in internal and external anchor texts.
  • the number of paid search ads showing in search results and competing for user attention.
  • the estimated paid search metrics (such as CPC and competition).  
In practice, you can analyze keyword competition by either using a set of free tools available online or just your custom spreadsheet (at least, this is how i started!) or even subscribe to several premium services such as market samurai moz pro and longtail pro that will save you time. It is recommended though before putting a hole in your pocket for paid tools, to start experimenting using some free tools, Google's advanced search operators and your custom spreadsheet. 

Keyword research, analysis and planning can sometimes become overwhelming or not productive. It is recommended that you always target keywords or keyphrases with relatively high search volumes and weak ranking sites in the search results. 

The future of keywords research(ed) 

At present, exploratory search is already transforming into a semantic search scheme  (from the Ancient Greek word sēmantikós; important) and keywords have started to be replaced by contextual entities, which is actually the real aim of the latest Hummingbird search algorithm update. The following video displays Google's official attempt in 2012 to shift from keyword to semantic search though "knowledge graph", a search technology that understands the real meaning behind keywords. In Feb 2013, Sergey Brin at TED2013 stated that "My vision when we started Google 15 years ago was that eventually you wouldn't have to have a search query at all."


Through the use of structured data, geotargeting, voice and social signals, the emerging semantic web will help the search engines deliver more accurate and personalized results, closer to the searcher intent.  This shift will affect SEO in the future by making the entire SEO process more difficult and consequently more fair. The image below illustrates how the Google search algorithm has evolved over time in terms of functions.    

November 2013,Vangelis Kovaios
Suggested reading:   
Google Search Algorithm Updates History 
Organic Search Acquisition [Infographic]      

Understanding Inbound Link Analysis