Enterprise Level Competitive Keyword Research

This is my first post in 2011 and i want to do something big. And when it comes to ‘big’ i can’t think of anything other than enterprise seo. It is doing seo on a grandiose level and is generally done on websites with hundred thousand pages or even million web pages.

Enterprise seo is different from a regular seo in a way that every step you follow in your seo process is scalable and is largely automated to reduce the turnaround time (time required to do seo and produce results) of a seo project.   By the time you will end up reading this post, you should be able to do competitive keyword research on a website with over 1 million web pages in less than 60 minutes or so. So here we go:

 

Step1: Scan your competitor’s website using Xenu link sleuth.

Xenu’s Link Sleuth is a free tool primarily used to check websites for broken links but it can also be used for many other things like scanning the title tags of your competitors and this is what we are going to do. Scan the title tags of the competitors as titles generally contain the keywords your competitor is targeting.

Speed up your crawling

Speed up your crawling

Xenu can crawl over 1 million URLs. But you would probably won’t like to wait for hours before the crawling finally stops. So if you want to speed up crawling then deselect the ‘check external links’ check box (to skip crawling of external links) and skip the images, CSS and Java Script files from being crawled by adding their folders in the box as shown above. Feel free to add as many folders, categories, or sections of competitor’s site you don’t want to be crawled by Xenu. Then click on the ‘more options’ button and uncheck all the check boxes as shown in the image above. The point here is, don’t waste your time crawling hundred and thousands of image files (unless you are esp. crawling for image optimization purpose), CSS files, Java script files and irrelevant sections of the site (like forum posts) which are not used by competitors to target keywords. All this will speed up crawling and save your precious time.

 

Step-2: Export the crawled data into excel and apply filters

Once you have crawled the website and have got the report then select ‘Export to TAB-separated File…’ from the Xenu’s ‘File’ menu. Open the file in MS Excel and apply filters to filter out the data by ‘text/html’ as shown in the image below:


Step-3: Export the data into a notepad

Copy and paste the title tag column from excel into a notepad file (.txt) to remove formatting. Remove all the brand terms (if any) from your title tags column data by using ‘Replace’ command (CTRL-H) in notepad. Your title tags column data now should contain a list of all the keywords your competitor is targeting. Keep a copy of this list as you will need it after some time.

Step-4: Export the data into your favorite keyword research tool

Copy and past the first 100 keywords into your favorite keyword research too like Google Adwords Keyword tool. You can export only 100 keywords at a time to Google keyword tool. If you want to remove this limitation then use Google Adwords API. Click on the check box ‘only show ideas closely related to my search terms’ before you get the keyword data from Google Adwords keyword tool.

Step-5: Download you keyword research data into excel and filter out the right data

Download the keyword research data from Google Keyword tool into excel. Remove all those keywords with less than 100 monthly search volume (or whatever you prefer). Create a new excel sheet and copy paste the original title tag list into it.

excel-sheet

Go to the ‘original title tag list’ sheet as shown in the image above. Add a new column named ‘Global monthly search volume’ (or local monthly search volume; whatever you prefer). Apply ‘vlookup’ to get search volume date for your original keyword list. I am interested only in the keywords my competitor is targeting. That’s why i have gone through so much trouble.

Now you have the list of all the keywords your competitor is targeting through title tags and you also have the search volume data of each keyword.

Step-6: Group your keywords into relevant categories

Just getting a huge list of keywords with their search volume data won’t help you much if you can’t categorize it into relevant groups. Again categorizing hundred and thousands of keywords is a very time consuming process. If you do it manually it may take several weeks. Here wordstream keyword grouper tool comes handy. This tool is free to use and can group a list of up to 10000 keywords (along with their search volume data) into relevant categories in few seconds. To use this tool just copy and paste the list of keywords into the wordstream keyword grouper box and click on the ‘Get Profitable Keywords Group’ button. You will get groups something like this:

wordstream-keyword-grouper

Click on the ‘email all groups’ button and then download all the keyword groups on your hard disk. Through these groups you can now decide the content categories/topics to write on which can help you in targeting the competitor’s keywords. This post is just a glimpse of how seo tasks are managed at the enterprise level.

Related PostKeyword Research for e-commerce Websites – Tutorial

 

Himanshu Sharma About the Author: is the founder of seotakeaways.com which provides SEO Consulting, PPC Management and Analytics Consulting services to medium and large size businesses. He holds a bachelors degree in ‘Internet Science’, is a member of 'Digital Analytics Association', a Google Analytics Certified Individual and a Certified Web Analyst. He is also the founder of EventEducation.com and EventPlanningForum.net.

My business thrives on referrals, so I really appreciate recommendations to people who would benefit from my help. Please feel free to endorse/forward my LinkedIn Profile to your clients, colleagues, friends and others you feel would benefit from SEO, PPC or Web Analytics.

 

 

  • http://Website Dana

    Great post. Can you tell me best enterprise seo tools out there? Thanks

    • seo himanshu

      I think searchlight is great tool for enterprise seo. You can request a demo here.

      • http://www.bing.com/ Yancy

        Information is power and now I’m a !@#$ing dictotar.

  • http://Website sushant singh

    I always look for innovative ways to get keyword research data. Thank you for the information. Truly appreciate it.

  • http://Website SEM Alex

    Really great idea for competitive research. I have never heard of Xenu before. Thanks for introducing me to such a wonderful tool.

  • http://Website Jitender

    Excellent work Himanshu. You are on your way to become enterprise seo :)

  • http://Website Dean Blount

    This post is great and very smart use of xenu link sleuth. I will definetely use this technique.

  • http://Website Rogue-SEO

    Never thought of using Xenu for competitive research. Thanks for sharing this trick.

  • http://Website Jerry Smith

    Excellent post Himanshu. It is a great way to see which keywords competitors are actually targeting. I guess your method can also be applied to small websites. It just speed up the whole keyword research task.

    • http://www.bing.com/ Marni

      Wow, this is in every rsepcet what I needed to know.

  • http://Website James

    There is wealth of information in your post and i found your approach really creative. Xenu can let you do variety of seo tasks and is a great tool for analyzing the site architecture.

  • http://Website Mary stewart

    It really is obvious and yet never occured to me that Xenu can be used to scan the title tags of competitors. I find your blog very informative. Thank you for sharing this stuff. Will definitely look forward to the next enterprise post from you.

  • http://Website Michael

    Enterprise seo is indeed taking off and i think conductor is a market leader in the enterprise market. I would love to see more posts from you on this subject. Thank you Himanshu.

  • http://Website Mike

    Never thought to use xenu this way. Great idea.

  • http://Website Jonathan Effler

    Excellent write up and great insight into enterprise competitive keyword research.

  • http://Website David

    It is always nice to know how other SEOs do keyword research. I am going to use this tactic on one of my competitor. Thanks for sharing.

  • http://Website SEOAndy

    I am working on a site at the moment which has more than 200,000 web pages. I am going to use this tactic to do keyword research on my client website.

  • http://www.channelfrog.com hyderalis

    Thanks Himanshu for writing this good info. I’ve been using Xenu link sleuth for creating sitemap for millions of pages. I think Xenu is one of the finest tool to get all the data you want & that is tool Free.

  • http://Website Suresh Aggarwal

    You have excelled yourself once again Himanshu. Great post.

  • http://Website Josh Miller

    This is a very interesting approach to do keyword analysis.

    • http://www.bing.com/ Veanna

      Stay with this guys, you’re hleipng a lot of people.

  • http://Website Faisal Khan

    Keyword research is a very important part of a seo campaign and i will be sure to use this post as guide next time i do keyword research. Thanks for this awesome post.

  • http://Website Rob

    This post seems like a great way of searching out keywords from competitors. Thanks for sharing.

  • http://Website 808SEO

    Fantastic post. I work for many big clients and we all know how difficult it is to optimize for big sites. I appreciate how you used xenu tool for competitive keyword analysis.

  • http://Website Amit kulkarni

    Your post came just at the right time. I was looking to do keyword anlaysis for a very big site.

  • http://Website Craig baxter

    Thanks. Very informative.

  • http://Website SEO Sam

    Really good example of how to do competitive analysis on a large scale.

  • http://Website seo-god

    I got a lot out of this great post. Excellent work.

  • http://Website Brian

    Wonderful post. I have just started using Xenu for site analysis and it is a great tool.

  • http://Website Manish

    That’s a great post as it shows how easy it is do keyword research for websites with million of webpages.

  • http://Website Kane

    Great post. I never thought of doing keyword research with xenu. When i do keyword research i listen what my client has to say about his products. How he and client describe them.

  • http://Website Christopher

    Thanks for this.Your post shows that how creativity can solve even big seo issues with such an ease.

  • http://Website seohaven

    By reading this we will be able to keyword research more effectively. Thanks.

  • http://Website Italian-SEO

    Thanks for the thoughts Himanshu. I also like your previous post on ‘real time keyword research’. You seem to be an expert in keyword research.

  • http://Website kansas-SEOGuy

    I love these posts that scale seo tasks and speed up the whole process. Great work Himanshu. Hope to see more from you.

  • http://www.redesignunit.com Custom Website Design

    Great work Himanshu and this post help to us also

  • http://Website Darre

    Why just not use SemRusH?

  • Ganeshan Nadarajan

    you should be able to do competitive keyword research on a website with over 1 million web pages in less than 60 minutes or so.