This guest post is by Dave Taylor of AskDaveTaylor.com.
Whether you’re writing about changing diapers, improving your bowling score, finding a job in the travel industry or how you get pictures off your cellphone, I think it’s a universal truth that if you’re writing online, you want better search engine results placement.
Most likely you’ve installed some SEO plugins that promise to improve your results and they might even be working, but if your site’s been up any length of time, it’s quite probable that things have started to break behind the scenes and hurt your results without you ever being notified. A scary prospect, really, and if it’s dramatic enough, you can start to really sink down the search results without any further explanation.
That’s why Google has its Webmaster Tools and while they’re primarily designed for people who have complete control over their Web site it can even be useful if you’re on blogger.com, wordpress.com or typepad.com. in fact, you don’t need to be a blogger to find it helpful: problems hurt any site, regardless of its structure.
Proving your own site
The first thing you need to do with Google’s Webmaster Tools is verify that the site you want to analyze is your own. This is typically done by adding a special line of HTML to the head of your home page, as I detail here.
If you can’t change your header, there are some alternatives that Google offers, but if you have zero administrative rights on the site, you might well be out of luck. If so, check with your hosting company to see if it offers alternative administrative tools that let you know about broken links, etc.
Key elements of a Webmaster Tools report
Once you have verified ownership of your site, you’ll see on the left side that the major areas are Site configuration, Your site on the web, +1 Metrics, Diagnostics, and Labs. Below it there’s some help that really highlights what you can glean from the Tools: Crawl errors, Search queries, Links to your site and Sitemaps. All good stuff.
Google Webmaster Tools overview for APparenting.com
There’s good analytic data that appears to be somewhat of an overlap with what you can get from Google Analytics (or your favorite analytics package if that’s not your particular cup of tea) and sometimes it reveals things that perhaps you didn’t want to know, like “sexy girls” is the #1 search for people who get to my Attachment Parenting Blog. Yikes. Not what I write about on my site nor anything I want people to be seeking when they arrive on my blog.
The heart of the Webmaster Tools, however, are the diagnostics because it’s the primary way we can learn what Google’s search spider finds broke on the site. Go to Diagnostics and it further breaks down into Malware, Crawl errors, Crawl stats, Fetch as Googlebot and HTML suggestions.
All good stuff, but let’s go into Crawl errors as it offers great bang for your proverbial buck.
Crawl errors Webmaster Tools reports for APparenting.com
Not too bad. This blog has a few hundred pages but I’m only seeing 36 of the hated 404 not found errors. Look closely and you’ll see that the format is bad link, error encountered, linked from and date detected. The first one is illustrative:
Link: http://www.apparenting.com/cosleeping-cpsc.html
Error: 404 (Not found)
Linked From: 10 pages
Detected: Jul 30, 2011
The real value is that if you click on the link that shows how many pages have a link to the bad URL, it’ll show you exactly what pages need to be fixed on your site and, sometimes, on other sites too. Here’s an example:
Specific crawl errors for APparenting.com
The first link is from another site called bubhub.com.au but all the other pages that link to this bad URL are on my own site. That’s something I can fix immediately.
Where to go from here
You can see we’ve just touched on the tip of the iceberg with the Google Webmaster Tools. It’s deep, it’s complicated, but even if you just poke around and look at the 404 errors generated for your own blog and fix as many as possible, you’ll be pleased to see how your ranking improves and, perhaps even more importantly, you’ll be happy to know that you’ve just improved your readers experience. And in the end, there’s nothing more important than happy readers, is there?
Dave Taylor has been blogging since the tools first appeared online. This is his 31st year online. His primary blog is the popular Ask Dave Taylor! offering up free tech support on a wide variety of topics including blogging and SEO. You can find him on all the major social networks through DaveTaylorOnline.com.
For me,its particurely useful to use the fetch as google bot function. If you for example change the url structure or add in a whole new section of the site, you will be able to have the whole site re-crawled. This can also reduce the amount of 404’s on your site. 404’s can arise when working on your site content and plugins and then having google crawl the site and picking up the links to that particular functionality which you then decide against using after testing which leads to the links to those pages becoming 404’s.
You are allowed to fetch as google bot, off the top of my head, 40 times a month, while having your whole site re-crawled is a definite 10 times a month
Thanks for the note, Baadier. I agree that using the Google fetch function is interesting (though it can also be more than a bit confusing). Here’s the thing: even if you only deal with 50% of the errors and problems that are reported, well, that’s still 50% better than ignoring it all. :-)
Great guide, I’m using Webmaster Tools every single day!
Thanks Dave for sharing this information, I was looking for it.
You are right it is just a tip of the iceberg, there is so much more to learn.
Good practical guide to using Webmaster Tools. Thanks for the post.
upi:dyd-dgyeo
So how actually do you fix the problem on the pages that lead to that 404 errors?
If it’s a page you control, then you correct the typo or obsolete link on the page. If it’s a third party page you can email then webmaster encouraging them to fix the problem and hope for the best. If there are a LOT of references to a now-broken URL on your site, you can also consider creating a new page that matches that particular query (or using a 301 permanent redirect rule in your server config). I’ve done the latter once or twice when the bad URL has migrated to a lot of different sites. Fast and easy. :-)
You didn’t actually tell me how to fix the 404 errors. How do I do that? I’m getting 30 404errors and 57 robots.txt errors. I need help, and I’m very new. Please help me, anyone?
Dave,Thanks for sharing this information.I’m using webmaster tools and it’s really helpful.
Dave,Thanks for sharing.I’m using webmaster tools and it’s really helpful.
I could really use more in-depth info on how some of these errors arise and also about fixing these errors.
For example, I’ve got three errors right now that I absolutely CANNOT find the bad link on the page, despite checking many, many times. Then I’ve got two that just showed up today . . . but the errors were detected close to a year ago and refer to a script that ran only the first couple of weeks that I had the site (before I switched to WordPress).
Do I need to take some action with Google, or just let it slide?
I’d let it slide, just keep an eye on things. Lots of moving parts in that Google machine, so it’s smart to check in and try to fix stuff, but sometimes, yeah, it can be baffling. :-)
And if you already have Google Analytics installed, it’s even easier to get Webmaster Tools working. Thanks Dave. Another notch closer to being a better blogger.
Great post on how to identify the 404 errors. I am not so sure about fixing them.
Hey, I read your post and I’ve known about a lot of my 404 errors and crawl errors for a few months now, but how do I fix them? Some of them are just failed redirects and things like that, but I come from a writing background and my back-end knowledge is limited (I’m learning, but it’s slower for me than writing tips). So how do I get those pesky crawl errors gone for good?
General strategy: identify the pages that have the bad links on them, open up those pages (either those entries in your blog tool or the pages themselves if you have a content management system), use the brewer’s search feature (Cmd-F or Ctrl-F, probably) to look for the bad link, fix it. Save the page. Fixed!
Have used the Webmaster tools. They do work and have helped with our corporate website. Without stats it seems hard to know where you’ve been and where you are going. Google Analytics has been a great help.
Excellent advice and I’ll have to check my site on a regular basis like monthly to make sure everything is in good shape.
Google Webmaster Tools is right up there as one of the best for my line of work. Really can’t live without it and without Xenu. Check that one out as well guys.
I never knew how to be a webmaster. Thank you for sharing a tip on how to fix such errors!
– Jack Leak
that is what i was facing with 404 due to wrong coding in wordpress template however it got fixed, but after the fix it really showed good ranking in searchengine
Very clear post – thank you. However, I still cannot find an answer to my current problems with 404 errors, either in your post or in the comments thread.
I have over 200 404 errors, and most of them I don’t know how to identify! When I click on one of the bad URLs, I get a WordPress notice stating:
“This is somewhat embarrassing, isn’t it? It seems we can’t find what you’re looking for. Perhaps searching, or one of the links below, can help.” These “links below” comprise multiple posts from my WP blog. It would take me an age to work through each of these in search of a faulty link, and 200 times the time to identify ALL such 404 errors.
This can’t be right. I must be misinderstanding something. If anyone has an idea of what’s going on here and how I can identify all these problem URLs – presumably from my blog – would be very appreciative.
BTW, I have a Broken Links WP plugin, and have systematically removed all broken links identified by that, so I don’t understand why I should still have so many 404 errors brought up in Google Webmaster Tools Diagnostics.
Cheers all
R
I have been working seo for one site few days back 404 errors were very all of a sudden now google web master tool is showing more than 100 errors. Can anybody let me know how to fix 404 errors?
Thanks
One thing I don’t like about Google Webmaster Tools is that it also detects some crawl errors that aren’t really errors. What I mean is if you block some pages via robots.txt file Webmaster Tools will see it as crawl errors.
I did a big review of Webmaster Tools not o long ago, feel free to check it out: http://newinternetorder.com/what-is-google-webmaster-tools/
nice infomation i am getting more than 80 not found urls in webmasters.
i am using wordpress (tech2fun.com)..at the end of each link my site name will come. how to fix it?
plz help me…
eg..tech2fun.com/post url/tech2fun
tech2fun excess
img: https://sites.google.com/site/funtrickz12/home/img/404-error.png
With regard to fixing 404 error with 301 redirects, I’m wondering if this causes traffic to decrease. There’s a bit decrease on my traffic since I used the WP-redirects plugin; still figuring out what might have caused.
I found 114 crawl errors 404 (Not found) in my blog, I was removed using removal url tools in webmasters ut it doesn’t help. I saw in removal page that the link was completely removed but in crawl errors still appear 114 errors…any suggestion please??