Using URL Shorteners, 301 and 302 Redirects to Spam Google
It has been a while since I have covered BlackHat Tactics, partly because I have been busy, but partly because I didnt want to be part of the problem of pushing them. However I decided, I rather cover them, so that sites that fall prey to these, can take preventative actions.
Typically sites that link out, but may not want to pass equity or to track outbound clicks, or want to cloak the urls would use redirect URLs, similar to URL shorteners. A good example is twitter – who uses the “t.co” domain to link to any links you post on the site in order to shorten them (in addition of using nofollows on those links).
However there may be a situation when those links or redirects get indexed themselves, which isn’t ideal.
This isnt great - these links have no value to search results pages, ever since they changed the way they index content despite the robots.txt directive many people dont realise that these can be an issue. And since these are simply links, really how is a normal user going to keep those pages out of the index? Plus the situation could have some interesting, non intended actions, I wonder if Aaron Wall knows he is an author on Youtube.com?
Most likely most people wont come across these results, however I have seen them being used to spam search engines, its an exploit often used by spammers by creating chains of redirects over time to get multiple listings for long tail keywords. It takes a while for Google to disregard these.
Using 301 and 302 redirects to Spam Search Results
Chained redirects aside, redirects, especially from URL shorteners can be used to spam, and to outrank sites.
It is no surprise that popular URL Shorteners pick up a lot of equity over time. This means that they have a lot of juice behind them, even if they are simply a bunch of redirects. Which means that you could potentially get any URL from these domains ranking easier than you could with say a new site. So Ideally by hijacking those URL shortened links and juicing them up, you are ranking by proxy. The funny thing is, I have noticed a change in the way Google handles these the last quarter of 2013, and have been keeping an eye.
The example that highlighted this to me was I had a couple of penalised sites that I was working on to clean up the backlink profiles. One of them had a 302 redirect that was indexed – once the site got hit for its brand, the 302 ranked. You can see a similar example in this post about RetailChoice by Dave Naylor where the 301 ranked for a little while till Dave called them out – I am pretty sure that the redirect was manually removed from the index, as the examples I was tracking privately stayed ranking for months.
URL Shorteners are pretty much redirect urls, so the behaviour would be the same as conventional redirects. So I tracked a few, and I was right, the dial on them ranking had been turned up, or there was a change in the way Google was treating them – below is the growth of rankings for Bit.Ly and a screenshot of SEMrush KWs that the site ranked for in Jan 2014 (Note: I think that the dial may be turned back down, as in Feb there are fewer results than in Jan from the stuff I was tracking).
The below is the growth of Po.St url shortener – not as start as Bitly, but I think it is being used less.
In fact the situation is so well known amongst certain groups, that The Goo.gl shortener is being used heavily to spam hotel serps using Google Local listings. You simply need to check the results shot out by SEMRush to see how well they work: http://www.semrush.com/info/goo.gl+(by+organic)?db=us
And although SEMRush shows a drop in the rankings, what it DOESN’T cover is the local rankings boost that each one of those listings have, so I assure you, there is a lot of traffic flowing through those local listings.
How Can I Spam Google Using URL Shorteners?
The process is simple:
- Shorten a URL
- Hammer with links
- Watch it rank
- Rinse, repeat
I wonder exactly HOW many people have caught on to the process? But you simple need to download the full SEMrush ranking file for Bit.ly to see the keywords it ranks for. It’s insane. Click to get the CSV free to see for yourself.
Across my checks, The Bit.ly and Goo.gl ones were the most abused, however scans of a bunch of them indicate some sort of patterns than can be used to exploit for rankings.
So how do I block redirects from being indexed in Google?
Yoast has a pretty nifty little post on keeping affiliate redirect links out of the SERPs, however I don’t agree with him on the rules he has set:
The basic process of cloaking affiliate links is simple:
- Create a folder from where you’ll serve your redirects, I use /out/.
- Block the /out/ folder in your domains robots.txt file by adding:
- Use a script in your redirect folder to redirect to your affiliate URLs.
I agree with all but no. 2. If you disallow the crawling of a page, the search engines will NOT see your XRobots directive, meaning the “virtual” page url WILL be indexed:
As you can see by the HTTP header check, the right Xrobots directives are on the page:
The problem is by blocking in Robots Txt, you fall foul of the same issue I have covered previously, search engine may end up indexing the link or virtual page if you block at domain level, but don’t allow a crawl. The use of X-Robots for virtual pages is correct however.
So for virtual pages, I do suggest the use of the X-Robots Directives (see Googles documentation on them here), but I suggest allowing the crawl of the domain or directory first so that the spiders can actually read the headers in order to obey them.
If you enjoyed this post – help me out and share it :)