Using URL Shorteners, 301 and 302 Redirects to Spam Google

It has been a while since I have covered BlackHat Tactics, partly because I have been busy, but partly because I didnt want to be part of the problem of pushing them. However I decided, I rather cover them, so that sites that fall prey to these, can take preventative actions.

Typically sites that link out, but may not want to pass equity or to track outbound clicks, or want to cloak the urls would use redirect URLs, similar to URL shorteners. A good example is twitter – who uses the “t.co” domain to link to any links you post on the site in order to shorten them (in addition of using nofollows on those links).

However there may be a situation when those links or redirects get indexed themselves, which isn’t ideal.

Twitter  Redirect URLs

Youtube redirects

This isnt great  - these links have no value to search results pages, ever since they changed the way they index content despite the robots.txt directive many people dont realise that these can be an issue. And since these are simply links, really how is a normal user going to keep those pages out of the index? Plus the situation could have some interesting, non intended actions, I wonder if Aaron Wall knows he is an author on Youtube.com?

aaron Wall youtube Author...

Most likely most people wont come across these results, however I have seen them being used to spam search engines, its an exploit often used by spammers by creating chains of redirects over time to get multiple listings for long tail keywords. It takes a while for Google to disregard these.

Using 301 and 302 redirects to Spam Search Results

Chained redirects aside, redirects, especially from URL shorteners can be used to spam, and to outrank sites.

It is no surprise that popular URL Shorteners pick up a lot of equity over time. This means that they have a lot of juice behind them, even if they are simply a bunch of redirects. Which means that you could potentially get any URL from these domains ranking easier than you could with say a new site. So Ideally by hijacking those URL shortened links and juicing them up, you are ranking by proxy. The funny thing is, I have noticed a change in the way Google handles these the last quarter of 2013, and have been keeping an eye.

The example that highlighted this to me was  I had a couple of penalised sites that I was working on to clean up the backlink profiles. One of them had a 302 redirect that was indexed – once the site got hit for its brand, the 302 ranked. You can see a similar example in this post about RetailChoice by Dave Naylor where the 301 ranked for a little while till Dave called them out – I am pretty sure that the redirect was manually removed from the index, as the examples I was tracking privately stayed ranking for months.

URL Shorteners are pretty much redirect urls, so the behaviour would be the same as conventional redirects. So I tracked a few, and I was right, the dial on them ranking had been turned up, or there was a change in the way Google was treating them – below is the growth of rankings for Bit.Ly and a screenshot of SEMrush KWs that the site ranked for in Jan 2014 (Note: I think that the dial may be turned back down, as in Feb there are fewer results than in Jan from the stuff I was tracking).

Bitly - Spammers Paradise

Jan 2014 KWs Bitly Ranked for

The below is the growth of Po.St url shortener – not as start as Bitly, but I think it is being used less.

In fact the situation is so well known amongst certain groups, that The Goo.gl shortener is being used heavily to spam hotel serps using Google Local listings. You simply need to check the results shot out by SEMRush to see how well they work: http://www.semrush.com/info/goo.gl+(by+organic)?db=us

goo gl redirect local spam

And although SEMRush shows a drop in the rankings, what it DOESN’T cover is the local rankings boost that each one of those listings have, so I assure you, there is a lot of traffic flowing through those local listings.

How Can I Spam Google Using URL Shorteners?

The process is simple:

  1. Shorten a URL
  2. Hammer with links
  3. Watch it rank
  4. Rinse, repeat
  5. Profit

I wonder exactly HOW many people have caught on to the process? But you simple need to download the full SEMrush ranking file for Bit.ly to see the keywords it ranks for. It’s insane. Click to get the CSV free to see for yourself.

Across my checks, The Bit.ly and Goo.gl ones were the most abused, however scans of a bunch of them indicate some sort of patterns than can be used to exploit for rankings.

So how do I block redirects from being indexed in Google?

Yoast has a pretty nifty little post on keeping affiliate redirect links out of the SERPs, however I don’t agree with him on the rules he has set:

The basic process of cloaking affiliate links is simple:

  1. Create a folder from where you’ll serve your redirects, I use /out/.
  2. Block the /out/ folder in your domains robots.txt file by adding:

Disallow: /out/

  1. Use a script in your redirect folder to redirect to your affiliate URLs.

I agree with all but no. 2. If you disallow the crawling of a page, the search engines will NOT see your XRobots directive, meaning the “virtual” page url WILL be indexed:

Yoast VPS Header Check

As you can see by the HTTP header check, the right Xrobots directives are on the page:

Yoast VPS Pages

The problem is by blocking in Robots Txt, you fall foul of the same issue I have covered previously, search engine may end up indexing the link or virtual page if you block at domain level, but don’t allow a crawl. The use of X-Robots for virtual pages is correct however.

So for virtual pages, I do suggest the use of the X-Robots Directives (see Googles documentation on them here), but I suggest allowing the crawl of the domain or directory first so that the spiders can actually read the headers in order to obey them.

If you enjoyed this post – help me out and share it :)

URL Shortners can be used to set up Social Media Vanity URLS.



 

Using URL Shorteners, 301 and 302 Redirects to Spam Google
User Rating: 3.9 (5 votes)
Rishi has been a consultant in online marketing for over 10 years, specialising in SEO, PPC, Affiliate Marketing, Email and Social Media. Over the years he has worked with many brands as well as many small businesses.
  • Pingback: Search Results pages - Should we "No Index" them in Robots txt files? | RefuGeeks()

  • http://blogprofitmedia.com Sam Woods

    Interesting tactic – you can just use a shortened link as a ‘buffer’ for your site so you could spam links at the shortened link and not your site? (making another tier of links) Surely doing this too much will be counterproductive? You’ve certainly got me thinking, I’m now wondering what to test it on :)

  • Gab Goldenberg

    Fascinating stuff here Rishi. It reminds me of stuff Dan Thies posted a few years back about similar 301/302 magic wool being pulled over Google’s eyes to hijacks site’s rankings.

    Any legit uses of this for whitehats? Could you rank a URL this way that wouldn’t be ranked on the home domain? Whitehat insofar as not stealing from merchants…

    • rishil

      @gabgoldenberg:disqus never a white hat use BUT I advise sites that have complete penalties to simply get a similar domain and 301 it on their penalised domain and then submit that domain to the index. It ranks for brand while they sort out penalty.

  • Pingback: Inbound Marketing Weekly Recap 03/03/14()

  • Jon Hogg

    Do t.co and bit.ly use 302 or 301 (I’m too lazy to check)? If a 301 then they shouldn’t rank by definition should they?

    • rishil

      No they shouldnt. But its an issue in the algo at the moment which can be manipulated in many more ways.

  • http://snipac.com/ Shajib Khan

    Try http://snipac.com/ this is a new service but great for all and it’s FREE. It has a lot of features like link life duration, password protection, Custom Alias, Description, Geotargeting, QR Code and more.

  • http://mizine.de/ Viktor Dite

    OMG! I have searched site:myOwnShort.link and all links are indexed! Bad bad bad! Please tell me please how to set noindex x-robots header tag?

    • http://mizine.de/ Viktor Dite

      .htaccess Header set X-Robots-Tag “noindex”