Website Design in Aberdeen, Scotland.
Development,
Content Management and Web Hosting
AVG 8, Twenga bot and the sharp rise in bandwidth
June 27, 2008
Bandwidth is not free. It is paid for by someone, somewhere and despite the illusion of unlimited and infinite bandwidth available to us, an illusion fostered by the growth of faster access solutions and mobile connectivity, it can be pretty damn expensive! Website owners ultimately pay for this bandwidth use, through their hosting bill, and broadband users tend to get hit by the cost of it too.Over the last few months we have noticed a sharp increase in our bandwidth use that was not easily explained. It turns out to be coming from the new AVG 8 LinkScanner. When you use AVG 8 and do a search on Google, AVG 8 starts to scan each site on that Google page, before you have even clicked on it! How clever is that?! Very clever, but not very economical! It's a bit like going into a restaurant and find you are being charged for the whole menu, although you have only had a haddock and chips!
And to make matters worse, AVG pretends that this bogus traffic is coming from real user clicks, specifically, it disguises itself as Explorer 6, so webmasters and website owners who traditionally scan their website statistics and log files to measure their websites' performance are being duped into thinking that their websites have become more popular. It is difficult to isolate the AVG traffic, and AVG deliberately makes it so.
Apparently Wikipedia-watch have estimated that LinkScanner traffic now outstrips legitimate traffic to the Wikiepedia website by a factor of 10! Have a think about the implications of that for a moment.
And of course it doesn't stop with AVG. Looking at the statistics for this website, this morning, I noticed that our top visitor this month has been the Twengabot, with 10009 visits to the website. Why Twengabot - a price comparison site - should even visit this site once would be a mystery, as we have no prices mentioned anywhere. But to visit 10,000+ times is just ridiculous. And each visit consumes bandwidth paid for, not by a Twengabot, which also brings us no benefit, but paid for by us.
Probably it is only a matter of time before bandwidth, like water perhaps, becomes chargeable to the user rather than the provider of it. There might well come a time when you pay to visit each website, even in some form of not yet invented micro units.
That might stop the bandwidth leechers dead in their tracks. Currently their business models rely on theft and deception really.
Comments (2)
TWengabot sucks
Pieter - Aug 30, 2008 9:37 AM
I have today blocked twenga from pirating my site. Twenga is roughly copying content from other sites, and publishing it again.
What a businessmodel!
Crawl-delay
Biliboy - Sep 01, 2008 2:30 PM
As it says in their site "The TwengaBot also fully supports the robots.txt protocol, particularly the crawl-delay instruction.
You might want to add the following line to your robot.txt file :
Crawl-delay: 5"
About the business model,I think that it can makes you money :
- Your price is the lowest for a product
- Twenga crawls it
- A Internet user that uses price comparison sites see it and come to your site and buy it.
It has cost you nothing...
