Posts in Tech

Moving away from W3 Total Cache back to WP Super Cache

Yeah more wordpress plugin talk, but since 1/3 of all S2R sites are powered one way or another by wordpress, its kinda something i deal with, so why am i moving away from the caching wonder kid W3 Total Cache to the old school WP Super Cache, well mostly after a few weeks of testing, i found that although both do a excellent job there are some basic strong points and weak points on both of them:

W3 Total Cache

  • Strong Points: Loads of Options, Good Control Panel, Good Performance
  • Weak Points: Doesn’t have a Good Basic Default Setting

WP Super Cache

  • Strong Points: Keeps it Simple, Good Control Panel, Good Performance
  • Weak Points: For this case, none

Soooo, what does this mean, first it means both are really good, but also that even after a lot of tweaking and reading a lot about its settings, its still a difficult process to push W3 Total Cache to use its potential, i have several different servers with different software running (Apache and Nginx, with different plugins and caching addons), and W3 Total Cache kinda feels a bit hit and miss, sometimes its wonderful sometimes it isn’t, while the “Keep it Simple Stupid” approach of WP Super Cache always brings good performance.

Also the fact that i tend to make tweaks and improve the code and performance both of the sites and server, WP Super Cache kinda gives me the best combination of rock solid performance and ease of use, while with W3 Total Cache i have to worry about a lot of different things to have it work properly, so therefore im moving away from W3 Total Cache to WP Super Cache ^_^’

Comparison of WordPress SEO Plugins

I’ve had my fair of problems and gripes with SEO on wordpress, especially with the “All in One SEO Pack” you can check it out Alternative to All in One SEO Pack ^_^

So that was like a year ago, nowadays there are loads of plugins that do what i want or part of what i want, however cause there is no point in testing 20 plugins, im narrowing it down, so for this review im choosing just plugins that have a minimum set of features that i need (like meta tags in the head, canonical urls,…), that have been updated recently (last 3 months), that aren’t on the first version (at least a couple of updates under the belt) and that have a control panel, also i’ve tested all these plugins for obvious misleading or security leaks, at this moment they have none that i can see (actually one has, sorry made this intro before the testing hehehe).

WordPress SEO by Yoast

  • Specs: 98kb zip / Average Rating on WordPress: 5 stars
  • Pros: its enable by default, nice detached admin page, explanations with loads of options, quite a few added features (like authentication for google webmasters or breadcrumbs, both nice but not necessary for SEO), import ability to several other SEO plugins.
  • Cons: inserts the most amount of junk on the site’s header including plugin version number than all the plugins in this list, quite a few irrelevant options/features.
  • Hummmm: Looking good, and it makes “All in One SEO Pack” look like amateur hour, it has almost the same amount of junk and promos but done nicely and cleanly, also its pretty well organized, if it wasn’t for all the junk inserted into your site’s header, it would have been a really high contender.

SEO Ultimate

  • Specs: 490kb zip / Average Rating on WordPress: 4 stars
  • Pros: huge amount of features (19 different modules), modular system (you can activate and deactivate features you want).
  • Cons: its not enable by default, some of these feature modules are just a one option affair, others offer features that are good for SEO research but that are not needed as a wordpress plugin (there are better tools and sites and ways to get that info), some modules seem incomplete, exploit scanner gave 3 severe warnings with SEO ultimate (obscured links and dropping tables).
  • Hummmm: wholy jesus, if gre’s high performance SEO is overkill, then SEO Ultimate is trying to live up to its name, by being ULTIMATE!!!! i would say that it could be a pretty nice companion to another seo plugin (by deactivating the modules with duplicate or irrelevant functions), still its the only one that gave security warnings, so with alternatives, i would stay away.

Greg’s High Performance SEO

  • Specs: 212kb zip / Average Rating on WordPress: 4.5 stars
  • Pros: simple, clear and very instructive admin page, loads of functions and explanations.
  • Cons: pub directly on your admin page (from, complex to setup, and to be 100% efficient needs to be hardwired into the theme, its not enable by default.
  • Hummmm: high performance or not, this is a case of overload, SEO is just a small part of a website performance and its not even the most important by far, a site with no SEO but with high quality content, performance and promotion will always win, this plugin is too much, also tweaking too much of SEO might do more harm than good, this one if for the ubber tinkerers.

Platinum SEO Pack

  • Specs: 137kb zip / Average Rating on WordPress: 4 stars
  • Pros: enabled by default, basically same feature set as “All in one SEO Pack”, clean detached admin page, ability to migrate from “All in one SEO Pack”.
  • Cons: still some junk on the site header including plugin version number (still a bit less than “All in one SEO Pack”).
  • Hummmm: im impressed by not being impressed, platinum seo is basically “All in one SEO Pack” without the shitty stuff, sooo pretty good ^_^


  • Specs: 435kb zip / Average Rating on WordPress: 4 stars (taking from their previously name)
  • Pros: ahhhhh…
  • Cons: not enable by default, huge amount of pages, complicated and confusing as hell, lots of functions are not available, while having lots of links to the pro version.
  • Hummmm: this one is supposed to be good with buddypress and wordpress mu, but in hindsight i rather have no SEO than whatever this is, so no pros, only cons, actually the only SEO plugin in this pack that i would stay away like it was the black plague.

All In One SEO Pack

  • Specs: 176kb zip / Average Rating on WordPress: 4 stars / Most Popular SEO Pack
  • Pros: Loads of features, default selection of options is good.
  • Cons: Its not enabled by default, huge amounts of junk and links and banners on the admin page, weird options, inserts junk on the site’s header including plugin version number.
  • Hummmm: I used to like it, but i think with time “all in one seo pack” as degraded itself, you can promote other stuff and still keep yourself useful and practical, the main fold doesn’t even have any options its just pub and shit, also some of the options and functions are not that useful in SEO or even practical.

Conclusion and What are you going to use?

Hummm this one is a big thought, but i would say the clear winners and real all in one seo alternatives are  WordPress SEO by Yoast and Platinum SEO Pack, but for diferent reasons, if you want a clean, simple SEO option for WordPress i would go with Platinum SEO Pack, it has everything you need, but if you want a bit of an edge and more options and better understanding of the features, then i would go with WordPress SEO by Yoast.

S2R is officially moving all their WordPress Sites from All In One SEO Pack to Platinum SEO Pack, mostly cause i don’t need the extra features that WordPress SEO by Yoast offers and i don’t like the added junk that WordPress SEO by Yoast adds to the site header.

Why use a 3 Tier Backup System?

We all know backups are important, if shit can happen, shit will happen, they are important both online and offline, but if you are hosting a site or several sites, its important to have a backup strategy, that’s why S2R created the 3 Tier Backup System (sounds way more exciting than what it is):

1) Hosting Provider Backup System

  • Choose a hosting provider that makes backups, having RAID and high availability and failover hardware, that’s all cool for performance and redundancy, but backups should be expected and not on the same hardware as the server (offsite or high end backup software is a plus), having backups from your host eases most problems and makes most disasters easier to handle, as such this is the first backup tier.

2) Offsite Backup System

  • Have a cheap vps or backup account from another hosting provider (if he also provides backups that’s a plus), then use your hosting panel or rsync or whatever backup system you prefer to use to make and transfer backups to this box, we normally choose a weekly schedule for this (running on weekends when traffic is low), there is no need for daily copies, cause the goal is to have a weekly clean backup, of course we store 3 backups, so 3 weekly backups are always available on the server, the idea is to use this encase your hosting provider goes bankrupt or closes your account for some reason (nowadays its more likely than you think), and you get cut off from your first tier backups, so therefore this is the second tier backups.

3) Local Backup System

  • This one is also easy to understand its a local backup of the accounts, in my case its to my custom made backup system (2TB mirrored to another 2TB, way more than enough for my sites and personal files + Mozy Backup of all of this), similar to RAID, this is done also weekly (could be done only monthly), this is done for the simple reason of peace of mind and safety, i never needed it, but there is no such thing as too many backups and having one locally guarantees that whatever happens to your sites they will always be able to comeback from any disaster, so this is the final third backup tier.

With a three tier backup system, it might look a little paranoid, and you might take some time and money to build it, but now that its done, its easy to add new sites and the peace of mind it gives is priceless, and now i can eat right in front of the computer hahahah ^_^

List of Fast Free DNS Servers

DNS (Domain Name System) Server is used to lookout domain names to ip address, it has more purposes, but for the most part thats their function, public DNS providers are free to use DNS servers that provide that function to anyone that needs it.

So since you are online the DNS service was probably provided by you ISP as part of their system to get you online, so why would you like to change your DNS provider from your ISP to a public one?

The main reasons, in degree of importance, in my view are:

1) Security, your ISP probably uses standard off-the-shelf dns software, also DNS although important is not a priority, security tend to suffer first, especially when new tricks and flaws are found, things like dns cache poisoning (injecting dns records that are not from a authoritative dns server), also phishing or malware or other kinds of security risks can be averted by using a stronger more secure dns provider.
2) Performance, every single time you put a new domain name, that name has to be resolved on a dns server for the domain to show up, that means performance is essential, especially comparing to your ISP dns server that is extremely close to you.
3) Control and Customise, the ability to take control of your dns, of using it to make your browsing faster or check it domains are there, if you mistyped, if there is malware on the page you want to visit, or blocking kids from accessing certain kinds of sites.

So the list here follows a different pattern since im going to check the speed of it (from 3 different locations) as well as my personal preference depending on features and overall performance:

1) Google Public DNS (Like: Speed and Standards / Dislike: None)


2) OpenDNS (Like: Speed and Security / Dislike: DNS Hijacking)


5) Level 3 (Like: Speed and Standards / Dislike: None)


4) Comodo Secure DNS (Like: Security / Dislike: None)


1) the checks were made from California (USA), London (UK) and Faro (Portugal) using namebench
2) i could have made a huge list, but i prefer to provide the cleanest most objective one, than a random copy paste list, so i removed lots of other good public dns providers, just cause at the time they were giving me errors and not performing normally, like scrubit dns, cisco or norton safe dns.
3) opendns would have won (just about the same performance as google public dns), but i personally prefer a dns that gives an error when it fails and doesn’t hijack or disrupt sessions by pushing their search when a server is having a glitch or dns issues, but that’s just me.

Optimize your WHM Cpanel

This isn’t gonna be a how-to increase the performance of whm/cpanel, there are loads of awesome tutorials online and i’ll link so some of those in the bottom, this is just some of the tips I’ve learned over the years, that can and do, and make a difference on the performance of your vps / vds / dedicated box using WHM Cpanel.

1) Disable Unused Services
WHM is a complete package it takes care of the services, of making and managing the account and updating the server and itself, because of its versatility it has all these services and more that you would want from a hosting server, however all of these services are not required and some of them are serious performance hogs, so these are the ones you should check out and if you dont need, disable:

  • SpamAssassin Server (spamd) – If you dont need anti-spam, nowadays most e-mail software/webmail provide anti-spam protection, so no point running it on your server, unless mail security is essential, also if you need it, you need to configure it properly so at least its not such a big performance hit.
  • Clamav Antivirus – Same thing as SpamAssassin, but for Anti-virus, and well clamav just detects the really blatant virus, so trojans and mallware might get trhough anyways, so you can disable this one, but if you need it its the same thing as SpamAssassin, configure the hell out of it.
  • Cpanel Pro – Added features, but not really needed.
  • Statistics Software – Well with google analytics and other offsite statistics software, you can live with just one stats software and webalizer is probably the lightest, disable the rest.
  • Mailman – Just use a php or offsite maillists systems.
  • Webmail – Unless you are hosting, most people are using pop3/imap or just offsite mail, so choose the most basic like squiremail, disable the rest.
  • Entropy Chat – enable only if you need it.
  • Melange – enable only if you need it.

2) Tweaking Settings

  • Uncheck Conserve Memory at the expense of using more cpu/diskio (however if you need the added memory, check it, but you take a performance hit).
  • Choose the best version for you of Apache, Mysql, FTP, PHP (just choose the one that fits better to your uses), and then tweak it, make some online searches on how to change the apache config’s or php settings (most are available right through WHM).
  • Make sure the default catch-all mail address is set to FAIL, so that it uses as little CPU time/Disk Space as possible.

4) Upgrade / Update
This one is simple… not always having the latest and greatest is the best course of action, but there is a tendency for newer software to have better performance, less bugs and generally more secure, so when in doubt, always update, in this case, its setting up whm to update itself to the CURRENT or STABLE releases and make sure security packages are also on automatic, also from time to time running the easy apache, and the updates on the Software section (Server and System).

5) Keep an Eye on it and Adjust
Since every box is different (although im talking here about LAMP+WHM/CPanel), there are multiple variations of LAMP (using CentOS instead of Debian, or php4 instead of php5), also it depends on the sites/services you are doing with the box, if your sites are dynamic you kinda want to adjust php, if its a file server, you kinda can move to nginx (or another lightweight server, instead of apache), if you have a lot of Mysql work, its another thing, so not all rules or changes will work best, so keeping an eye on the performance of the box is important, not only if the CPU and RAM are good, but also how quick your pages load.

I know there are a lot of more tweaks you can make so check these articles for more in depth tweaking (or just make a search yourself), however just doing the ones i told should keep your box humming away pretty good ^_^

Check …
Optimizing cPanel/WHM
Optimize High-Traffic Servers
Configuring and Optimizing MySQL For WHM Dedicated Server/VPS

Choosing the right Apache? Apache 2.0 vs Apache 2.2

Well i ordered a new box and found out that it was running Apache 2.0.63, a very stable version of Apache (still used by lots of webhosts), but since like a year ago, i’ve moved to the 2.2 branch, so to decide what was the best choice i decided to do some testing between both version to choose, humm performance wise?, and yes i know 2.2 is much better regarding speed, but performance is not just about speed, so i just installed one and then the other, did some tests and let it run for a full 24hours midweek (cause weekends the traffic drops a bit)

So what are the results, interestingly enough… im gonna make some charts ^_^

Average 24h CPU Load (Restricted to 1CPU)

Average 24h Memory (Restricted to a total of 1GB)

Average Requests of File Types

Ok charts are cute but were are the numbers? well there isnt a point or need really, since the site hosted (and test files) on the box are not a perfect benchmark of Apache’s abilities, this should be considered above all a real world test, than a well measured test, but we can take a conclusion out of this, just looking at the charts, Apache 2.2 branch does seem to perform faster than the Apache 2.0 branch, although that performance increase is traded by a slightly higher cpu and memory consumption (about 0.3% memory increase and about a 6% CPU load increase ), hummm i can live with that,  2.2.15 WIN! hehehe, besides newer software has a tendency to bring better performance and security, and in the long run thats always smart thinking.

How to Build your Pagerank

Improving your site’s SEO (Search Engine Optimization) and Google PageRank requires a combination of technical optimizations and strategic content development. Although if you want the quick rundown, just make great content, that 90% of the time is enough, because people want good content and therefore the search engines want good content!

Quality content creation – Develop high-quality, original content that provides value to your audience. This can include blog articles, guides, tutorials, videos, or infographics. Engaging and informative content not only attracts visitors but also encourages other websites to link back to your site, improving your PageRank. It’s like creating a captivating story that keeps your readers hooked and eagerly sharing it with others.

Keyword research and optimization – Conduct thorough keyword research to identify relevant search terms that your target audience is using. Optimize your website’s content, including titles, headings, meta tags, and body text, to incorporate these keywords naturally. It’s like sprinkling the right keywords throughout your website’s content, so search engines can understand what your site is all about.

Website performance optimization – Ensure your website loads quickly and performs well across different devices. Optimize images, minify CSS and JavaScript files, and enable caching to improve page speed. A fast and responsive website is a important marker of quality and therefore Google likes you more.

Mobile-friendliness – With the majority of internet users accessing websites on mobile devices, it’s crucial to have a mobile-friendly site. Make sure your website is responsive and provides a seamless experience on smartphones and tablets.

On-page optimization – Pay attention to on-page factors like title tags, meta descriptions, and header tags. Craft unique and compelling titles and meta descriptions that accurately represent your content and entice users to click through from search results. It’s like writing captivating headlines and engaging summaries that make people curious to explore further.

Backlink building – Earn high-quality backlinks from authoritative and relevant websites. Engage in guest blogging, collaborate with influencers or create shareable content that naturally attracts backlinks. Backlinks are like votes of confidence from other websites, signaling to search engines that your content is actually valuable and trustworthy.

Regular updates and maintenance – Keep your website updated with fresh content, fix broken links, and ensure all pages are accessible to search engines. Regular updates shows search engines that your site is actively managed and relevant.

Also please remember, SEO and PageRank improvements take time and continuous effort, its always a work in progress, and if you keep improving it will improve your rank and visibility! Until next time!

How to Perform a Server Move Without Downtime

Moving data between two servers without downtime requires some careful planning and execution. I’ve done this a thousand times, its also a good benchmark on the quality of your backups, so lets check my guidelines to a smooth server move!

Backup your old server – First, you need to make a fresh backup of your old server, this will guarantee that if something goes wrong you will still have a way to go back, you will have some downtime but nothing is loss, so start with a new backup.

Set up the new server – You need to prepare the new server to ensure it is fully configured and ready to receive the data. This includes installing the necessary software, configuring the server environment, and optimizing performance. Think of it as setting up a cozy new home for your data, complete with all the essential amenities.

Synchronise the data – To minimize downtime, you’ll want to synchronise the data between the old and new servers. This can be achieved through various methods, such as database replication or file synchronisation. i tend to ssh into the new server and rsync everything. It’s like making sure all your belongings are safely transported from your old house to the new one, ensuring nothing gets left behind.

Important Note Here, normally before starting the sync i put the old site in maintenance so there is no change or updates while the data is syncing, especially if the site is big or popular.

Test the migration – Before fully transitioning to the new server, it’s crucial to thoroughly test the migration process. This involves verifying that all data has been successfully transferred and that the new server functions correctly. Think of it as a dress rehearsal, where you ensure that everything runs smoothly before the big performance. I tend to use ssh with the diff command to achieve this.

Switch the DNS – Once you’re confident that the new server is ready to take over, you can update the DNS settings to point to the new server’s IP address. This step ensures that when users access your website or application, they are directed to the new server seamlessly. It’s like updating the address in your contact information so that people can find your new location without any hiccups.

Monitor and verify – After the DNS switch, closely monitor the traffic and performance on both servers. Keep an eye on any potential issues and ensure that the new server is handling requests correctly. Also make sure that other services dependent on that old server are transferred over like e-mail.

By following these steps, you can migrate data between servers with minimal to no downtime. However, it’s important to note that each migration scenario is unique, and careful planning and testing are crucial to ensure a smooth transition and always ALWAYS ALWAYS!!!!!! Have plenty of backups!

How to Protect your Sites

Well one of my sites was taken down for a couple of hours after it was completely screwed from a hack (well from script kiddies, but still), that deleted admin accounts and posts and added re-directs and other nasty stuff, cleaning it up would mean several hours and some things might be completely lost forever anyways, so what to do? before this happened, during or after to fix it, so what do i do to keep my sites online and protected, ill separate these into 3 major points:

Preventive Protection (before any problem)

  • Always have the latest updates to your online software, yes i know sometimes it brings new bugs, but most of the times its better to take the time to find workarounds and still update to the latest than opening yourself to an attack;
  • Always have multiple backups, all my hosts have backups but i also make my own to other servers (weekly) as well as a to my own computers (montly),  this ensures that even if there is a catastrophically bad failure (your host dies on you or deletes your account) that you are still able to bounce back pretty quickly;
  • Make sure your hosting is separate from your domains, since keeping those 2 together means if you need to jump to another host that you will always have problems (also have always a backup host that you like, and trust to jump to quickly if need be);
  • Use popular software, yes it might be a bigger target for hacks and security issues, but the chance of having updates and fixes is also much larger;
  • Resilient Hosting, doesn’t need to be cloud hosting or some strange arrangement, just needs to be from good hosting companies with good track records, they ensure that most hardware/server failures will never happen and if they did, that a fix would be done quickly and efficiently

Immediate Protection (when you first detect the problem)

  1. Put the site Offline, if you are on a apache server it normally means an update to the htaccess/htpasswd, you don’t want your users getting affected by your compromised site;
  2. Check to see how was the site compromised, was it the server, a bad admin, software flaw, try and find how did this happen;
  3. After you find out the flaw, search and see if there is a fix to it (server/software update), banning an admin, whatever it is, cause after you fix it, you need to make sure it doesn’t happen again.

Reactive Protection (how to fix the problem)

  • Best way is always, just delete the whole site and bring back the latest stable backup, sure you will lose some content or news but you have a guarantee that your site comes back crisp and clean, fixing it by hand means you can miss something and still keep your site compromised;
  • Make a test run and check if everything is alright, make sure to make the necessary adjustments before bringing the site back online;
  • Fix the security issue, if you found out what was the problem, go ahead and do the updates or workarounds, so this doesn’t happen again;
  • Make a brand new backup immediately before bringing the site back on, this ensures that if the site is still vulnerable, that you can bring it back up quickly, without much loss.

So that’s it, yes i know its basically using backups, and yes there are other ways, but this is the easiest more efficient way to protect your site from premature death ^_^

How to Deal with Web Scraping

Hummmm since i have several galleries, one thing i encounter often is web scrapers, personally i don’t mind if anybody takes the entire gallery home or even if they re-post it somewhere else, that is fine with me, this is the web, if i wanted the gallery to be private i would made it so, if its public and free then go right ahead…

However the content itself is not the problem, the problem here is that the vast majority of web scrapers has bad default settings or the users that use them put too aggressive settings, its not uncommon for the load on the server to go from 0.10 to 1 in a heartbeat or even go down, i know its partly my fault i personally like to restrict as little as possible the server or the software (i could use several methods to restrict connections or to ban ip’s if there are too many connections), however because i don’t, sometimes i get into trouble, so this is what i normally do.

First of all i have this on the .htaccces (mod_rewrite), that helps blocking most of scrapping software (unless it spoofs as a browsers hehehe):

RewriteEngine OnRewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]

RewriteCond %{HTTP_USER_AGENT} ^Bot\ [OR]

RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]

RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]

RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]

RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]

RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]

RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]

RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]

RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]

RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]

RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]

RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]

RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]

RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]

RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]

RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]

RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]

RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]

RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]

RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]

RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]

RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]

RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]

RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]

RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]

RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]

RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]

RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]

RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]

RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]

RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]

RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]

RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]

RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]

RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]

RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]

RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]

RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]

RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]

RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]

RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]

RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]

RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]

RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]

RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]

RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]

RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]

RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]

RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]

RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]

RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]

RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]

RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]

RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]

RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]

RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]

RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]

RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]

RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]


RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]

RewriteCond %{HTTP_USER_AGENT} ^Zeus

RewriteRule ^.* – [F,L]

I monitor the load of the server if the load as a spike for more than a couple of minutes i check the apache log, if there are lots of connections to the same site from the same ip, i ban this ip with .htaccess, adding this line:

Order allow,deny
Deny from
Allow from all

( is the ip on the logs) and check the load after a couple of minutes, if its down fine, if they jumped ip’s, i’ll do one of two things, if they keep on the same ip range i’ll just block that ip range, like so:

Order allow,deny
Deny from 100.100.100.
Allow from all

If they aren’t then i limit the connections to 10 on the cache or image directory (not on all the site), i know it will hurt all users but its better than nothing, just adding this line:

MaxClients 10

If it still persists, i’ll just close the image directory,adding this line again on the cache or image directory (depending on the type of image gallery software you are using):

deny from all

So the site stays up as well as the thumbnails, just the images won’t be accessible for a while, all of these are temporary measures, but for now, for me they do the trick, most of the times banning the ip is enough of a cure, and those i always leave on the .htaccess the other options i normally remove them the next day after the connections storm has passed, bottom line is if you want to scrape instead of bombing the server for an hour, make it so it downloads slowly during a couple of hours it makes a big diference and everyone gets what they want.

ImageBoard Spam List

Since i run several successful anonymous imageboards, one of the ways to prevent spam is to have a sort of spam list (i would say blacklist) of all the domains url’s that are not able to be posted on the board (since one of the main reasons to spam the boards is to post links to fishing or illegal sites), this way this prevents them being posted by bots altogether.

So its an essential tool to have a clean board and to keep things clean, even if later on more automated tools like akismet or defensio are included, this is still a nice clean and fast way to keep most spam or idiotic posts from the site, so the file is a simple spam.txt file on the root of the site, one domain/site per line, this of course is most useful to other imageboard hosts that use the same system (wakaba or kusaba clone software), so here is our very own custom spam.txt list hehehe for free yayy 1131 domains (i’m actually thinking about making our own system, so anyone can add links and have the latest version for their board so their system is always protected heheheh)

UPDATE (from tentakle and neechan):

Uhh since the last post we have added a bunch more, so there ya go an updated spam.txt this new list has 1372 domains that are/were used in spam posting on imageboards, the old link as well hehehe, you can use something like WinMerge or TextOpus to help merge your existing spam file with my awesome one ^^