Posts tagged s2r

New Portuguese Search Engine

Yep, i had some requests and some tweaks to do, so i made a new site, its called PT1 or Portugal 1, its based on google search with a portuguese free directory of the best portuguese sites we like (the search engine and all the added services are done, stuff like quick link search engines for firefox or search toolbar with some cool stuff ), the directory is still not live yet, also ill add more services later on, but for now it looks simple and cool and works like a charm ^_^

Moving to Asynchronous Tracking

So in this move to… well move faster hehehe, im pushing Google Analytics Asynchronous Tracking on all S2R sites that use Google Analytics (almost all, the ones that don’t, they use Reinvigorate ^_^ ), although I’ve tested through this past week the speed changes between normal code and the asynchronous code, i can say i don’t see much of a speed difference, that could be, because well, my sites already run fast, but anyways when you join all of those little tweaks that speed sites up, it makes a big difference, especially if you get bursts of traffic ^_^

New and Improved S2R

yep i made some improvements on S2R heheh, just making it nicer, still all of this “google now adding speed as another factor in ranking” is making everyone all nervous and getting CDN and doing all this changes, well i’ve always tried to make my sites, quick and easy, both in the webdesign/structure/hosting side of things, because i want my sites to be fast…, so… well im always trying to put one less picture in a site, or offload content or optimize code or have faster better hosting anyways, so i guess no change here ^_^’

Our Very Awesome Web Directory

Ok, major improvement on or very own Web Directory, aka orgme.com, and we are actually thinking about making a Portuguese version, ok maybe a bit different, and yes i know there are billions of web directories, but most are not nice at all (you have to pay or give reciprocal links or has misleading links or affiliate links or other nasty stuff), so we will try and make ours nice and helpful and clean and not weird or dodgy ^_^

Nee.Hostcult.com is Down

Yep sorry about that, so a couple of our sites are down for the moment >_<, the nee server is doing a FSCK as of now (probably due to a bad shutdown or some other glitch that didn’t come up before), once the FSCK is finished, the server should be back up ^_^’

update: every site is back up ^_^ total downtime 1h15m, yeah FSCK takes loads of time…but better that, than a corrupted hard drive hehehe as always never forget to make your own backups! ;D

How to Protect your Sites

Well one of my sites was taken down for a couple of hours after it was completely screwed from a hack (well from script kiddies, but still), that deleted admin accounts and posts and added re-directs and other nasty stuff, cleaning it up would mean several hours and some things might be completely lost forever anyways, so what to do? before this happened, during or after to fix it, so what do i do to keep my sites online and protected, ill separate these into 3 major points:

Preventive Protection (before any problem)

  • Always have the latest updates to your online software, yes i know sometimes it brings new bugs, but most of the times its better to take the time to find workarounds and still update to the latest than opening yourself to an attack;
  • Always have multiple backups, all my hosts have backups but i also make my own to other servers (weekly) as well as a to my own computers (montly),  this ensures that even if there is a catastrophically bad failure (your host dies on you or deletes your account) that you are still able to bounce back pretty quickly;
  • Make sure your hosting is separate from your domains, since keeping those 2 together means if you need to jump to another host that you will always have problems (also have always a backup host that you like, and trust to jump to quickly if need be);
  • Use popular software, yes it might be a bigger target for hacks and security issues, but the chance of having updates and fixes is also much larger;
  • Resilient Hosting, doesn’t need to be cloud hosting or some strange arrangement, just needs to be from good hosting companies with good track records, they ensure that most hardware/server failures will never happen and if they did, that a fix would be done quickly and efficiently

Immediate Protection (when you first detect the problem)

  1. Put the site Offline, if you are on a apache server it normally means an update to the htaccess/htpasswd, you don’t want your users getting affected by your compromised site;
  2. Check to see how was the site compromised, was it the server, a bad admin, software flaw, try and find how did this happen;
  3. After you find out the flaw, search and see if there is a fix to it (server/software update), banning an admin, whatever it is, cause after you fix it, you need to make sure it doesn’t happen again.

Reactive Protection (how to fix the problem)

  • Best way is always, just delete the whole site and bring back the latest stable backup, sure you will lose some content or news but you have a guarantee that your site comes back crisp and clean, fixing it by hand means you can miss something and still keep your site compromised;
  • Make a test run and check if everything is alright, make sure to make the necessary adjustments before bringing the site back online;
  • Fix the security issue, if you found out what was the problem, go ahead and do the updates or workarounds, so this doesn’t happen again;
  • Make a brand new backup immediately before bringing the site back on, this ensures that if the site is still vulnerable, that you can bring it back up quickly, without much loss.

So that’s it, yes i know its basically using backups, and yes there are other ways, but this is the easiest more efficient way to protect your site from premature death ^_^

Brand New Tentakle

Yep new design for tentakle.com, i went with a simple campy look, 50’s horror look hehehe, makes it way nicer to check the awesome imageboard, also added several new boards, lets see how all the uses will like it, so yeah check here for some slimy tentacle hentai pictures ^_^

Timetables and Expectations

Hehehe, yeah i’ve been incredibly busy working on a lot of stuff outside of s2r (personal things, friends sites and was hired to do some other stuff), anyways most of it is done by now, so i’m moving back into s2r mode, still in these last couple of months i’ve merged and moved a lot of my websites around (so i was indeed also busy with s2r), basically i merged some of my shared hosting and one vps into just one “cloud storage” kind of vps, so performance should be way better all through the s2r network of sites (taking some more popular sites out of a vps or shared helps the other sites that remain), sooo what can everyone expect?

 Hummm well first i’m going to cleaned up a lot of code in a lot of sites, then re-start some of my stalled projects, then finish some sites that i’ve been working on… thats pretty much it, so expect panchira.org, hentaish.com, fiendish.org or vidcult.com to have their makeovers and some new stuff to pop up ^_^’

As a side note i’ve released my personal site Hugo Silva (my name…) hehehe, yeah bland, but its more to join my social profiles in one place and later on my cv, thats the whole point of it.

How to Deal with Web Scraping

Hummmm since i have several galleries, one thing i encounter often is web scrapers, personally i don’t mind if anybody takes the entire gallery home or even if they re-post it somewhere else, that is fine with me, this is the web, if i wanted the gallery to be private i would made it so, if its public and free then go right ahead…



However the content itself is not the problem, the problem here is that the vast majority of web scrapers has bad default settings or the users that use them put too aggressive settings, its not uncommon for the load on the server to go from 0.10 to 1 in a heartbeat or even go down, i know its partly my fault i personally like to restrict as little as possible the server or the software (i could use several methods to restrict connections or to ban ip’s if there are too many connections), however because i don’t, sometimes i get into trouble, so this is what i normally do.

First of all i have this on the .htaccces (mod_rewrite), that helps blocking most of scrapping software (unless it spoofs as a browsers hehehe):

RewriteEngine OnRewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]

RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]

RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]

RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]

RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]

RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]

RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]

RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]

RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]

RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]

RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]

RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]

RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]

RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]

RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]

RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]

RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]

RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]

RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]

RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]

RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]

RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]

RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]

RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]

RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]

RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]

RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]

RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]

RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]

RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]

RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]

RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]

RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]

RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]

RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]

RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]

RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]

RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]

RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]

RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]

RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]

RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]

RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]

RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]

RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]

RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]

RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]

RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]

RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]

RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]

RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]

RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]

RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]

RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]

RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]

RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]

RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]

RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]

RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]

RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]

RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]

RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]

RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]

RewriteCond %{HTTP_USER_AGENT} ^Zeus

RewriteRule ^.* – [F,L]

I monitor the load of the server if the load as a spike for more than a couple of minutes i check the apache log, if there are lots of connections to the same site from the same ip, i ban this ip with .htaccess, adding this line:

Order allow,deny
Deny from 100.100.100.100
Allow from all

(100.100.100.100 is the ip on the logs) and check the load after a couple of minutes, if its down fine, if they jumped ip’s, i’ll do one of two things, if they keep on the same ip range i’ll just block that ip range, like so:

Order allow,deny
Deny from 100.100.100.
Allow from all

If they aren’t then i limit the connections to 10 on the cache or image directory (not on all the site), i know it will hurt all users but its better than nothing, just adding this line:

MaxClients 10

If it still persists, i’ll just close the image directory,adding this line again on the cache or image directory (depending on the type of image gallery software you are using):

deny from all

So the site stays up as well as the thumbnails, just the images won’t be accessible for a while, all of these are temporary measures, but for now, for me they do the trick, most of the times banning the ip is enough of a cure, and those i always leave on the .htaccess the other options i normally remove them the next day after the connections storm has passed, bottom line is if you want to scrape instead of bombing the server for an hour, make it so it downloads slowly during a couple of hours it makes a big diference and everyone gets what they want.

Downtime on Server 1

Seems Jaguarpc is having some problems with their network… grrrr this is becoming troublesome, its not the first time in this past few months there have been a couple (comparing with basically years of stability…) and they are always very vague about it, the server is rock solid, but the network performance is not on par, so because of this server 1 is down, so a lot of sites, including s2r.org are down as well ^_^’ lets see how long does this drag…. and if i have to start moving to one of my other providers….

Micro Update

Humm sorry for the lack of updates, kinda going to change how this work, now all hostcult/s2r updates will come from here, and i’ll push the older s2r blog to here hehehe, keeping it simple and nice to everyone, more updates in a couple of days, trying to improve performance on a couple of sites and opening a couple of new ones before August is gone hehehe ^-^