There's seems to be a hellish of a lot of down time on the site at the moment.
There's seems to be a hellish of a lot of down time on the site at the moment.
Nothing to worry about, it's just the gremlins looking for a place to hide, but Lazeez keeps flushing them out.
typo edit
I've been getting a lot of "site refusing connection" or "site not responding" notices over the past week or so, and only from SOL. Today seems the worst so far.
I've been having similar problems since the notice
Tomorrow night Friday morning at 1:00am EDT (05:00 GMT) the site will go offline for a short maintenance. Hopefully will fix the recent reliability problems.
was displayed on 2019-06-27. They're bad enough to make the site unusable around 22:00Z (early evening in the Americas), but the site seems fine around 08:00Z (small hours of the morning in the Americas).
Same here, be nice to get an update from Lazeez on what's going on. Been about a week of refusals, slow, etc. Might be nice to have a site other than SOL to go to for information.
About 4 hours ago it was offline for me for about an hour - I checked with a site checker that said it was offline for everyone for that period but it came back about 3 hours ago and it's worked well for me since after I cleared cookies and got new ones.
Going to echo the others, it's been spotty in the morning/early afternoon hours on the few occasions I've tried to use the site over the past couple of weeks.
DOS attacks are a possibility. There ain't nothing like being a successful good guy to attract bad guys.
DOS attacks are a possibility. There ain't nothing like being a successful good guy to attract bad guys.
especially from Russian site owners upset about getting DMCA notices from their hosting service.
DDOS attacks for about 2 weeks now.
Today it was the worst. I guess they relied on it being canada day and everybody here is off and they stepped up the effort. At one point over a million separate clients were accessing the site. They managed to make the site unresponsive for a good 3 hours between 14:30 GMT and 18:30 GMT.
Anyway, we worked hard instead of partying hard and poured many additional resources into the defense. Costly endeavor, but think we managed to bring everything back online and so far the site has been quite responsive despite the ongoing attack.
Hopefully their resources aren't that extensive otherwise it would become quite expensive to defend against.
Sorry for any inconvenience. I hope you guys understand that it's beyond my immediate control.
Sorry for any inconvenience. I hope you guys understand that it's beyond my immediate control.
Thanks for the update. Dealing with scum is an issue we all have to live with.
Sorry for any inconvenience. I hope you guys understand that it's beyond my immediate control.
I just want to say Thank You for your work on making and keeping this site great. And thank you for working on it while you could have been out partying yesterday, instead. Your efforts are much appreciated.
Sorry for any inconvenience. I hope you guys understand that it's beyond my immediate control.
We understand.
I suggest that you post a notification on the Home page regarding the attack to let SOL members who do not participate in the Forum know what is going on and that they need to expect delays logging into the site and delays in the site responding to their requests until the attacker stops their DDOS activities.
The site is still having issues, but it has gotten better in the last couple of hours.
I know it's not easy to determine the origin, but do you have any idea where they have been originating from?
It's a botnet. They're coming from everywhere.
We had to tighten the firewall and restrict it to two connections per client per second as a security measure. I tried to embed as many resources into each page instead of linking to the resources like javascripts and stylesheets. But due to the restriction in connections, you might get some resources that won't load.
It's a botnet. They're coming from everywhere.
But do you know who's behind it? Is it someone out to get SOL?
But do you know who's behind it? Is it someone out to get SOL?
It's what the word says: a botNET, millions of different connections, not a single source you can point at. It's virtually impossible to find out who is controlling the net and even if you could they are probably in a location where you can't touch them (i.e. Russia, China, etc). Even worse, the controllers of the botnet probably aren't the ones who want to take SOL down. They just sell the resources to 'clients' who want to attack a site or network. It's actually very cheap: Cost of a DDos attack service.
There is a lot of competition for storiesonline for a long time now. There are competitors that pay a lot of money for example to depress our search engine rankings. On average there are spammy links planted all over especially on russian sites. Just last month I had to disavow more than 500 additional russian domains from google's search engine. One domain had over 1.5 million links to storiesonline. Three Russian domains are forwarded to storiesonline for example. It seems that affects our search engine rankings and somebody is doing it.
I put up an ad on google and google received few DMCA take down notices on the ad's landing page.
The site is always being subjected to nasty behavior. From attacks, to search engine manipulation, to content scraping. There are a lot of WTF coming out of my mouth for things that people do affect SOL. And it's been going on for years and years. There is nothing new.
The latest attacks are simply the latest, and so far the biggest attacks. I honestly never thought anybody would put so much resources attacking SOL. There are no demands for anything yet, so for now, the goal seem to be to make SOL seem very unreliable.
Ok, I never heard of DDoS before so what I am suggesting probably makes no sense.
What if you have one server that allows access to the site without logging in and all the other servers require a logged in user. Wouldn't they only be able to attack the single server that is accessible without being logged in? If they send something to the other servers they'd be rejected. Or would the rejections be enough to bring the system down?
Or would the rejections be enough to bring the system down?
Yes.
The time the server spends dealing with, even then to just ignore, the attack requests are what blocks out legitimate requests.
NB- it doesn't actually bring the attacked system down, it just blocks others from being able to connect to it.
restrict it to two connections per client per second
That's odd, you have a single request par page architecture, and the css/js is offloaded to another domain, so despite the fact that browser allow 6 connections per domain, Since you converted to SSL, it's more than likely that legitimate client make only one connection.
If the firewall is shared for all domains, and you did disable ssl on css/js as I observed, then that would explain the need for 3 connections and the partial load, enabling ssl again should have put it back to 2.
But if you are doing this at a firewall, in front of the load balancer, the Side effect would be to cut-off most user using proxies and vpn.
So it's probably a bad mitigation idea.
From the shape of the response I did get during the DDOS, It looks like they managed to overload the bandwidth, from you reaction (disabling ssl on js/css) I suspect the CPU was overloaded too. If your load balancer cache / reverse proxy was any good, that shouldn't have been the case. Unless the DDOS did fuzz the GET requests to bypass the load balancer cache.
Were the DDoS http requests fuzzed?
If the request where fuzzed and If you can't filter that out at the load balancer, then I would suggest ditching Apache for nginx.
But I would also say that Switch Blayde idea make sense, the DDoS http traffic was probably not logged on, so throttling not logged on users at the load balancer make more sense than refusing connection to vpn/proxy users. Giving preferential treatment to legitimate (logged-on) client should be easy.
At least, your hosting service did not blackhole your server IPs to protect their network from the DDOS, So I guess the request traffic was still manageable...
DOS attacks rely on things in the networking stack like number of allowable concurrent connections and stuff like that to overwhelm a site and make it hard for legitimate users to connect. Something like establishing a connection and then waiting to almost the timeout interval to send a single packet, making the servers wait for the clients the maximum time possible and using many, many clients thus reserving networking capacity to the servers while not really interacting with servers in meaningful interactions. So no logged in vs not logged in attacks.
Also, where did you observe additional resources using HTTP instead of HTTPS? If you saw it, I appreciate a pointer as there is supposed to be no mixed media on the site and all should be going through https. HTTP is not an attack mitigation strategy.
For now it seems that things are fairly under control and the sites are accessible despite there being more than 5 folds the normal networking traffic.
http vs https seen for page storiesonline.net/a/aroslav/2
on the 1/7 around 18h GMT
2 ressources were referenced as http 1 js, 1 css.
Since then, the js was removed and the css is back on https.
waiting to almost the timeout
https://en.wikipedia.org/wiki/Slowloris_(computer_security)
Really that much dificulty handling a Slowloris attack?! That's hilarious (sorry).
If that was really the case, you really should switch to nginx, it is immune to slowloris.
Are you really really sure you correctly identified the attack?
I should point out that the server response packets where spread out over 100sec for a 28kb response.
To me it looks more like the Slowloris "symptoms" are a side effect of bandwidth starvation.
A slowloris attack would give me a timeout (I experienced that a lot), but would not give a spread out response like I observed.
The attackers are using multiple types of attacks. As you can see from the response time now, we're doing well despite having a huge number of clients trying to connect to the site. The firewall has dropped over 100,000 connections in the last 2 hours.
As for bandwidth, the site has 10 GB line, and at no point it was anywhere near 10% of bandwidth usage. It's not bandwidth starvation.
We're doing ok for now. Hopefully they get bored soon.
As for the http/https thing, we haven't changed any of those settings for like a year now. Are you sure it's not your browser that's trying to fall back to http just in case when it's not getting the resources from https?
As for bandwidth, the site has 10 GB line, and at no point it was anywhere near 10% of bandwidth usage. It's not bandwidth starvation.
I'm assuming (and I'm not expecting you to give any details of any kind) that you've prepared for a reflection attack as well. A botnet with each compromised system doing a reflection attack could drown that 10GB connection.
Have you thought about using the existing Twitter feed, or a new one, to provide status updates when things aren't working? I'm assuming you have some kind of automated monitoring to let you know when things to awry, could it post an update?
Have you thought about using the existing Twitter feed, or a new one, to provide status updates when things aren't working? I'm assuming you have some kind of automated monitoring to let you know when things to awry, could it post an update?
Not twitter or any other social media. Some of us try to desperately avoid that crap. Or at least also a status somewhere else. Storiesonline.org was also slow but still reachable. We could have posted a status message on ReaderInfo.net but the site isn't very well known yet and we didn't known what was happening.
Some kind of status message would have been nice though. We could have patiently waited it out without getting frustrated with failing connections and not knowing if it was our own connection or on the SOL side.
and not knowing if it was our own connection or on the SOL side.
That one was easy. Other sites were behaving normally, in particular other sites in the US / Canada.
This Forum replaced a Newsgroup - https://groups.google.com/forum/#!forum/storiesonline - almost 4 years ago. While the Forum is in many ways far superior to what came before it, it is next to useless when the site has serious problems.
not knowing if it was our own connection
There are sites to tell you about website status, e.g.:
https://www.uptrends.com/tools/uptime
https://www.isitdownrightnow.com/storiesonline.net.html
and, trying to connect to Google and/or some other high availability sites will give good indications as to where problems lie. Don't overuse the 'is it up' tools as that's pretty much the same as what the dDoS attacks are doing.
Yes I know that you can see if other sites remain accessible so you know it's not your connection but it wouldn't be the first time that I had problems accessing a site because of other problems.
Sites that check if a site is available are not always reliable. They either depends on multiple reports from users or try a few times and base their report on that. They say nothing about the why and the how long.
Not twitter or any other social media. Some of us try to desperately avoid that crap.
I tried to get the members on my Yahoo Group to switch to Twitter. Not one did.
I tried to get the members on my Yahoo Group to switch to Twitter. Not one did.
If you want a like-for-like replacement for Yahoo Groups, try https://groups.io/
I can't remember where I found out about it - it will either have been here or in dotB's group - but I suggested it to the owner of another group and it works exactly as it should. The coding is pretty much the original YG coding, before Yahoo started messing around with it. I believe there are migration tools available, the migrated group has several hundred members and I'm sure the owner did not enter them by hand.
Not twitter or any other social media. Some of us try to desperately avoid that crap.
FWIW, you don't have to have a Twitter account (I don't and won't) to see the StoriesOnline feed. Makes it easy to pop over and check to see if an update has been posted.
I post notes when something is happening and I think about it and have enough time to do it instead of dealing with what's going on.
Nothing automated yet. But then again, you don't want your attacker to know exactly when their attack starts working enough to trigger automated tools.
Nothing automated yet. But then again, you don't want your attacker to know exactly when their attack starts working enough to trigger automated tools.
That assumes that your attacker knows where you post your status updates, but I understand your thoughts.
That assumes that your attacker knows where you post your status updates,
Please, explain how you would conceal knowledge of such a status update to a net savvy botnet owner in such a way that they can't see it. BUT It is plainly visible and sufficiently flagged that EVERY legitimate member/user can find it unaided..??
I'm sure many interested parties await your response...
ps
If you own a botnet, please read no further. Thanks.
Please, explain how you would conceal knowledge of such a status update to a net savvy botnet owner in such a way that they can't see it
You create one with a random name and only publish it to logged in users. That way they would need to be a registered user to get the information. Not guaranteed, but will avoid script kiddies
You create one with a random name and only publish it to logged in users.
If you cared enough to instigate an attack against a site which allowed free membership, you wouldn't obtain one?
Not guaranteed,
Naivety is cute, but not effective.
There was a weird side effect of this recent episode for me. I was unable even to find the site for the entire day yesterday (7/1). In addition, I found about 25 stories were sent to my Kindle. Some I had downloaded in the past; others I hadn't. Like I said, weird.
Nothing automated yet. But then again, you don't want your attacker to know exactly when their attack starts working enough to trigger automated tools.
Personally I'd much prefer there to be no public recognition of these attacks by the site/yourself. As you say, real time responses can be useful to an attacker, similarly, public acknowledgement is like awarding them brownie points. It only adds to the entries on their CV's.
Every so often a notification on the main page warns of planned downtime, date, times and likely duration. If the site is slow or unreachable outside those notified times, guess what..?? Yup, try again later. Simple.
As nice as instant personalised warnings to members in real time of each attack, are we not a little more adult than the "me now", "instant gratification" generation??
My 2c
Well, you could use the anti-ddos service Cloudflare. Unless... cloudflare has an outage, like today: Cloudflare Status. Lots of 502 errors but they seem to have it under control by now.
Oh, and back to this site's problems - does it use Cloudflare? According to Slashdot they have been experiencing problems today.
https://www.businessinsider.com/cloudflare-outage-causes-major-websites-across-internet-to-go-down-2019-7
There has also been a problem with one of the security firms for the net.
https://www.bbc.co.uk/news/technology-48841815