DDOS? Attack? Bug? I dunno.

[ledow]

Hi,

Just checked on my server when it was unresponsive to find the following lines in my log and the process in an infinite loop:

INFO [2011-01-19 11:35:09,581] [Thread-6]: Adding client 'Faulty1' from 173.89.242.1:10585, playerId=6
ERROR [2011-01-19 11:35:09,581] [Thread-6]: java.lang.NullPointerException
java.lang.NullPointerException
        at PB.a(SourceFile:335)
        at IF.a(SourceFile:137)
        at IF.a(SourceFile:108)
        at Nw.a(SourceFile:157)
        at Bl.a(SourceFile:57)
        at Lk.a(SourceFile:39)
        at Bl.a_(SourceFile:38)
        at HO.a(SourceFile:427)
        at zV.a(SourceFile:114)
        at zV.a(SourceFile:79)
        at Xo.a(SourceFile:102)
        at HO.c(SourceFile:415)
        at HO.d(SourceFile:351)
        at HO.e(SourceFile:219)
        at CK.s(SourceFile:237)
        at CK.run(SourceFile:260)
        at java.lang.Thread.run(Unknown Source)


ERROR [2011-01-19 11:38:49,249] [Thread-6]: java.io.IOException: Operation not permitted
java.io.IOException: Operation not permitted
        at java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.net.DatagramSocket.send(Unknown Source)
        at ku.a(SourceFile:205)
        at ku.a(SourceFile:177)
        at LK.a(SourceFile:18)
        at LK.a(SourceFile:30)
        at HO.d(SourceFile:199)
        at CK.s(SourceFile:245)
        at CK.run(SourceFile:260)
        at java.lang.Thread.run(Unknown Source)

The IP doesn't correspond to any of my users historically, and the server was in death from a sustained hit of joins / NULLPointerExceptions / etc. from the (non-vapor-id?) IP in question. It looks like it's been spamming my logs with those three errors for the last few hours.

I blocked the IP on the firewall but the machine needed a reboot to get back to operation (looks like Java just "hung" thousands of ports and kept throwing the IOExceptions).

I'm currently hunting through the logs for anything suspicious - altitude was only running as an "altitude" user but I want to make sure nothing else was tampered with, because NULLPointerExceptions can be dangerous. Is this a bug in java, Altitude, an attack or just paranoia?

[ledow]

Weirdly, I still get these errors with the same data even when that IP is totally blocked from the firewall (in / out / forward, udp / tcp, source AND dest). And I still get them if I use a (clean, unmodified, previously-tested) 64-bit java that hasn't been touched (so not subject to java updates, etc.).

[ledow]

Restored the server from a backup from before all this started - same problem.
Tried switching JVM's and updating them - same problem.
Disk checks for corruption come back all clear - same problem.

Anyone suggest something? I can't get an Altitude server to stay up past one of these "join's".

[ledow]

Reinstalled the Altitude server from the latest server installer into a clean folder - same problem.

[Boko]

Do these joins happen straight after you start up the server? Or at random periods of time after your started them?

[ledow]

Random - I think it's linked to when ANYONE tries to join, but it's a busy server normally so as soon as it "comes back online", it's only a second or two before someone tries to join.

I think there's some kind of bug where ANYONE joining is seen as "faulty1" on that IP and then I get the NullPointerException from there. Until someone joins though (i.e. if I hide it behind a firewall), it doesn't happen.

[lamster]

This looks like a network issue to me: the server is unable to bind (or maybe just send) data on the UDP port. Have you modified your firewall/security settings recently? Have you checked for a "zombie" process (e.g. your Altitude server) that has the desired UDP port bound and refuses to release it? Sometimes when people set up scripts to auto-launch their server they'll run into a situation that launches it twice and the zombie has the UDP socket reserved while the process they notice runs normally but is unable to communicate.

[ledow]

Originally Posted by lamster

the server is unable to bind (or maybe just send) data on the UDP port

The server updates perfectly, and gets into the master list. Also, ALL people joining the server appear as that "faulty1" with that IP address - I never see any other name / vaporid / IP try to join now (previously they would obviously appear in the logs as the correct name / IP).

Originally Posted by lamster

Have you modified your firewall/security settings recently?

No changes AT ALL have been made to this server beforehand. It's been running for 30+ days without any configuration changes whatsoever (I only logged into it because it *wasn't* working for altitude, otherwise I never have to touch it). The problem started at about 6:30 this morning and is ongoing despite a complete (block-level) restoration of the server from both a day-old and a week-old backup.

There are no connectivity problems that I can see (e.g. It contacts the master lists properly, I downloaded and installed a fresh Altitude install and it does the same, and I updated an old copy of the altitude folder by running it -a known-good working backup of the altitude folder from several months ago - it updated normally and then produces the same problem).

To my knowledge there are zero changes to the networking infrastructure around the server.

Originally Posted by lamster

Have you checked for a "zombie" process (e.g. your Altitude server) that has the desired UDP port bound and refuses to release it?

The only services running on the machine are SVN and altitude. Everything else is switched off from init. Only SVN starts at boot. Altitude is manually run on each reboot from SSH. There are no rogue processes, nothing unusual, nothing binding those ports and the server has been rebooted and restored from backup several times. SVN operates perfectly (but I know that's not much of a help).

Originally Posted by lamster

Sometimes when people set up scripts to auto-launch their server they'll run into a situation that launches it twice and the zombie has the UDP socket reserved while the process they notice runs normally but is unable to communicate.

I've run it from my normal script, I've run in manually (server_launcher), I've done "killall -SIGTERM java" beforehand and I've done it from a fresh reboot (altitude isn't run on startup at all, I do it manually from the command line). When java isn't running, nothing is bound on that port or trying to bind that port. Java (and server_launcher) isn't running before I start and starts normally in all other respects, and doesn't linger afterwards (constant restarts on each NullPointerException but server_launcher just respawns it).

[ledow]

Just did a brief experiment with the only firewall (iptables on the local machine) completely disabled (ACCEPT ALL for all tables) and all other processes killed - same problem.

[lamster]

Hmm -- very weird. There is in fact a user named "Faulty1" logged in from that IP address -- he has been online for about a day -- perhaps his client is hung in some sort of buggy state that involves continually pinging your IP with server-breaking packets. Are you certain that you've configured your firewall to reject all traffic (in particular UDP) from that IP address? I would expect that to fix the problem, assuming it's really the packets from Faulty1 and not something else going on. The NullPointerExceptions are nothing to worry about from a security perspective -- the server is a java process and those exceptions are being caught within the VM; nothing has hinted at violating the sandbox -- that said, they do generally indicate some sort of problem in the game code.

[ledow]

With the IP blocked, I get NullPointerExceptions and other IOException errors which sometimes crash the server. With the IP not blocked, I only get NullPointerExceptions and the server still runs (i.e. people can play) but I still get flooded logs with constant connections from that address.

I'm talking to my upstream providers to try to stop those packets making it to the machine at all but, yes, I did blacklist the address for a while - something still tries to communicate to that IP though, even with that address blacklisted in the firewall (first rule of iptables, in every table, deny all, from or to that address, on any interface!).

I'm assuming something in the protocol is trying to get my server to hole-punch a way to him and failing? I have no idea. At the moment, he's constantly connecting (and being timed out after 20,000 milliseconds or whatever) and I'm having to let it through or the java process is unstable.

[ledow]

First, the problem was fixed by the previous update. Thanks lamster, whether that was the update itself or just some tweak while the update was going out, your response was swift, precise and you were obviously digging through code to try to find the problem.

Secondly, you would really need to justify any accusation of unreasonable banning.

The server is paid for, by me, one person with a not-fantastic income and a not-fantastic skill at the game, so that I can have a game away from idiots that I can't get rid of on other servers. I'm only EVER on when I want to play the game - if I just got an ego trip out of banning people, I could sit in spectate mode all the time and ban people for anything I liked, anyway. Altitude is my leisure time and I pay for it to be perfect so that I don't HAVE to deal with other people. The only rule of the server is "Play the game, don't stop others doing the same".

I pay for that server JUST so that I can play a game of football each evening for about an hour or so, have a guaranteed spot on the server and not have to tolerate idiots while doing so. The maintenance and financial burden is worth it. Other people being able to play when I'm not online is an added bonus that I don't mind paying for because it adds a sense of community and I see lots of "regulars" that I know how to play with.

But occasionally, I get immature people who are basically there to cause annoyance. Whether it's repeatedly spamming the chat section with junk, cussing every person on the server who misses a pass (I'm a crap player myself, my ratios ALL hover about 1.3, but I just enjoy it, and my server is "beginner-friendly", so you can mock but to go further than that will see me clamp down on you), deliberately throwing the game so their mates can score goals, or even just being an social-arse by using vastly unacceptable insults to strangers. Anything that interferes with the game is *NOT* good for me and spoils the point of me paying for my own server.

I'm not bothered about swearing at all - you'll see that if you've ever played with me - but there are a number of people who have taken things to extremes. Several players get to the point where I've been asked to "do something about them", via whispers, by other players who are regular, polite, and playing the game. And sometimes people are just bugging me to the point that I can't play my game. That leaves me with little option but to act to preserve the fun element of the game for the rest of the players. I'd rather "lose" one annoying player who's never online than the regular players who obviously *just* want to play and respect the server "rule". Ideal server for me? Hundreds of people who just play the game properly and chat in a fun but mature manner and me NEVER having to open the console even once. About 95% of the time that's exactly what happens.

So, if someone does something and it starts distracting me from the game and/or others ask me to act (which has the same distracting effect), it comes time to act. First, always, comes a warning not to do what they are doing. This might be as subtle as "Don't spam the chat, please" or as obvious as "Stop doing that or you're out". About 25% of the time, that's the last I see of the problem and we carry on playing. About 25% of the time the person in question gets into an apology loop because they think sucking up to the admin is the thing to do (which can be just as annoying but I tend to just leave them be - at least they are apologising).

The other 50% of the time the person in question is so stupid that they say something like "F*** you" or similar, whether in public chat or whisper. Rarely do these people realise I'm the admin despite my server having my name in the title, but even sometimes they *DO* realise I'm admin and still respond to me in that fashion.

So they've done something I don't want to have to deal with, to the point that I've *had* to deal with it, then not heeded a warning (which probably *WON'T* be polite but, hell, it's a warning) and/or escalated the situation. I have three choices: ignore, kick or ban. I don't ignore that because it's already interfered with my game and therefore "cost" me money.

A kick *inevitably* brings those people back as soon as they can to cause more trouble or argue about it on the game itself (disturbing others). I just want to play. The fact I had to bring down the console to even type something has distracted me from my game, and that's spoiling it for me.

So if a player has got to this point, then I ALWAYS ban for life. There are other European football servers. They won't suffer. I don't *want* to spend hours judging whether that person made an "innocent mistake" of doing something, ignoring my and others warnings and then propagating that through to a ban, or working out whether they are "okay now" to come back (I don't need the power-trip of being god and having people sucking up to come back - once you're out, that's the end of it). If they're stupid enough to do it once, I don't want them playing when I'm not online and annoying others, or coming back to do the same again at a later date. So it's a permanent ban (which shows as 20 years on Altitude) every time.

About 70% of the time they then want to add me (ironically) as a friend and talk about it, or come on the forums and whinge about it. I don't care. At that point, nothing they say will make me reverse a ban. I've been running popular gameservers constantly since the Doom days and nothing has ever made me reverse a ban. I tend to ignore such attempts at contact because, again, they are just distracting me from the game and it's highly unlikely I'll want to have that conversation. I don't really care if they agree with my reasoning or not. One user even hounded me through the forums via private messaging for two weeks after I banned them - the forum admins said I should have reported it but it was easier to just ignore - I don't want the hassle, and I don't want them sanctioned globally, I just want to be left alone personally by that person.

Currently, since I started in August last year, there have been 11 bans on Ledow's Football Server. There was also one temporary ban, but that was lifted after ten minutes and was for something I didn't deem it disruptive enough to justify a full ban and it was touch-and-go if they were playing for a reaction or just being mature but funny.

Not counting that, that's one ban per 14 days of server operation. Considering the server is full every evening and is often full during the day too, that's about one ban per several hundred unique players.

Looking through my banlist (which has reasons listed for bans), I have one banned for using the word "n***er" in a completely out-of-context, insulting way. Two for various "F*** you's" when they were warned about pathetically trivial things. Two are actually for having a consistently stupid ping and being constantly (20+ times) ping-kicked, to stop them hogging a player slot.

One was brought to my attention by a regular as soon as I came online one day and had been throwing games when I wasn't around - the logs seemed to confirm that, and the regular was trusted, so it just went down as an instant ban. Two more were banned for gameplay violations (deliberate passing to enemy, etc. while they were high-Ace rankings so no excuse of not knowing what they were doing). One was banned for arguing over team balancing after I'd already brought up two successful public votes to balance the teams (one of which I voted FOR the idea even if I thought it was unnecessary, personally, but just for the sake of harmony) - ten minutes later they were still arguing about my techniques and for not "making the server fair". That one went on for so long in-game, though, and with such persistence after I tried to just say "Okay, fine, let's just play" that I had to get rid of them.

The other two have their full reasons missing (Just "Because" because I obviously wanted them out quickly and didn't want to bother with a full reasoning because the ban command takes long enough to type correctly as it is) but were probably just idiots on chat. There's a point at which players telling other people they are useless turns into venomous statements and spoils the game. None of those bans do I consider unreasonable - you may disagree but, to be honest, I don't really care. When you're paying the bills on a server that I have a guaranteed space on and on which I always play the game properly with a full crowd of others doing the same, then you can complain.

Currently, I pull in 50Gb a month in traffic just for altitude. That's a LOT of traffic for a little game server. I can probably name 20-30 regulars that I see *every* time I go on if I stay on long enough. I have about 10-20 more added to friends so that they can contact me direct when I come online if there's a problem with the game / server / players. None of them have complained about me being heavy-handed and most of them can see my bans "brewing" long before I do so, mainly because it takes some perseverance on the player's part, and ignoring warnings, before they actually get banned.

And, in the end, I don't really care if you think that's heavy-handed but I actually *like* to advertise my ban policy because it is good advertisement for those people who DO just want to play a game. Many people find it very reasonable and just the sort of thing they are after when they look for a quick game. If that's not for you, there are other servers. But equally, if you want a nice quiet, friendly game without any hassle, you're more than welcome to come and try it. Dozens of people find my server rules so draconian and unacceptable that they are there every night. You're welcome to join them. :-)

[ledow]

Originally Posted by lamster

The NullPointerExceptions are nothing to worry about from a security perspective -- the server is a java process and those exceptions are being caught within the VM; nothing has hinted at violating the sandbox -- that said, they do generally indicate some sort of problem in the game code.

I'm a C programmer - I don't trust anything that throws Null around! :-) And though I believe it's Java and safely isolated, and that I trust it wasn't deliberate, and though I know it's running in its own secured environment running as its own user, with minimal permissions, on a pretty unimportant server, with a tight firewall stopping it doing things it shouldn't, I still don't like it doing it. About the only thing you'd get from my server even with root access would be 50 files of crappy C code for a game I'm writing.

But it only takes a Java VM bug (especially with the "old" bundled versions of Java inside the altitude folder), or or some sort of code compromise that allows it to run arbitrary Java code for it to cause havoc on a server.

As it turned out, there was nothing suspicious in the end. I still restored from a backup just-in-case, however I fully appreciate that there's nothing to be worried about in this particular case, and you seemed to have fixed the problem, and it pays to be suspicious of everything, anyway.

Altitude downloads updates through its own private mechanism, which also could potentially involve its start-up scripts, executables, or updates to its own "jre" folder containing the java executable used to run the game (in a standard config, anyway). In theory, you (or someone that compromises the program) could make it receive a fake "update" that changes the server_launcher script to "format c:" (or OS-equivalent) or anything else.

Any sort of "hijack" of the program could be dangerous, potentially, but I trust the team enough that it's pretty harmless so long as you're responding to bug-reports of this nature and that it won't be misused by yourselves.

But don't say "Null Pointer" to a C-programmer. They're likely to start gibbering and shaking. :-)

[lamster]

Originally Posted by ledow

I'm a C programmer - I don't trust anything that throws Null around! :-)

I understand Let me know if you run into any new issues. As a side note, congrats and thanks for running one of the most popular servers!

[ledow]

Hi Lamster,

Still getting odd things - see below.

INFO [2011-01-27 14:53:23,527] [Thread-7]: Adding client 'Unautherized Account' from 71.192.52.250:27279, playerId=6
ERROR [2011-01-27 14:53:23,527] [Thread-7]: java.lang.NullPointerException
java.lang.NullPointerException
        at SB.a(SourceFile:335)
        at il.a(SourceFile:137)
        at il.a(SourceFile:108)
        at DS.a(SourceFile:157)
        at Ts.a(SourceFile:57)
        at jG.a(SourceFile:39)
        at Ts.a(SourceFile:38)
        at WQ.a(SourceFile:427)
        at aN.a(SourceFile:114)
        at aN.a(SourceFile:79)
        at HR.a(SourceFile:102)
        at HR.a(SourceFile:140)
        at WQ.c(SourceFile:415)
        at WQ.d(SourceFile:351)
        at WQ.e(SourceFile:219)
        at fk.s(SourceFile:237)
        at fk.run(SourceFile:260)
        at java.lang.Thread.run(Unknown Source)
INFO [2011-01-27 14:53:43,532] [Thread-7]: Unresponsive client Client[name=Unautherized Account, ip=71.192.52.250:27279, playerId=6]; 20006.0 milliseconds since keep-alive received
INFO [2011-01-27 14:53:43,533] [Thread-7]: Removing client 'Unautherized Account' from 71.192.52.250:27279, playerId=6, message: Connection to game server timed out

Also, if I search logs for that IP, I see a couple of joins with valid vaporid's in the past but nothing suspicious.

[SnipErlite]

Originally Posted by ledow

The server is paid for, by me, one person with a not-fantastic income and a not-fantastic skill at the game, so that I can have a game away from idiots that I can't get rid of on other servers.

Just read through this, and I just wanted to say thanks for the running of your servers. They're good fun to play on, good latency wise and you're a good mod. Thank you (Y)

Batman

[Boko]

**** just got necro'd.

He's a good mod though, he banned me.

[SnipErlite]

Originally Posted by Boko

**** just got necro'd.

He's a good mod though, he banned me.

Hey it was only about 6 months...

>.>

<.<