Server throwing everyone out

[Cracker]

When the co-op and ball server are both (nearly) full the server sometimes throws everyone out with these messages:

[port 27276] [11 nov 2012 21:03:09,104] INFO: Unresponsive client Client[name=, ip=, playerId=3]; 151 updates written without response
[port 27276] [11 nov 2012 21:03:09,104] INFO: Unresponsive client Client[name=, ip=, playerId=11]; 151 updates written without response
[port 27276] [11 nov 2012 21:03:09,104] INFO: Unresponsive client Client[name=, ip=, playerId=6]; 151 updates written without response
[port 27276] [11 nov 2012 21:03:09,105] INFO: Removing client 'name' from ip, playerId=8, message: Connection to game server timed out
[port 27276] [11 nov 2012 21:03:09,105] INFO: Removing client 'name' from ip, playerId=11, message: Connection to game server timed out

and so on for every player in there (names and ips taken out)

When everyone is back in it gives a lot of warnings..:

[11 nov 2012 21:03:10,801] INFO: [Data Transfer Since Launch] Outgoing: 1,071 GB, Incoming: 0,471 GB
[11 nov 2012 21:03:11,005] INFO: Update check result: LOCAL_VERSION_IS_UP_TO_DATE, free memory: 50,5 / 123,8
[port 27280] [11 nov 2012 21:03:13,020] WARN: value -389 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,020] WARN: value -120 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,020] WARN: value -329 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,105] WARN: value -389 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,105] WARN: value -120 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,105] WARN: value -329 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,219] WARN: value -390 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,219] WARN: value -121 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,219] WARN: value -329 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,514] WARN: value -391 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,514] WARN: value -122 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,514] WARN: value -330 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,618] WARN: value -391 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,618] WARN: value -122 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,618] WARN: value -331 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,872] WARN: value -392 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,872] WARN: value -123 is less than min(-50)
[port 27280] [11 nov 2012 21:03:13,872] WARN: value -332 is less than min(-50)
[port 27276] [11 nov 2012 21:03:14,089] INFO: assigning Tjerki to left_team
[port 27280] [11 nov 2012 21:03:14,525] WARN: value -394 is less than min(-50)
[port 27280] [11 nov 2012 21:03:14,525] WARN: value -125 is less than min(-50)
[port 27280] [11 nov 2012 21:03:14,525] WARN: value -334 is less than min(-50)
[port 27280] [11 nov 2012 21:03:14,985] WARN: value -396 is less than min(-50)
[port 27280] [11 nov 2012 21:03:14,985] WARN: value -127 is less than min(-50)
[port 27280] [11 nov 2012 21:03:14,985] WARN: value -336 is less than min(-50)

The servers are run on a dedicated server hosted at my friends home (a 100mbit connection, so shouldnt be a problem?). Any solutions/things we can do to stop this? Could a particular coop map be causing this? Or is it just overcapacity and therefore one of the servers we run should be taken down?

Thnx in advance, Crack

[sukosevato]

Guess we'll have to up the memory for the servers then.

Atm they run on only 128 mb. I'll increase that to 512 mb.

[Cracker]

O hai suko :P Ya I just wanted to make sure that there aren't particular coop maps causing this (because the warn values are errors in maps I believe, but wasnt sure >.>)

[sukosevato]

I've upped it to 512 mb already a few hours ago.

If it happens when it's nearly full it's likely that it's memory shortage. If it still happens it might indeed be a map. May be worth it to write down on which map it happens, see if you recognize a pattern.

[sukosevato]

14:55:00,202] [Thread-6]: Update check result: LOCAL_VERSION_IS_UP_TO_DATE, free memory: 25,9 / 111,6 MB
INFO [2012-11-12 14:55:21,590] [Thread-19]: Handling join request from ip -> mb@ed486dd7<vaporId=77120648-f201-4a06-a0aa-d34cc8f1f2ba, nickName=EshwarRoxX, loggedIn=true, ip=ip>
INFO [2012-11-12 14:55:21,707] [Thread-19]: Adding client 'EshwarRoxX' from ip, playerId=3
INFO [2012-11-12 14:56:01,300] [Thread-16]: Handling join request from ip -> mb@52d4b243<vaporId=9fb96700-2a8d-4184-9b2b-14867ccb8026, nickName=AlexLittle907, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:09,529] [Thread-16]: Handling join request from ip -> mb@ed486dd7<vaporId=77120648-f201-4a06-a0aa-d34cc8f1f2ba, nickName=EshwarRoxX, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:10,958] [Thread-19]: Handling join request from ip -> mb@ed975065<vaporId=c7588291-3bad-4a16-81d5-b68e90b72e4f, nickName=Liquid`Meat, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:11,078] [Thread-19]: Adding client 'Liquid`Meat' from ip, playerId=4
INFO [2012-11-12 14:56:12,133] [Thread-19]: DLFMHS caching file data for 1de_coop_test.altx
INFO [2012-11-12 14:56:12,133] [Thread-19]: Got initiate download request from ip
INFO [2012-11-12 14:56:12,254] [Thread-19]: DLFMHS sending send-complete to ip
WARN [2012-11-12 14:56:16,083] [Thread-19]: value 5.2550896E7 is greater than max(3276.0)
WARN [2012-11-12 14:56:16,185] [Thread-19]: value 7.6115808E7 is greater than max(3276.0)
WARN [2012-11-12 14:56:16,321] [Thread-19]: value 2.8336952E7 is greater than max(3276.0)
INFO [2012-11-12 14:56:16,732] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:17,686] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:17,686] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:22,183] [Thread-19]: DLFMHS releasing file data for 1de_coop_test.altx
INFO [2012-11-12 14:56:27,642] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:37,597] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:47,581] [Thread-16]: Handling join request from ip-> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:53,781] [Thread-19]: Server got disconnect request from Client[name=Liquid`Meat, ip=95.96.37.138:27272, playerId=4]
INFO [2012-11-12 14:56:53,781] [Thread-19]: Removing client 'Liquid`Meat' from ip, playerId=4, message: null
INFO [2012-11-12 14:56:54,952] [Thread-19]: Handling join request from ip -> mb@ed975065<vaporId=c7588291-3bad-4a16-81d5-b68e90b72e4f, nickName=Liquid`Meat, loggedIn=true, ip=ip>
INFO [2012-11-12 14:56:55,069] [Thread-19]: Adding client 'Liquid`Meat' from ip, playerId=5
INFO [2012-11-12 14:56:57,549] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:57:05,210] [Thread-6]: [Data Transfer Since Launch] Outgoing: 0,167 GB, Incoming: 0,076 GB
INFO [2012-11-12 14:57:05,414] [Thread-6]: Update check result: LOCAL_VERSION_IS_UP_TO_DATE, free memory: 11,2 / 111,6 MB
INFO [2012-11-12 14:57:07,521] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:57:17,513] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
WARN [2012-11-12 14:57:19,850] [Thread-19]: value 9.9935488E7 is greater than max(3276.0)
INFO [2012-11-12 14:57:27,499] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:57:37,475] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:57:47,416] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:57:55,445] [Thread-19]: Handling join request from ip -> mb@78f5c426<vaporId=38e96de9-502e-4f73-99a1-c239899324a4, nickName=Max<>, loggedIn=true, ip=ip>
INFO [2012-11-12 14:57:55,563] [Thread-19]: Adding client 'Max<>' from ip, playerId=4
INFO [2012-11-12 14:57:57,369] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:57:59,397] [Thread-19]: DLFMHS caching file data for 1de_coop_test.altx
INFO [2012-11-12 14:57:59,397] [Thread-19]: Got initiate download request from ip
INFO [2012-11-12 14:57:59,515] [Thread-19]: DLFMHS sending send-complete to ip
INFO [2012-11-12 14:58:07,358] [Thread-16]: Handling join request from ip -> mb@96108336<vaporId=3cb14b4c-4ba0-473c-97ba-4d6676bbc201, nickName=Step0_SVK, loggedIn=true, ip=ip>
INFO [2012-11-12 14:58:09,451] [Thread-19]: DLFMHS releasing file data for 1de_coop_test.altx
WARN [2012-11-12 14:58:12,734] [Thread-16]: Server hitch detected: 405.0 milliseconds, free memory is 69,6 / 111,6 MB
WARN [2012-11-12 14:58:12,734] [Thread-13]: Server hitch detected: 406.0 milliseconds, free memory is 69,6 / 111,6 MB
WARN [2012-11-12 14:58:12,736] [Thread-19]: Server hitch detected: 405.0 milliseconds, free memory is 69,4 / 111,6 MB
WARN [2012-11-12 14:58:12,745] [Thread-10]: Server hitch detected: 413.0 milliseconds, free memory is 69,4 / 111,6 MB


Pretty odd. Server slowly fills with players. And once it starts to run low on memory it hitches and suddenly has a lot more free memory.

I've not seen it go past 111,6 MB either. Even though i've made the following file:

server_launcher.vmoptions (in the same folder as the server_launcher.exe)
-Xmx512m (this is inside it)

I've then relaunched the server, but it's not really taking advantage of the extra memory and sticks at 111,6 MB and when it gets close to it it starts hitching / spewing these errors:

WARN [2012-11-12 14:57:19,850] [Thread-19]: value 9.9935488E7 is greater than max(3276.0)

Any idea what that 3276.0 max is? Is it the memory which it runs close to? These errors disappear after the server hitch when the free memory jumps back to 50-60 mb. And then after it slowly fills up again you'll see these errors appear again till the next hitch.

It's a windows machine. It's not network related. I've had a ping 8.8.8.8 -t running and no hitch in that occurred. It's also not a resource problem (except for the altitude process not using more then 111,6 MB. Since there is 2 GB memory left and cpu is doing nothing at about 5-7%).

Any ideas?

[sukosevato]

Made the server start with more memory by default.

Since it very slowly was giving it self extra memory. It got to about 138 MB. but every time that happened it would hitch.

So i've let it start with 256 mb standard now. See if this resolves the hitches.

Will report back for those that want to know.

[sukosevato]

I'm still getting the same errors, even though it has plenty of memory now:

http://imgur.com/aXMVV

As you can see 94,5 memory of 246,5 free.

There are about 17-19 players connected / playing while this happens.

What does this 'greater than max (3276.0)' refer to?

[lamster]

Those 32760 printouts shouldn't be causing any issues. If you're not already using both the startup and max heap flags you could try that:
-Xms512m
-Xmx512m
Those shouldn't make a difference here though -- I'm guessing the issue is either too many bots (try numberOfBots <= 2) or another process on the box periodically interrupting/stalling the server process (security/anti-virus software, etc).

[sukosevato]

Ok if those errors don't matter i'll ignore them.

I've done some more trouble shooting and made some additional changes.

Changed the Virtual Machine relative weight to the others. It's 5x more important when fighting for cpu availability. Though the quad core never runs on more then 25% and rarely a peak to 50%. So i doubt that'll change anything. Same for memory availability. Though it has 4 gb reserved for it and it uses about 2 gb of that.

I've started the altitude servers with the xms / xmx tags but now with both at 512m. Since i've noticed that if you start it with 512 mb, but it doesn't need it, it'll still only use about 200 mb in task manager. So that shouldn't be a problem either.

I've still noticed an occasional hitch however. Been doing pings on the hyper V / game server / other virtual machines to see if those latency spikes increase at all of them at about the same time. Which is true, so i've decided to set up some network QoS. Even though the 100/100 mbit fiber link shouldn't be close to being saturated yet.

The port range of altitude now has the highest QoS.
Meanwhile i've configured the torrenting on a different VM to use a custom port range vs random and that port range now has the lowest QoS.

I also increased the size of the log files from 100 kb max to 4000 kb. Seeing as I would only see a few hitches in a 100 kb log before the older entries started getting deleted again. Maybe i'll be able to recognize a pattern of when those hitches occur if i have a larger sample size.

Full crashes where everybody would get disconnected if the servers were nearly full haven't occured since i upped the memory from 128 mb though. Only a few hitches left.

The server does not run any kind of AV. So that can't be interfering either. It does run the windows firewall, but forwarding and everything is set up properly.

I haven't been able to detect any problems in the windows log files around the spikes.

The server however does not have Java installed or anything. While if i'm correct altitude runs on Java, might it run more efficiently if I install a java runtime environment? Maybe the memory management of the server will perform better?

Will report back with what i find.

[Karl]

Altitude ships with Java 1.6.0_16. It's possible (but very unlikely) that using a newer version of Java and the G1 garbage collector could reduce or eliminate your pauses.

http://www.theserverside.com/news/th...hread_id=54321

Your mileage may vary and changing Java versions in Altitude is a bit annoying. Wouldn't recommend it unless you're desperate.

[lamster]

Noticed you mentioned torrenting -- even if you've limited the torrent upstream/downstream bandwidth to avoid network congestion, you may see the occasional massive I/O-related performance stutter when a download completes or when a dormant file is accessed for the first time.

[sukosevato]

The completion of torrents can't be a problem.
The torrents run on a separate HDD. Precisely because of I/o problems. It refuses to download faster than about 3-5 mb / s, when we were torrenting on the same disks as the OS disks. Because of disk 100% active time.

Now it's on a completely separate disk that does nothing but serve / download torrents.

I've just checked the altitude log file, which is about 900 kb by now.
And i found about 20 hitches. All of them are immediately after / during a map gets loaded. So those aren't a problem.

I think the setting up of a QoS for the altitude ports / giving the server more memory made the biggest difference.

I've also noticed that we need a new router :P Current router can't rout more then 150 mbps total and about 300-400 connections.

I noticed that ping 192.168.1.1 -t weren't 1 ms, but more like 9-10 ms on average. And outgoing pings weren't great either. Because the router was too busy.

So we've placed the torrenting on much stricter limitations. First max 3 mb down / 1 mb up. But still about 280 connections open. So we cut that in half as well. Max 150 now. Should improve the overall latency.

Only thing i still find odd are those errors: http://pastebin.com/d5YMNzMK

Server isn't doing anything except running with people in it. But it does give a bunch of errors and after that it suddenly goes from:

INFO [2012-11-13 18:56:46,788] [Thread-6]: Update check result: LOCAL_VERSION_IS_UP_TO_DATE, free memory: 116,3 / 494,9 MB

to:
INFO [2012-11-13 18:58:46,996] [Thread-6]: Update check result: LOCAL_VERSION_IS_UP_TO_DATE, free memory: 43,3 / 494,9 MB

So those errors just sucked up 70 MB of memory.

It's still running smoothly though. So i guess those errors don't say anything like you said. Still a bit odd :P

Might try changing the java to see if it makes any noticeable difference :P

[sukosevato]

Long story short: Router is being a dick :P

We've noticed that if anyone in the network, torrents with more then 50 open connections. That the ping to the router will detoriate rather quickly.

So with 200 open connections, you'll have about
10-20 ms ping to 192.168.1.1

Instead of < 1 ms, this is causing the massive disconnects since the router is dropping connections because it can't handle the load.

So even though maybe 40% of the 100 mbit link is being used. It's still horrible because the cpu of the router is maxed / memory is full. Only has 32 mb ram in it. But it's supposed to be able to route 150 mbps. Not with too many open connections apparently.

So we're buying an Asus RT-N66U, should be able to handle the load / a lot of open connections without causing massive disconnects.

Will report back when the new router is installed. It should solve all issues.

[Cracker]

We just had these messages again and everyone (about 20 players over 2 servers) was kicked out:

[port 27280] [06 feb 2013 20:02:31,263] INFO: Unresponsive client Client[name=[x Crackos x], ip=ip, playerId=7]; 151 updates written without response
[port 27280] [06 feb 2013 20:02:31,265] INFO: Removing client '[x Crackos x]' from ip=ip, playerId=7, message: Connection to game server timed out

There is enough free memory so that can't be it:

[06 feb 2013 20:02:32,801] INFO: [Data Transfer Since Launch] Outgoing: 0,533 GB, Incoming: 0,237 GB
[06 feb 2013 20:02:32,005] INFO: Update check result: LOCAL_VERSION_IS_UP_TO_DATE, free memory: 307,8 / 494,9 MB

Torrent problems and router problems were already fixed.
Anyone knows what happened here and how we can fix this?

Thnx in advance

EDIT: Happened again and now with this messages:


[port 27279] [07 feb 2013 21:14:36,920] ERROR: [IP: 212.226.74.165:51401] error reading pre-state for nuon 44
[port 27279] java.lang.NullPointerException[port 27279]
[port 27279] at Jp.a(SourceFile:209)[port 27279]
[port 27279] at Jp.a(SourceFile:192)[port 27279]
[port 27279] at ew.a(SourceFile:37)[port 27279]
[port 27279] at VN.a(SourceFile:245)[port 27279]
[port 27279] at aat.a(SourceFile:427)[port 27279]
[port 27279] at EA.a(SourceFile:18)[port 27279]
[port 27279] at aat.c(SourceFile:415)[port 27279]
[port 27279] at aat.d(SourceFile:351)[port 27279]
[port 27279] at aat.e(SourceFile:219)[port 27279]
[port 27279] at fJ.t(SourceFile:240)[port 27279]
[port 27279] at fJ.run(SourceFile:263)[port 27279]
[port 27279] at java.lang.Thread.run(Unknown Source)[port 27279]
[port 27279] [07 feb 2013 21:14:36,945] ERROR: Failed to execute post-state modifiers for Client[name=<[eVo]JWhatever>, ip=ip, playerId=17]
[port 27279] java.lang.RuntimeException: readPostState(44) failed - null[port 27279]
[port 27279] at ol.a(SourceFile:29)[port 27279]
[port 27279] at Jp.i(SourceFile:266)[port 27279]
[port 27279] at VN.v(SourceFile:779)[port 27279]
[port 27279] at VN.a(SourceFile:313)[port 27279]
[port 27279] at fJ.t(SourceFile:241)[port 27279]
[port 27279] at fJ.run(SourceFile:263)[port 27279]
[port 27279] at java.lang.Thread.run(Unknown Source)[port 27279]
[port 27279] Caused by: java.lang.NullPointerException[port 27279]
[port 27279] at ol.a(SourceFile:22)[port 27279]
[port 27279] ... 6 more[port 27279]
[port 27279] [07 feb 2013 21:14:36,963] ERROR: [IP: 62.20.209.185:27272] error reading pre-state for nuon 47
[port 27279] java.lang.NullPointerException[port 27279]
[port 27279] at Jp.a(SourceFile:209)[port 27279]
[port 27279] at Jp.a(SourceFile:192)[port 27279]
[port 27279] at ew.a(SourceFile:37)[port 27279]
[port 27279] at VN.a(SourceFile:245)[port 27279]
[port 27279] at aat.a(SourceFile:427)[port 27279]
[port 27279] at EA.a(SourceFile:18)[port 27279]
[port 27279] at aat.c(SourceFile:415)[port 27279]
[port 27279] at aat.d(SourceFile:351)[port 27279]
[port 27279] at aat.e(SourceFile:219)[port 27279]
[port 27279] at fJ.t(SourceFile:240)[port 27279]
[port 27279] at fJ.run(SourceFile:263)[port 27279]
[port 27279] at java.lang.Thread.run(Unknown Source)[port 27279]
[port 27279] [07 feb 2013 21:14:36,982] ERROR: Failed to execute post-state modifiers for Client[name={arr} Squid&Whale, ip=ip, playerId=5]
[port 27279] java.lang.RuntimeException: readPostState(47) failed - null[port 27279]
[port 27279] at ol.a(SourceFile:29)[port 27279]
[port 27279] at Jp.i(SourceFile:266)[port 27279]
[port 27279] at VN.v(SourceFile:779)[port 27279]
[port 27279] at VN.a(SourceFile:313)[port 27279]
[port 27279] at fJ.t(SourceFile:241)[port 27279]
[port 27279] at fJ.run(SourceFile:263)[port 27279]
[port 27279] at java.lang.Thread.run(Unknown Source)[port 27279]
[port 27279] Caused by: java.lang.NullPointerException[port 27279]
[port 27279] at ol.a(SourceFile:22)[port 27279]
[port 27279] ... 6 more[port 27279]
[port 27279] [07 feb 2013 21:14:37,255] ERROR: [IP: 212.226.74.165:51401] error reading pre-state for nuon 44
[port 27279] java.lang.NullPointerException[port 27279]
[port 27279] at Jp.a(SourceFile:209)[port 27279]
[port 27279] at Jp.a(SourceFile:192)[port 27279]
[port 27279] at ew.a(SourceFile:37)[port 27279]
[port 27279] at VN.a(SourceFile:245)[port 27279]
[port 27279] at aat.a(SourceFile:427)[port 27279]
[port 27279] at EA.a(SourceFile:18)[port 27279]
[port 27279] at aat.c(SourceFile:415)[port 27279]
[port 27279] at aat.d(SourceFile:351)[port 27279]
[port 27279] at aat.e(SourceFile:219)[port 27279]
[port 27279] at fJ.t(SourceFile:240)[port 27279]
[port 27279] at fJ.run(SourceFile:263)[port 27279]
[port 27279] at java.lang.Thread.run(Unknown Source)[port 27279]
[port 27279] [07 feb 2013 21:14:37,281] ERROR: [IP: 62.20.209.185:27272] error reading pre-state for nuon 47
[port 27279] java.lang.NullPointerException[port 27279]
[port 27279] at Jp.a(SourceFile:209)[port 27279]
[port 27279] at Jp.a(SourceFile:192)[port 27279]
[port 27279] at ew.a(SourceFile:37)[port 27279]
[port 27279] at VN.a(SourceFile:245)[port 27279]
[port 27279] at aat.a(SourceFile:427)[port 27279]
[port 27279] at EA.a(SourceFile:18)[port 27279]
[port 27279] at aat.c(SourceFile:415)[port 27279]
[port 27279] at aat.d(SourceFile:351)[port 27279]
[port 27279] at aat.e(SourceFile:219)[port 27279]
[port 27279] at fJ.t(SourceFile:240)[port 27279]
[port 27279] at fJ.run(SourceFile:263)[port 27279]
[port 27279] at java.lang.Thread.run(Unknown Source)[port 27279]
[port 27279] [07 feb 2013 21:14:37,282] ERROR: Failed to execute post-state modifiers for Client[name={arr} Squid&Whale, ip=ip, playerId=5]
[port 27279] java.lang.RuntimeException: readPostState(47) failed - null[port 27279]
[port 27279] at ol.a(SourceFile:29)[port 27279]
[port 27279] at Jp.i(SourceFile:266)[port 27279]
[port 27279] at VN.v(SourceFile:779)[port 27279]
[port 27279] at VN.a(SourceFile:313)[port 27279]
[port 27279] at fJ.t(SourceFile:241)[port 27279]
[port 27279] at fJ.run(SourceFile:263)[port 27279]
[port 27279] at java.lang.Thread.run(Unknown Source)[port 27279]
[port 27279] Caused by: java.lang.NullPointerException[port 27279]
[port 27279] at ol.a(SourceFile:22)[port 27279]
[port 27279] ... 6 more[port 27279]
[port 27279] [07 feb 2013 21:14:37,283] ERROR: Failed to execute post-state modifiers for Client[name=<[eVo]JWhatever>, ip=ip, playerId=17]
[port 27279] java.lang.RuntimeException: readPostState(44) failed - null[port 27279]
[port 27279] at ol.a(SourceFile:29)[port 27279]
[port 27279] at Jp.i(SourceFile:266)[port 27279]
[port 27279] at VN.v(SourceFile:779)[port 27279]
[port 27279] at VN.a(SourceFile:313)[port 27279]
[port 27279] at fJ.t(SourceFile:241)[port 27279]
[port 27279] at fJ.run(SourceFile:263)[port 27279]
[port 27279] at java.lang.Thread.run(Unknown Source)[port 27279]
[port 27279] Caused by: java.lang.NullPointerException[port 27279]
[port 27279] at ol.a(SourceFile:22)[port 27279]
[port 27279] ... 6 more[port 27279]