Degraded net code (in Australian Alti games?)

[Gruntbuggly]

Hi guys,

Just wondering if anything might have happened in any recent patches to the net code which would affect lag in Australian servers? Both our public servers and match servers appear in the last 4 weeks to be getting a huge amount of extra lag. The servers are run via different ISPs so that rules out an individual hosting issue. This isn't something imagined and it's mostly the 1.5+ years players that are noticing major differences.

Thanks for your great work and continued patches, just interested to see if there could be a deeper cause than bad luck.

Gruntbuggly

Edit: I should also have pointed out that in Aussie Alti we play ball 90% of the time, the other 10% is TBD.

[lamster]

Nothing changed in the netcode, but it's conceivable that there's a new memory leak that could, after a sufficiently long period of play without restart, cause performance issues that look like network spikes (I recently restarted the official servers that seem to have run low on memory after several weeks of play). I recommend checking the server log (~Altitude/log/ServerLauncher.log) for lines like

Server hitch detected: 4430 milliseconds, free memory is 2.5 / 128.0 MB

In this case restarting the servers should temporarily fix the problem (until another low memory situation occurs after weeks of play). Otherwise I'd suspect (A) network issues either with some of the players or both of the ISPs, or (B) performance issues with the servers (sometimes a VPS host will begin "overselling" their CPU/memory causing applications hosted on those boxes to experience severe performance issues).

[Gruntbuggly]

Cheers lamster, I'll pass this info on to the server admins.

[VipMattMan]

Originally Posted by lamster

Nothing changed in the netcode, but it's conceivable that there's a new memory leak that could, after a sufficiently long period of play without restart, cause performance issues that look like network spikes (I recently restarted the official servers that seem to have run low on memory after several weeks of play).

I think this is very likely the case. The performance of the ladder servers started degrading last week, and I just noticed that Dojo seems to be starting to go through this process as well.

Pings are typically fine, but responsiveness is decreasing as far as pick-ups and things like that go.

[phong]

It looks like this is happening once a week or so on the ladder servers.

ServerLauncher.log.2:WARN [2011-09-14 17:56:51,659] [Thread-14]: Server hitch detected: 1508.0 milliseconds, free memory is 4.3 / 89.1 MB

ServerLauncher.log.6:WARN [2011-09-09 19:22:13,774] [Thread-8]: Server hitch detected: 1185.0 milliseconds, free memory is 10.0 / 78.9 MB

This isn't a VPS however a dedicated box with 32GB of ram. Is there a way to manually increase java memory for the server launcher? Sunshine has mentioned to me people have been complaining of spiky game play when latency looks good (what vip said).

[malkin]

Originally Posted by VipMattMan

I think this is very likely the case. The performance of the ladder servers started degrading last week, and I just noticed that Dojo seems to be starting to go through this process as well.

Pings are typically fine, but responsiveness is decreasing as far as pick-ups and things like that go.

+1 Matt. I have to be so careful now when passing around a euro

[Tekn0]

Originally Posted by malkin

+1 Matt. I have to be so careful now when passing around a euro

Us euros have the same problem.

[lamster]

Originally Posted by phong

It looks like this is happening once a week or so on the ladder servers.

ServerLauncher.log.2:WARN [2011-09-14 17:56:51,659] [Thread-14]: Server hitch detected: 1508.0 milliseconds, free memory is 4.3 / 89.1 MB

ServerLauncher.log.6:WARN [2011-09-09 19:22:13,774] [Thread-8]: Server hitch detected: 1185.0 milliseconds, free memory is 10.0 / 78.9 MB

This isn't a VPS however a dedicated box with 32GB of ram. Is there a way to manually increase java memory for the server launcher? Sunshine has mentioned to me people have been complaining of spiky game play when latency looks good (what vip said).

Those hitches represent system pauses as perceived by the server launcher task. These can occur when the system is under heavy load (starving the launcher task), when the launcher is out of memory and desperately looking for more ("free memory" will report / 128.0 MB), or because the launcher randomly performed a huge garbage collection (which definitely looks to be the case here). Note that the pauses in your log are short and infrequent -- 1.2 and 1.5 seconds over 5 days -- and would typically not have any impact on gameplay (any hitch can cause a small hiccup in gameplay but it's rare for real players, whose home network connections cause plenty of stutters on their own, to notice hitches unless they're 3+ seconds long and occur regularly at <5 minute intervals). If a memory leak were responsible you would see free memory lines indicating very low memory available (less than 3.0 MB typically) and completely maxed "internal heap" (128.0 MB), and you'd see far more stuttering. In your entries the available heap is 78.9 and 89.1 -- well below the default max of 128.0. That said, if you'd like to increase the "max internal heap" from its default value of 128 MB (to 256 MB, for example) you can create a file server_launcher.vmoptions (in the altitude install directory, next to the server_launcher executable) with the following single line of data:
-Xmx256m

What players in this thread have described (poor responsiveness of pick-ups, difficulty passing around some players, despite no obvious ping problems) is essentially always related to packet loss. The most likely explanation here is a (possibly seasonal, time-of-day dependent) degradation in link quality from certain distant regions to the ladder server's internet provider. Packet loss can also occur if the server machine or machines that share its network pool are under heavy network load (e.g. uploading a large automated backup with insufficient throttling can temporarily consume all available upstream bandwidth).

[phong]

Thanks lam, I'll give that a shot and keep lookin

[Bockit]

Are there any utilities I can use to detect packet loss so we can confirm the cause and see what is possible to fix it?

I'm running one of the servers Gruntbuggly was referring to.

[lamster]

There aren't any good ways to see packet loss in the current build (I'll think about adding something), but players experiencing problems could try pinging your server while they play to look for loss.

1) Note the server ip you've connected to in the chat box (e.g. 75.102.27.114)
2) Start -> Run -> cmd
3) start a recurring ping on the target server IP, for example
ping -t 75.102.27.114
4) Play for a while, noting periods of poor performance
5) Tab back to the command prompt and press Ctrl+C to terminate the recurring ping. It will print a packet loss statistic (Lost = 31, (8% loss)).

Ping packet loss won't necessarily match UDP packet loss, but it correlates very strongly in most cases.