Server Hitches aka lag spikes

[sukosevato]

Hey all. I'm running a couple of servers 'Oh-Clan servers'

I've been running 2 14 slot servers with custom maps for a couple of months now. And I keep getting server hitches.

When I look on the server while a game is going on I keep seeing:

[12 Sep 2011 11:36:01,841] WARN: Server hitch detected: 1773.0 milliseconds, free memory is 141.1 / 244.7 MB
[12 Sep 2011 11:38:01,837] WARN: Server hitch detected: 1753.0 milliseconds, free memory is 138.1 / 245.4 MB
[12 Sep 2011 11:40:01,848] WARN: Server hitch detected: 1772.0 milliseconds, free memory is 190.2 / 246.4 MB
[12 Sep 2011 11:42:01,837] WARN: Server hitch detected: 1743.0 milliseconds, free memory is 130.6 / 247.6 MB
[12 Sep 2011 11:44:01,837] WARN: Server hitch detected: 1737.0 milliseconds, free memory is 181.0 / 248.4 MB
[12 Sep 2011 11:46:01,837] WARN: Server hitch detected: 1752.0 milliseconds, free memory is 160.9 / 248.6 MB
[12 Sep 2011 11:48:01,838] WARN: Server hitch detected: 1735.0 milliseconds, free memory is 152.6 / 248.6 MB
[12 Sep 2011 11:50:01,837] WARN: Server hitch detected: 1764.0 milliseconds, free memory is 145.8 / 249.2 MB
[12 Sep 2011 11:52:01,838] WARN: Server hitch detected: 1771.0 milliseconds, free memory is 167.8 / 249.6 MB
[12 Sep 2011 11:54:01,842] WARN: Server hitch detected: 1791.0 milliseconds, free memory is 125.8 / 249.7 MB

As you can see its exactly every 2 minutes. And now that I put them next to each other I noticed every hitch is ALWAYS in the range of 837 - 850 ms

Anything specific happening around then? I'm pretty sure its a bug in the server software seeing as I ruled out everything else.

-Gave it more memory 256 MB start up 768 mb burstable, and you can see it isn't running out of memory either.
-Runs on a i5 2500k with cpu usage of less then 10% all the time
-installed on a SSD
-Fiber connection, but over lan same problems.

The fact that it occurs ALWAYS at the same time at the same interval is pretty much enough evidence I think that this is a server related bug? Something is causing the server to 'hang' every 2 minutes precisely at the same action. Any idea what happens then? Want more logs or anything. Ask away. I my self have no clue what else this can be caused by

Except maybe that the server is hooked up wirelessly (but its literally 20 cm away from the router and the connection is stable as hell, never noticed anything on the mc servers etc.. I'll be hooking it up to a switch later this week anyway but I doubt it'll solve this problem)





WARN: Server hitch detected: 1694.0 milliseconds, free memory is 11.7 / 71.6 MB

It happens every couple of minutes and can be annoying in game because its a lag spike and can be deciding in a ball game if on your screen the enemy continues in 1 direction while in fact they are not. Can create quite a bit of confusion just enough to score.

I am wondering what causes these hitches. I doubt it has anything to do with the server it runs on.

I am running the server on a i5 2500k with 8 GB ram on a 60 GB SSD, OCZ agility 3. On a 100 / 100 mbit fiber connection. On Ubuntu 10.04 LTS

Its not my internet or anything. Its a dedicated server so nothing runs on it except a website / forum / a couple of minecraft servers and the altitude servers. But still got like 4 GB of ram left. and CPU usage is about 10-20% with the i5 running at 1.6 Ghz to save power.

The server is hosted at a friends place that has fiber. When my friend connects over LAN to the server these hitches still occur. So i'm pretty sure it is not a connection problem.

What can it be then?

I'm thinking it is either a bug in the server software. Or the lack of memory or something? That the proces runs out of memory and 'clears' its memory a bit which causes the lag spike?

I have no clue. Hoping you guys have seen this before / know a possible fix.

Hope you guys can shed some light on this for me.

---edit2

WARN: Server hitch detected: 1773.0 milliseconds, free memory is 141.1 / 244.7 MB

After assigning more memory to the server, the servers still gets these hitches. Even with plenty of memory left as you can see. I'm pretty much out of ideas what can cause this except a bug in the server software.

Last thing I can think of is the fact the server is connected wireless. (because at my friends house they got too few router ports and nobody there wanted to sacrifice his / her 100 mbit downloads and 25 mbit is plenty of upload for the server via wireless) Gonna hook the server up via a switch i got some time ago though. But I doubt this can be the cause since the MC servers and everything else have no problems at all. So i'm really thinking its a bug in the server software.




---edit

Figured out how to assign more memory to the servers. (they launch pretty much the same way as Minecraft servers do with xms / xmx) They start with 256 MB ram now and can go up to 768 MB. Will post in a bit if hitches still occur.

I read on the wiki that 1 instance of the server will not use more then 150 MB of ram and 300 MB on start up etc.. I'm running 4 servers in this. Added a few recently, but hitches also occured when we ran only 1 server. Is there any way to allocate more memory to the servers? So it has say 500 MB of ram for the servers

[lamster]

The constant interval and consistent timing of the hitches is extremely suspicious and strongly suggests an external process running on the server machine is responsible. What OS is the host machine running? Use whatever available profiling tool (on Linux I'd check top, on Windows I'd start Task Manager) to watch for "spiking" processes (CPU consumption peaks every 2 minutes). You may want to check your crontab (linux) or disable anti-virus (windows) as a first test.

[sukosevato]

This is what the performace of the server is like. All the time. There are rare cpu spikes going upwards to 40% (when the CPU is downclocked to 1.6 Ghz still. Never seen it go to 100% not even on a single core. Since it'll clock it self to up to 4.2 Ghz if it got stuff to do, rarely does that though)
But those spikes are very rare and in no way consistent with the hitches.

The OS is Ubuntu 10.04 LTS, 64 bits. Desktop version. Runs on kernel 2.6.32-33-generic. GNOME 2.30.2

When i look in the same CPU performance monitoring program it says load overages of 0.84 / 0.95 / 0.92 for the last 1,5,15 minutes. So CPU is being used for like 1%. I highly doubt that's the problem. And average ram usage is about 2-3 GB.

About the crontab, what exactly is that / where can i find this / what info do you need? :P I googled it and it is a set of stuff which happens at a interval like a virus scan that happens every 2 minutes or backups being made. As far as i know there are no such things. Got no anti virus or anything on linux. And the only regular thing that happens are minecraft back ups and those are every 3 hours. And yes then you notice a slight peek in cpu usage but again not even more then 40% on 1.6 Ghz.

Any log files you want? system information? or suggestions? I personally think it has nothing to do with the server. The server has a TON of performance left and its something software'ish that is causing this. Definetly not running out of resources.

Thanks for your reply btw.

[lamster]

You can open a terminal and type
crontab -l
to view existing crontabs.

More likely is that something deep in the system (network drivers, hard drive controller, etc) is performing some specific system-freezing activity exactly every two minutes (the hitches are almost perfectly aligned to your system timer -- e.g. 52:00 + 1.77 second hitch -> log report at 52:01.83 -- if it was caused by periodic work within the Altitude process you would not see this perfect system timer correlation because we don't trigger on predetermined system times).

You could check your power conservation settings (hibernation, hard drive power down interval, etc) and disable any power saving options. I've also heard of certain versions of network controllers automatically running a wireless scan (freezing everything) every two minutes, etc. Good luck, stuff like this is hard to track down.

[sukosevato]

Did the crontab and there are no crontabs :P So that solves that.

So yea.. What can we do to test this :P Next weekend i'm gonna hook up the server through wired internet. To make sure the wireless drivers aren't playing funny when it comes to altitude.

If the problem then persists i'm gonna rule out the router / internal network at my friends place by hosting the servers on his i5 2500k windows pc. To see if they run fine on his pc. If they do it'll be something wrong with the ubuntu install then i guess.

Loads of trouble shooting to do then :P Lucky me

[sukosevato]

Found a topic from a guy that had the same wireless adapter and the same problem. aka precisely every 2 minutes a little hang. He was also using Ubuntu (older version).

He upgraded the drivers for the wireless card and it fixed his problem. So think we're gonna try that as well.

Atm drivers 1.8.1.1 are on the card.

And the newest available drivers are 2.4.0.0

So we're gonna try that. Will let you know around friday when we're gonna update them how it went.

thanks so far for your help Lamster.

[sukosevato]

Server has a wired internet connection now. Cut down about 10 ms in ping and the hitches are gone.

[lamster]

Glad to hear it's fixed, thanks for posting with an update. Drivers work in mysterious ways...

[sukosevato]

True that. Drivers are weird things :P

So far everything is good.

We're running about 7 servers now. The [oh] servers.

A friend of mine who is even more into altitude is the one who edits the servers. He has been editting a lot of maps as well. Like making 1 life maps easier to practice on, with respawn or extra power ups and what not.

We also have a second 'game mode' server. Where its basically ball, except there are bombs and turrents introduced into the game and the goals are blocked by bases and other tactical passage ways are also blocked by turrets.

So you first have to use the bombs to clear out the goal, and of course you try to shoot down the others if they get close with a bomb to your own goal. Its quite a fun game mode. Its a bit of everything. And a nicely placed bomb can take out 3 opponents, even if you're in a small loopy. So team play is really necessary.