V1.5 as of Fre, 20. Feb 2004
I read a couple of papers and Internetpages I found to figure out whats the best configuration of an UT Server and what parameters influence its performance in what way.
Here is what I found out. Most stuff is pretty obvious, I guess you allready know that.
There are mainly 2 Issues:
A. Bandwidth of server
/ client connection
B. CPU Power of the given server
Both Aspekts are discussed in Detail in the following Sections. please note, that this Document is about UT Servers, and not about UT2003. Anyhow, many aspects of this are very much the same with UT2003/4, since UT2003/4 uses the same engine then UT, only a much newer Version. Most Aspects are also true for Games which are based on the Unreal Engine, such as Splinter Cell, Star Trek Elite Force, Tactical Ops, Strike Force and many others.
For your information:
Unreal 1 -> Engine Versions up to 250 (226 is the last patch I guess)
UT -> Engine Versions 300 up to (currently) 451
UT XBOX (aka Unreal Champinoship)- > Engine Versions up to 900
Unreal 2 -(aka The Awakening)> Engine Version 1400 (current 1403)
UT2003 -> Engine Versions around 2200 (currently the highest version is 2225.1)
UT2004 -> Engine Versions 3000 and up (the demo has 3120)
Unreal 2 XMP (multiplayer Version of Unreal2) -> Engine Numbers around 6400 to curently 7710 (these numbers are a bit out of order, since the XMP is more a patched 1400 engine then realy an enigne later then the UT2004 engine , this might lead to a number conflict later...)
A. Bandwidth Issues
What Bandwidth does a UT-Server need?
That depends on how many Players
want to connect at a given time and what Bandwidth all these players
need. The players bandwidth is dependent on 4 Parameters:
1. what connection bandwidth
does a player have?
2. what is the max connection speed configured in the players
.INI File
3. what is the max connection speed for a single player configured
in the .INI File of the server
4. what is the max throughtput connection speed of the path between
the player and the server
All of these issues are discussed
below:
First of all: As a general rule of thumb the LOWEST bandwidth
of 1-4 determines the players connection speed to the server.
Within these topics lays a lot of reasons for poor gameplay.
1. what connection bandwidth does a player have?
This is the raw bandwith of a players connection, thus how big
is his line. This may vary from 56 K modem lines to XDSL or cable
lines. XDSL lines come in various flavors. A main difference is
syncronictiy. ADSL Lines (thus ASYNCRON DSL) have a good DOWNSTREAM
bandwidth (from the net to the player) but a small UPSTREAM bandwidth
(from the player to the net). SDSL Lines are syncronus DSL lines,
both direktions have the same bandwidth.
The vast amjority of DSL lines is asyncronus. The have as an example
700K downstream, but only 160 K upstream. In such a situation
the player could only send 160 Kbit/s data to the server and Receive
up to 700 Kbit/s.
This fits quite well the players
needs:
Allthough mostly UT Traffik is syncronus, it usualy takes never
more bandwith then about 100kbit/s (and even that only under rare
circumstances, such as Exesive Servers). So most UP channels are
broad enough for gameplay, and the broader DOWNlinks are good
for decent map download.
That is also the reason why 56K Modem lines are worse then they should be: a 56K modem does some sort of switching in the upstream and downstream to get the maximum bandwidth and also needs to convert signal from digital to analg and back, this switching and processing time causes a bad ping. A 64Kbit/s ISDN line, allthough only 20% better bandwith gives a much smother gamplay.
So for most situations not the limited bandwidth of a modem itself is the problem, but the dataprocessing which results in bad ping. (ping is discussed more closely in 4. of this chapter)
Unfortuantaly this is not the
onyl item in players connection. Another issue is the type of
error-correction the line-provider builds in his lines.
Many providers do errorcorrection on linelevel (OSI-Layer 2).
Thus any package which is uncomplete, has checksum errors or other
problems is discarted. The line-terminators, means the boxes which
are installed at each lines end, most people call it MODEM or
ROUTER (which is both wrong) communicate with each other to RESEND
the discarted faulty package.
This gives the customer the illusion of an error free line, all data pakages which are coming out of the line-terminator box are 100% error free. The backdraft is, that some of them are coming a bit later then they should... namely the resent faulty packages.
But also all pakages which are okay will travel a bit longer through these lines due to the extended checking process. This gives higher lantency to such lines. Please note that "latency" is the technical term for the UT-Term "ping". So whenever a UT Gamer says he has "bad ping", a network technichan whould say "the conection has high latency".
Some providers don´t do error-corection on Layer 2 (the connection layer) at all. Here the faulty packages are presented to the computer and the application which needs the packages... in our case UT. UT itself has then to figure out faulty datapakages and deal with them.
Unfortunately, UT is pretty good in dealing with faulty pakages up to a certain percentage (about 2%-3%), but VERY BAD in dealing with datapakages which are coming later then they should. (if you think about it... pretty logical, because timely presentation of a given persons position is pretty essential for UT)
As a result all players who have a line WITH built in error-correction will have gameplay with much higher ping then players with lines which do NOT have error-correction!
Example: Germans T-Online T-DSL Lines are with error-correction. All players using this lines are encountering laggier gameplay in comparison to players with ARCOR lines, which do not have error-correction on line level. (Marketing whise ARCOR calls the lack of correction on line level "fast forward mode") allthoug the raw bandwidth of both lines might be the same!
2. what is the max connection speed is configured in the players .INI File
This is the part in the .INI File where it reads:
[Engine.Player]
ConfiguredInternetSpeed=20000
This number may be configured
in the OPTION menue by choosing the Linetype such as LAN, Cable,
modem etc. To make things a bit more complicated this gives the
desired bandwidth in BYTES per Seconds, not bits per second.
To calculate the corosponding line bandwidth, you need to multiply
this with 10 (8 bits per Byte plus start and stop bit). To calculate
the numbers needed in the .INI file from the line speed you have,
simply divide it by 10.
Example: You have a 56 Kbit Modem, which actuall connects with 44 Kbit/s. That is 44*1024 (becaus as you all know 1 K is in computer land 1024 not 1000) makes 45056. Diveded by 10 makes some 4500. So thats the number you should type into your .INI File.
If you type a larger number
then your line can actually handle, you will increase packet loss,
because the server sends you more data then your line can deliver.
BAD thing. So be carefull with this. It is very important to match
this number with yout downstream rate. If in doubt, make it LOWER.
It makes no sense to get higher then 25000 here, resulting in
244 Kbit/s. So even if your connection can handle more then 244
Kbit/s that does not help you, because more data will be not sent
from the server (well that is not EXACTELY true, as discussed
later).
Most sources say, that u will rarely have more 8000 bytes/sec comunication with a server. You can check this yourself by hitting F6 during gameplay and watsch the numbers. U will almost never see somthing above 8000.
So if you are looking for a new line for your home gaming, more then a 256 Kbit/s line SDSL (Syncrone) is not needed for very decent UT connections! Even a 128 kbit/s line is absolutely suffient.
Anyhow, this does not determine the connetion speed alone:
3. what is the max connection speed for a single player configured in the .INI File of the server
The server has in its .INI file a very simmilar section:
[IpDrv.TcpNetDriver]
...
MaxClientRate=20000
This is corrosponding to the bandwidth settings in the players .INI File. If the bandwidth on both sides are different the client connects with the LOWER speed of these 2 settings. Many servers still have the default here, which is 5000 ONLY! So even if you have a Super-UltraDSL Line with 455 TERRAbit Bandwidth and configure your .INI File to 23534636, you will still geht 5000 as configured on the server.... thus onyl litlle more then a modemuser!!!
Rising the MaxClientRate on the server side gives 2 problems. First and obvoius it eats up the servers conection bandwidth rapidely. Lets assume, the server is behind a 2 mbit/s S2M Digital line (pretty good in the world of internet). If you grant per MaxClientRate=5000 every client only maximum aprxiamatly 50 Kbit/s (a very good modem line) then the Server can be played by some 35-38 players with no problem. Since the maxium a UT Server can have is 32, you then should never have problems resulting from BANDWITH.
If you rise MaxClientRate to 20000, then that means that every player can eat up to 204 Kbit/s (if their line is good enouth). As a result onyl 8 players can connect to the server! (well... only if the cleint really eats up the bandwith. Thats usually only possible in extremyl rare situations such as Mr.Pants Exessive servers or the like)
So a practical maximum is the following: The server is connected with Ethernet to the router and thus to the internet. All players have BIG XDSL Lines. Then you can get some 32 players on the server. (and 32 is also the maxium number a UT Server can be configured to... how handy!)
What happens if the bandwidth needed is larger then the servers line? Then the server will queue the datapackages he needs to send out. This will result in bad ping. If you see the ping getting higher and higher on a server DURING gameplay, getting good again after map change or restart then most likely the linespeed all users are connected with adds up to more then the servers line acutall can handle.
Another backdraft is the following: Not all users have the same line. If the MaxClientRate is 20000, then XDSL or cable users can get very decend datastreams and therefore good and smooth gameplay. But users who have only a modem will connect with much lower speed. The gap between the users and the cable/XDSL user is pretty big, the modem player has much less chances in winning.
(More in detail discussion
what datastream means in UT gameplay will be discussed later in
this page)
As a server administrator its most importent to know the audience
of a server. If all players have good lines, then a larger number
in MaxClientRate is good, provided the servers connection can
handle it. If you have more users with modem or ISDN singel channel
(64 Kbit/s) or want more players at a time on your server you
have to lower MaxClientRate. Of course you can always limit the
user son a server by MaxClients (how much players can conect to
a server)-
Practically this is a balance between MaxClients and MaxClientRate.
One last aspect: The download
speed of maps from the server to the client is limited by this
too!
So if the servers MaxClientRate is only 5000, even players with
the fastest connections will get not more then modem DL-Speed!!
This is the main reason why its a good idea to put the files on
a Webserver and redirect the Downlaod.
4. what is the max throughtput connection speed of the path between the player and the server
To get the picture her we need to discuss some basics first:
How are data transmitted
over the internet?
Some basic knowledge about internet data transfer, that will help
you to understand problems, that may occur and slow down your
online-play:
If a server sends data over the internet to a receiver (client)
or vice-versa, he will split these data into data packets. Each
packet will be sent separately. The server will look for the fastest
connection to the client at that time. The connection is usually
not a direct connection, but the data-packets are handed over
by a lot of transfer-servers. Each of these hand-overs is called
a "hop".
Usually the path of the 1st data packet defines the way, that
all following packets take, so if a "traffic jam" occurs
later during transmittal, the packets will not automatically be
diverted to a faster way.
To get an idea of the ways, that your data packets travel over
the internet, you can get yourself a TRACEROUTE application (shareware
available for all plattforms, just search download.com)
What influences the quality
of my UT online play?
How good you can play UT online over the internet depends on your
connection to the server, that you play on. The quality of this
connection is determined by 3 things:
1. The bandwidth of
the connection: We allready discussed that.
2. The "ping"
time. That is the time, that a data-packet needs to travel
from your machine to the server and back. In UT online play, it
is the server, who decides about the status of the map. Be aware,
that the picture of the map, that you see on your screen while
playing UT online, is not the current picture of the world, but
that the picture is delayed by the ping time during transmittal.
So if you fire at an opponent, you may miss him, because he moved
a bit during the time, that the picture needed to be sent to you.
That means, if your ping-time is too high, you will have major
difficulties to hit anybody, even if you have a good aim. Yeah,
we all know.
3. The packet-loss
rate. That is the percentage of data-packets, that get lost or
corrupted during transfer. Your PL rate should be below 3%. Higher
PL results in strange effects: You skip around in the level, even
through walls and ceilings, as if you were ghostlike. Your player
model simply doesn't move as you tell him to do; UT is virtually
unplayable. So, with high PL, even a low-ping connection is worthless!
Beyond 5-6% paketloss you can simply forget gameplay.
Please note, that the error
correction mentioned earlier makes pingtime bad, but packetloss
good.
So to be a bit more precise: It the paketloss of the players line
itself whould be more then 3%-4% error correction whould be a
GOOD thing. But thats usually not the case. Paket loss happens
more somewhere else in the way the paket travles through the internet,
not in the XDSL line of the player. One of the MAIN reason for
paketloss is a server whos line is to small to handle the add
up conection bandwidth of all clients.
Please note also, that MOST Players call a server LAGGY or talk
about LAG
***not***
if the ping is bad,
but if the paketloss is high. Paketloss is a much worse thing
then bad ping.
It makes your game "stuter". You might all know these
situations where suddenly the game "hangs" for a 1/10
second or so and then its fluent again yo are allready dead. This
is caused either through unsufficent bandwith on the serverside
or pakeloss. The solution for the player is most of the time to
LOWER his Connectionspeed in his .INI file as you might understand
now... ;-)
Also all of the discussions
such as:
Player A: Oh its so laggy, cannot play really---
Player B: Why? Your ping is fine!
are useless. Fine ping does not mean lagfree gaming.
How can I improve my ping
and packetloss?
The ping is influenced by three things:
1. The time for the 1st hop
from your computer to your ISP is influenced mainly by your modem
or line... hoew to influence that? Get a better line.
2. The transfer time and bandwidth
of your ISP, as well as your ISP's connection to an internet backbone.
Ways to improve this: Get a better ISP.
3. The "distance" of the server from
your ISP's internet connection point. "Distance" in
this case is not the geographical distance. There is a certain
correlation between geographical distance and transfer time, but
not necessarily.
The UT server browser shows you the ping times to each server.
Always choose a server with a low ping! Before you connect, highlight
the server in the list and ping it a few times to see, if the
ping is stable.
The packetloss is mainly a matter of the connection cabling. Not
the player nor the servers admin will be abel to influenve that
, because it usually happens somewhere in the world on the way
the packets take from the server to the client.
This may may variy greatly EVEN WITHIN the game. That is one of the reason why sometimes the ping changes even while gaming. (other reasons are allready diskussed above)
OK. So far this part.
B. Server CPU Power issues
Generall CPU Issues
Most CPU Power in playing UT
is needed to calculate the actuall picture. So the clients computer
should have a good display subsystem ("Graphic Card")
as we all know. I´m not going to discuss these matters here...
only one thing maybe: The most important factor is how long the
computers display subsystem needs to calculate the poygone structure
which ends up to the inplay picture.
This then results in the max framerate. The unreal engine tries
to maximize framerates through dynamicaly lower the quality if
the MaxDesired Framerate cannot be reached with the current quality
settings. Usually a framerate of about 30-40 is enouth, more gives
IMO no better gameplay. Almost all computer today can get that.
Nevertheless at some occasions gameplay stutters. This usally
happens when the polygones in sight are risign above a certain
level. To minimize the polygones UT has to handle at a given time,
maps are divied into "Zones", so that the UT engine
has only to take all polygones in this zone into consideration.
If some of you have worked with UnrealEd, you know the concept
of Zones: It divides the map into subareas.
Some maps do have very large zones, especailly large open land
maps, because a zone must minimunm reach that far as a palyer
can look in the map. These maps tend to stutter on slower machines,
some of them have so large zones, that even my Macintosh cannot
handle them (har har har).
Please note also that here lies the main reason for the UTbuild2000+ (known as UT 2003) performance issues: this engine can handle maps with 100+ more details then the UTuild300-500 (UnrealTournament) engine.
The server however does not
need to display anything. So even slower machines can make decend
servers. To have a look on the servers CPU needs, lets first have
a look on what the server has to do:
The server keeps all information of the world as it is in geamplay.
The server knows where all actors are at a given point. Please
note, that the term "actor" refers in UT not only to
players but to EVERYTHING which can move in any way... so basically
everything beside pure terrain.
Missiles are actors. PowerUps are actors. Weapons are actors.
Smoketrails are actors. Doors are actors.
Think of the server as a huge
EXCEL-Sheet. In the cells are some numbers for ALL actors in the
current game. The numbers determine postion, speed, acelleration,
direction, time and the like.
All cells have formulars attached which are able to calculate
how the actors numbers change trough time. If you fire a rocket
in this direction, the Rocket ist an actor. Inside the large "Excelsheet"
of the server is some cell containig data as: ROCKET, Fired by,
fired at, fired when, has this speed, heads this direction, has
this time to life befor exploding etc.
UT now recalculates this "Excelsheet" from time to time.
All these results are sent to the players computer if he needs
them, thus if he can SEE the given actor. So if you can see the
rocket, the server knows that by comparing the actor ROCKET and
the actor PLAYER in his "Excelsheet". If the are close
by and nothing is in the line of sight, then they can see each
other. If that is the case, the server sends the information via
Internet to the players UT engine: "Hey fellow, please show
your Computerowner a rocket, heading this and that direction.
New information will follow shortly..."
Obvioulsy the more often the server can calculate his "Excelsheet" the smother is the gaemplay. Imagine, the server could do that only one time per second, then the rocket whould not change its position for one second on your screen. Please note, that it whould not tracel in slow motion, the server automatically takes the passed time into calculation. So the rocket whould after one second "jump" on the postion it should be one second later.
The UT Server usually recalculates his datasheet 20 times per second. Now this is interisting in many ways. Lets have a view on some effects all this might have:
The calcualtion of the servers "Datasheet" needs more CPU Power the more actors you have in the game. So as well larger maps with more Powerups, Waepons, doors cars etc as more players need more CPU Power in the server. In oposite to the client, Zones or the like do not help, the server needs to know the state of all actors all time. If at a given point the server cannot handle the calcutlions any more he drops cycles. Lets assume, the given server needs 1/18 th of a second to do a calcultion cycle, then the server cannot calculate 20 cycles in the second anymore but only 18. This may vary inside the game pretty much, because the number of actors changes a lot, depending on the number of missiles shot. This is esentially critical in games where many missiles are on its way, such as PANTS EXSESSIVE Games.
To eleminate the effekt this whould have on gameplay, the client tries to prepredict what happens in the meantime. If the clients UT Engine does not get information about the travel of an actor it calculates ITSELF based on older information where the actor should be now (for example 1/2 second later)
This works very nice for rockets and missiles in general, because most of them do not change traveldirection after they have been shot. But for players or guided Redeemer Warheads (as an example) that is not true.
Thus if you see a player suddenly disappear right in front of your eys it might be, that a dropped connection OR a dropped calculation cycle was covered by your Cleint machine... only to learn 1/3 second later, that the player is NOT there, because he has turned to the left or the like.
Another effekt you all kown is that players "floate" in a given direction untill they hit a wall, this might also be a hint for Server CPU overload.
This leads us to a very important conclusion: Everytime the picture stutters, or the play gets laggy or the framerate drops, it is NOT the Servers CPU power, but either the clients graphic subsystem power or certain problems with the connection. Lack of CPU Power usually results in "jumpy" behavior of OTHER Players. Compare that to a totally connection loss. You all have noticed, that that does NOT results in stuttering images or things like that, only all movement off ALL actors beside yourself dont get any ACCELERATION anymore. You cannot shot, others either float in the direction they where heading last or the just stay where they are. Still you can run through the map with no problem.
Another thought: If the Server recalculates the scene "only" 20 times a second, whats the benefit of framerates higher then 20 on the clientside? Well, your OWN movement looks smoother till a level of up to some 30-max40 frames. Beyond that its absolutely useless.
Even worse: the Server calcualtes the scene all 1/20 second, but if he can send the results depends on the connection rate of the player! Modem players might get updates onyl 10 times a second or so!
Beside raw CPU Power to recalulate the scene, there are other needs the server has. Sending the appropriate data to lets say 20 Players is not an easy task. There are computers which cannot handle more than 1-2 Mbit/s in its network subsystem. Such a computer whould give a strange picture: You might connect it with Ethernet to the net. It might have a FAST CPU, which needs only 30% of its power to recalculate the scenes, all .INI stuff is confgured ok... nevertheless, only 10 DSL users can connect without getting lousy ping and paketloss. Wat´s wrong? Well the Server can calculate everything but his IP Driver might be to slow to deliver all the data to the Etherntadapter. The server has some sort of an INTERNAL bandwidth problem.
So raw CPU Power does not mean that the computer is a good UT Server, Mhz alone does not help. Allthough I have no good expirience in this I prepredict, that some old 500 mhz systems might make better servers then some 1.6 Ghz machines which are hardwareside illconfigured.
One last aspect : The Number of Cycles the server calcualtes per second can be changed in the server .INI File:
[IpDrv.TcpNetDriver]
...
NetServerMaxTickRate=20
Yep! So in this case a server does not recalculate the scene more then 20 times a second even if the CPU has lots of power left.
Obvieously, rising this level
to say 30 WHOULD have benefit if:
- the servers CPU power is sufficient
- the server networking subsystems can handle the amount of data
- the servers Inernet connection can handle the amount of data
- the clients line can handel his part of the data
- the clients computer can handle the additional data
- the clients graphic subsystem can show more then 20 frames/second.
Rising the level to more then 30 is useless, because the human eye has a max resolution of somwhere between 25-30 frames per second (movies have 24 frames as you know).
Please note, that fumbeling with this has linear effekt on all components: 20 ticks versus 30 ticks means 33% more CPU load, 33% more bandwidth consumption etc etc.
Special conciderations on LINUX Systems
Running a UT Server on UNIX
servers seems to be not such an easy task. Many
people I have talked to recently have told me to use Windows or
even Macintosh.
Unfortunately I dont have a free Mac and no Windows Server at
all, but a LINUX machines for free.
The main reason is, that LINUX
is no real-time operating system. Real time
means, that u can actual prepredict how long a given task takes...
Allways.
If the UT Server is configured to "serverTick=20" (just an example) then that means, that 20 times a second the UT server needs to calculate all playersmovements, positions, missiles etc and also send this data out to all players.
The UT Server is dependent
from its capability to calculate that in time.
Getting the results only 1/10 of a second later is TOO LATE.
Unfortunately , LINUX is built exactly the other way round: Its main aim is to be VERY STABLE. Thus if there is something the operating system does NOT like, is processes which bind a certain CPU Power very strict to itself... exactly what UT is trying to do! Instead of that the Operating System is GRANTING the UTServer Program CPU time.... If the UT Server Program needs a bit more CPUPower... bad luck! -> packet loss, high ping etc etc.
Futrthermore the Operating System always does its own tasks first, thats why LINUX is vers stable. So sometimes a really unneedet task such as cheking the print queue only to find out that there is nothing to print, takes CPU Cycles from UT... which needed onyl another 1/10 second CPU time to fullfill this cycles calculations.... but due to printerchecking failed.
Please note, that LINUX even grants only limited CPU cycles, even if the CPU is IDLE! ("no, you dont get CPU time now, you ugly process you. I will spare it so that if something important is happening I have CPU time at hand")
This was the reason why we had problems with more then aprox. 12 players, although the CPU was only used about 25% for UT and did NOTHING rest of the time. (1.4 Ghz Atlon)
To resolve this problem, You have to changed the setup of the server. The main difference is, that you have to patch the Servers LINUX Kernel to an "Preemptive" Process. This should allow the Operating System to give UT more CPU power, BUT it makes the Server somewhat unstable.
Also it helps toNICE the UTserver really high, and also RENICE it per CRON Job lets say all 10 Minutes. RENICING to -20 is good. This gives UT much mor CPU Time then nstalling it on an unpatched System. That gives more playes a better ping.
(b.t.w.: thanks to [CZ]MacMorty for helpfull hints)
OK folks, these have been my findings so far... so you see, a good server is NOT a matter of questions such as "what connection do you have"
Unfortunately. But m8ts... we will MASTER THIS!