Tonight’s major goal was to go to bed early. That… and to do weapon equipping! I could cheat a little bit because I have done this before and had a perfectly working shape mounting and mount node function from CastleGuard2. But setting up some data for the items, deciding how to pass it around efficiently, and assigning a weapon to myself by default was much trickier.

Here’s a little video of how it all worked out tonight.

Item Equipping Complete! (Basic version)

I’ve been continuing development on my RPG Kit framework, right now it’s still the process of building out the multiplayer and networking functionality. Here are the more significant developments:

Running Shiva Server on a Cloud (Rackspace)

Thanks to Eric Machala (emachala on the stonetrip.com forums/irc) I know how to setup a game server in the cloud. He posted a great How-To here. There weren’t any code changes, and one Shiva Server license is required per server instance. The great news for this is that the Linux (Ubuntu) server runs many times faster than the dedicated Win2003 server I have on GoDaddy. It can also be scaled up at any time with the click of a button on the rackspace.com website. To be clear, this is just the Shiva Server piece which performs the connectivity for clients and relays packets/establishes channels. There isn’t a way yet to run a dedicated Shiva app itself in Linux, though I’m sure it’s coming.

The cost of running a server is minimal: $0.015 per hour, and a bandwidth allotment of 50GB or so. You pay as you go and can see how much bandwidth you’re using, etc. Minimum cost is $10.95 per month on Rackspace, no setup fees or other costs associated with normal duties. Once you have the server setup the way you want, you can save an image of it and fire up new servers at will (provided you have the additional licenses for the Shiva Server). If you are ‘cursed’ with success and need to scale up rapidly, this is THE WAY to go!

Auto-Transferring Server

I’ve called this a few things in the past, but it’s another of the things which has stopped me from doing what I want to do in the big picture. Without the ability to run a dedicated server for logic which can run things like AI monsters and such, the only real options are to fire up Shiva clients, connect them to channels and scenes, and let them handle the AI duties. You need this if you want to have an authoritative way to make decisions for monsters, NPCs, or anything else which needs to ‘think’ for itself and can communicate its actions to all the connected clients. In addition, you’d need one of these per channel/scene, which could be dozens or hundreds of server programs running.

I’ve had the theory that if I could come up with a way for any single client in a scene to run the AI, I wouldn’t need to necessarily have these dedicated game servers running. The theoretical process requires that if a client that was handling server duties dropped off or disconnected, that we could fail over gracefully to another client and they could resume the server’s responsibilities. If I can accomplish this I don’t really need to worry about spinning up new game servers for logic, only for connection loads.

It’s been a very complex undertaking and I’ve tried it a few times and only managed to make a bunch of spaghetti code which is difficult to follow. I made a point to think about this in different terms this weekend though, and here’s how I went about it.

I asked some simple questions, instead of trying to establish an algorithm in pseudo code.

Q: What is a server?

A: A server runs AI and sends out messages to the clients who are interested.

Q: How can I reliably determine which client will be the server in a way that all other clients can agree with? (DONE)

A: This was a tougher question but ended up having an easier answer than I thought it would. There may be hidden issues but overall the basic precept works. Each client is given (by the Shiva Server) a numeric UserID which is unique. The server increments UserIDs as it runs. So the lowest UserID can be the server. When we get a notification that the previous user who was acting as the server has gone away, we can pick the next lowest UserID to be the server. If it happens to be me, I can immediately begin acting like a server.

Q: How can the new server pick up where the previous one left off? (DONE)

A: This is also tricky, but the basics are conceptually simple. When a server creates a new object that it will control, it sends out a message to all the other clients to create that object. When that happens, the clients tuck a reference to that object into a special table in the Server AI module, as does the real server. When the remote objects are deleted, that is also maintained. In the event that I am made the server, I already have a complete list of all the objects which must be maintained. An additional optimization here would be to know when the server last performed processing on its objects, so that the new server can pick up at about the same interval instead of too quickly.

Q: So how does the server process the objects in a way which doesn’t bog it down? (PARTIAL)

A: This is only a partial answer for now, but it works in initial tests. The server has a single routine which it runs through all its objects and processes them. This routine can be scheduled with a postEvent, or I can run it in a State Loop and examine a time differential in which to perform an update. Right now it is a postEvent.  The objects might have different AI or no AI but need to make certain checks. I’ll need to come up with a way to send the object a message which tells it to run one ‘tick’ of server processing, in which they can pick a new move location, pick a new target, etc.  The objects themselves will be responsible for sending out network updates, but this will probably also be done when the server tells the object to send out the update, so that it doesn’t overbear the client acting as the server.

The test

My initial test was to just verify the changing and transferring of the server as clients joined/dropped. I was pleased to see it happened pretty much instantly, and a notification shows up in chat that we are the new server, or that such and such user has become the server. From there, I worked in creating a remote object (a blue cube). The server runs its processing routine and simply moves the cube by brute force. On the other end, the cube is created but doesn’t move yet (it’s not networked after creation yet). When the original server drops off, the new server immediately begins moving the cube, and so on.

Next steps are to determine the cleanest way to tell objects to perform decisions and send out network updates so their movements and actions can propagate to all players. I’d like to use the existing AIEntity class to do this, and maybe just turn down the net frequency of the net updates selectively. I am in danger of making that class far too complex though, so I need to be careful. I could use an additional AI on the objects to do this stuff, but it feels messy.

One other additional improvement which is necessary now is to work in the client prediction based on previous velocity for objects. I’d thought I would need it and was avoiding it, but if I turn down network frequency on the server objects, the clients are definitely going to need to be able to predict movements in the absence of new move instructions.

Anyway, one by one I am tackling these things which have been holding me back, and this one is a doozy! So far so good, I’m one more step closer to a sensible large scale online game framework (I avoided the term ‘MMO’ there hehe)

When the process is more stable I will do a new code drop and documentation on it. I still haven’t begun to add too many game specific changes, except for a largish change in the move control and camera.

License to Use: You can use the code and tutorials presented within freely for your own projects. If you use anything here, I’d appreciate a credit, and also appreciate it if you drop me a line to let me know!

Networking Breakthrough!

(Updated project file at the bottom)

Tonight saw an amazing new breakthrough in my network technology. I still can’t believe I stumbled on this. My commute is about 2 hours each way when traffic is ‘normal’, and I’ve taken to thinking up ways to add various network optimizations. I do think I had something decent to begin with, and indeed I was very excited about it, but it definitely had lag problems and skipping from a jittery Internet.  The only good way to increase the smoothness was to turn up the fidelity.

This solution came upon me after I had explored many various permutations of network optimization. I was experimenting with ping delay, frame time, scheduling moves into the future, packet ordering, with nothing helping the lag except for some error correction. The packet ordering worked OK actually. I also experimented with various different combinations of SnapShots per second, and network updates per second.

This other solution which I keep mentioning was not difficult to code, and some obvious optimizations showed up for it too.

The simple solution

Instead of scheduling moves into the future (and overlapping them wildly), instead I stored them in a table (analogous to a list/queue) in Shiva. Then, at a regular polling interval, I read the first move from the table and delete it. I do this at the same interval that the move snapshots are captured at, roughly. If the moves in the table get to be too long, I increase the polling interval. If the move table is *really* long (>30 moves which would mean 3 seconds’ worth), I crank up the polling speed dramatically. This accounts for lag spikes.

I begin the polling as soon as we are given a network user ID. Check the Video

The first change is in RemoteObjectAI.unpackMessage:

 if(nMessageType==1) then
        --this.unpackMove ()
        --We used to unpack the move, but now we are just collecting it in a table
        local t = this.tMoveTable ( )
        table.add ( t, sMessage)
        return
    end

You will also need to modify your original move packet, no need to send out the timestamp difference anymore (saves some space too)

Here’s the ‘magic’ function!

function RemoteObjectAI.onGetNextMove (  )

--------------------------------------------------------------------------------
--Check the move table for existing moves. If more than one exists, we
--grab the top value, turn it into the current target position, and delete it
--This happens 10 times a second for a very smooth effect
local tMoves = this.tMoveTable ( )
local nNumMoves = table.getSize ( tMoves )
--log.message ( nNumMoves )
--No new moves, just schedule the next call into the future
--Also means we're not moving, so turn the speed down to 0
if(nNumMoves ==0) then
this.nCurrentSpeed ( 0)
if(this.hAvatar ( ))    then
object.setAIVariable ( this.hAvatar ( ), "AIEntity","nCurrentSpeed",this.nCurrentSpeed ( ))
end
this.postEvent ( this.nMovePollingInterval ( ),"onGetNextMove" )
return
end
--normal polling interval, at 10 times a second, should be the same rate they were captured at
if(nNumMoves < 10) then
--log.message ( "Setting polling to 0.1" )
this.nMovePollingInterval ( 0.1)
end
--Check to see if we're stockpiling too many. If so, speed up on grabbing them
if(nNumMoves > 15) then
log.message ( "Increasing polling to 0.075" )
this.nMovePollingInterval ( 0.05)
end
--If it's really long, speed up the polling interval
if(nNumMoves > 30) then
log.message ( "Increasing polling to 0.05" )
this.nMovePollingInterval ( 0.033)
end
--Explode the top move from the move table into the 'Message' table so we can get at it's data
local tMoveInfo = this.tMessage ( )
table.empty ( tMoveInfo )
local sMoveInfo = table.getFirst ( tMoves )
table.removeFirst ( tMoves )
--Turn this into a nice neat table
string.explode(sMoveInfo,tMoveInfo,";")
local stringtoNumber = string.toNumber
local tablegetAt = table.getAt
--Should be 1, which is a move event
--local nEventType = stringtoNumber(tablegetAt(tCurrentMessage,0))
local x = stringtoNumber(tablegetAt(tMoveInfo,1))
local y = stringtoNumber(tablegetAt(tMoveInfo,2))
local z = stringtoNumber(tablegetAt(tMoveInfo,3))
local ry = stringtoNumber(tablegetAt(tMoveInfo,4))
local nSpeed = stringtoNumber(tablegetAt(tMoveInfo,5))
--Update the new target coordinates
this.nTargetX (x )
this.nTargetY ( y)
this.nTargetZ ( z)
this.nTargetRY ( ry)
--Original Method
--this.nTimeStep ( 1/this.nFrameCounter ( ))
--FPS Comparison Method
--Figure out how many frames on average we need to interpolate over
--nDiff is the total time between move packets. Find a value to interpolate
local nLastTime = application.getTotalFrameTime ( )
local nDiff =  nLastTime - this.nLastMovePacket ( )
this.nLastMovePacket (nLastTime)
local nMyFrameTime = application.getLastFrameTime ( )
local nNumFrames = nDiff / nMyFrameTime
this.nTimeStep ( 1/nNumFrames)
--Figure out how many frames on average we need to interpolate over
this.nTimeStep ( 1/this.nFrameCounter ( ))
this.nFrameCounter ( 0)
this.nMoveFactor ( 0)
--Speed change? If so, tell our attached avatar how fast we're travelling
if(nSpeed ~= this.nCurrentSpeed ( )) then
if(this.hAvatar ( ))    then
object.setAIVariable ( this.hAvatar ( ), "AIEntity","nCurrentSpeed",this.nCurrentSpeed ( ))
end
end
this.nCurrentSpeed ( nSpeed)
this.postEvent ( this.nMovePollingInterval ( ),"onGetNextMove" )
--------------------------------------------------------------------------------
end

And here is the updated project file. If you come up with improvements, please share!

Full Project Archive: RPGDemo-NetworkBoost-20110118