View Full Version : Quick Update

September 16th, 2015, 09:08 AM
A quick update on the past week of work.

Most of the time programming is a pretty dry subject, and since I actually have an image on hand, I thought I'd take the opportunity to share it. The attached is what I call a "heat map"- the software is configured to walk through an entire route and take performance measurements. Red is on the slower end of things ("hot"), black is fast ("cool"). Analysing a route like this on a given device and for a particular set of settings takes a few hours. This image is from KSC on a low-end machine.


We can use these images to determine which routes and which parts of the routes need the most attention. We can compare how different settings affect this, and we can determine whether specific hardware matches up to our performance expectations. We can determine whether there have been any substantial changes between one build and the next. It all sounds good, and it is, but there are also a few downsides. The amount of time taken to run these tests can be prohibitive for some uses, for example.

Speaking of hardware variations, we've also spent a lot of time recently on tracking GPU failures on various configurations. Unfortunately, GPUs are years behind the rest of the industry in terms of gracefully handling failures. A simple mistake in our code, or a driver bug, can cause the GPU to freeze up or can shut the computer down completely. We don't get notified of the cause, and because the GPU runs asynchronously to the CPU, we can't trace the cause back to the offending line. This makes it impossible to track one-off events, but this week we've hit a few hardware configurations which fail reproducibly in only about 15 minutes testing. Having something reproducible like this means that we can try "random" changes to the program, to see which changes resolve the failure. Of course, these changes also break the gameplay completely, but that doesn't necessarily worry us while we're trying to track down the failures. Once we can identify a specific cause, or at least narrow things down enough to take some educated guesses, we are able to try refactoring the code in question to see if we can find an alternate path which both works correctly AND avoids crashing the hardware.

Sometimes, this can lead to other problems- a "necessary fix" on one GPU might hurt performance on another GPU, for example. In these scenarios, we may have to solve the same problem in several different ways, and then detect which one to use on your specific hardware when you start the game.



September 16th, 2015, 09:10 AM
It's the Fossil Fuels session, by the way. If you're bored, try to figure out what the obvious circle is. I'll let you know if you get it right. :)


September 16th, 2015, 12:04 PM
I'm not bored enough to load the map but my guess would be an Auran Duchess :-)

September 17th, 2015, 09:07 PM
You'd be wrong :)

Another tidbit: myself and our lead engine dev have spent most of the last week working out why the latest engine optimisations hard-crash one particular machine. Turns out that there's a fairly nasty driver-level (or OS-level, given that this is a Mac and the drivers are built into the OS) bug specific to this machine. This stuff is the frustrating, time-wasting side of games dev. On the upside, we've isolated what appears to be a robust workaround without sacrificing too much performance on that hardware, and filed a bug report with the upstream vendor.


September 17th, 2015, 10:21 PM
It's the Fossil Fuels session, by the way. If you're bored, try to figure out what the obvious circle is. I'll let you know if you get it right.

Frederick Falls? Sound is the only thing I can think of with a circular cutoff like that.

September 17th, 2015, 10:28 PM
The circle is a bit of a red herring; it's the viewing distance for the item in question. Inside that circle, you can see the item and are paying the performance cost (at least if you're looking in the right direction.)

I should also note that the correct answer is going to be more of an explanation of the performance loss than naming any singular asset.

enough hints for now,


September 18th, 2015, 06:40 AM
I'm guessing it might be the whole consist while its moving?

September 18th, 2015, 06:46 AM
I'm guessing it might be the whole consist while its moving?

Definitely thinking in the right direction, but no.


September 18th, 2015, 09:01 AM
The active view camera.

September 18th, 2015, 10:49 AM
hmmm, centered in the image, the compass? Any chance this function will be made available for route creators and others interested in optimizing their routes?

September 20th, 2015, 02:21 PM
Anything to do with fans, and critical-animation?

September 20th, 2015, 06:40 PM
It's not animation, but still thinking in roughly the right direction.