AMD or Intel better?

I'm running an Asus Crosshair V Forumula MOBO with an AMD FX8350 Vishera clocked at 5Ghz with water cooling and a total of 64GB of memory. Trainz runs fine however there are always opportunities for improvements. I haven't done it in awhile, but I need to run an XPerf set on the system while Trainz is running to see where the bottlenecks are. I have 4TB Seagate drives, 7200RPM so I/O at the disk isn't a problem but yes, a 32 bit app won't be able to address beyond 2GB of Address Space, but that doesn't seem to be an issue. I see Trainz hovering anywhere from 400 to 600MB of use but that doesn't show the Virtual Address (VA) use. I'm also running in windowed mode as well because Windows 8 + Trainz in Full Screen just doesn't work for me and I need to multi-task sometimes. I have a dual SLI (Radeon 7900 series GPUs) so graphics isn't a problem either.

Now if we could get N3V to build this as a 64 Bit app, we could just suck the assets into the VA space and page it out as the route progresses, eliminating the back and forth between the main app and TADDaemon. Also TADDaemon appears to do a lot of idle thumb twiddling, validating assets that are already validated.. it's not a CPU waster but it's still there. Also, Trainz communicates locally with TADDaemon via TCP/IP sockets, vs a shared memory cache. That would make the asset loads and lookups hella fast...

:)
 
I... Also, Trainz communicates locally with TADDaemon via TCP/IP sockets, vs a shared memory cache. That would make the asset loads and lookups hella fast...

:)
Interesting post. I wrote, and maintained for a few years, programs that communicated via named pipes either on the same computer or across a LAN. This was aeons ago before sockets got popular. How would two or more programs doing IPC manage a shared memory cache?

Sorry for the off topic post but this got my interest.
 
Last edited:
Excellent question.. Here's a couple of articles on two different ways of doing that. One is using memory mapped files and another is using shared memory with queue structures. In essence you can use signalling mechanisms such as Events, Mutexes or Semaphores to gate access to the data structures you define and peer processes using those structures can exchange information. For example, you could create a simple LIFO queue in a shared memory segment and use a Mutex to gate writing to it and reading from it. To attain real speed you may have each process with its own queue of requests lets say and they signal the availability of information with an Event. The Server, waits for that event and then traverses that queue looking for work to do.. It uses a bit in the queue structure to flag that it's processed the request, allowing the Client to remove it from the list. So if you think of asset requests.. Queue up the entire Asset Map and all dependencies and then signal the Server to get them from the DB/File system. The Daemon then flags the client once the request is done. That's just an example but I think you can see that since you're just dealing with address space and simple signalling mechanisms, the performance gains can be significant.

Here's some sample code and a full explanation of one way, just using shared memory on Windows.

http://www.codeproject.com/Articles/14740/Fast-IPC-Communication-Using-Shared-Memory-and-Int


Then, there's the Boost Interprocess library which provides an abstraction layer if you're building for multiple platforms, i.e., Unix, Windows, MacOS etc. On Windows it uses a Shared File Mapping which is another IPC mechanism.

http://www.boost.org/doc/libs/1_47_...ess.sharedmemorybetweenprocesses.sharedmemory


I've worked on quite a few HPC projects and in terms of synchronization and data exchange. If you're on the same box, Shared Memory IPC is the fastest. TCP/IP sockets are great for portability but not from the standpoint of performance. You have to traverse all the way up the IP stack and if you're using TCP rather than UDP you have all of the same things in terms of buffers, sliding windows, connection establishment and tear down all of which slows things way down. Think of it this way, the packets go through all of the same paths as any network traffic with the exception of not being delivered to the Data Link or PHY layers.

Another mechanism that's portable is using Named Pipes for IPC. You have a data structure where you can shove data in at one side and somebody at the far end receives it. Using this technique is sometimes 5 to 10 times faster than TCP/Sockets on the same system for a given workload and operating system.

Another idea that could be used is in-memory caching of all assets. When an asset gets read from the DB/File System, you cache it in address space. Then you create an access / age timer on it and push out older cached objects from memory, pulling them back in when necessary. Depending on data structure size (domain) this could require lots of virtual address space, beyond 2GB afforded by 32 bit applications. Now switch over to 64 bit and you could cache a lot of information. In terms of space I have about 20GB of asset data right now in my Trainz 2012 installation. If you allocate a 4GB cache (And most routes/scenarios wouldn't consume that) You could essentially eliminate the need for IPC altogether. First, search for asset, if it's in the cache (some hash algorithm determines it based on KUID) then read from address space, otherwise go load it from disk. Periodically you go through the cache to see the access counters of the objects. If they haven't incremented since the last time you looked, flush them out of memory.

Anyway, I hope that clarifies what I was bringing up.
 
Also TADDaemon appears to do a lot of idle thumb twiddling, validating assets that are already validated.. it's not a CPU waster but it's still there.

One thing it definitely is, it's a bloody time waster, sitting there for tens of minutes just validating crap that it's already installed and committed.

Regarding the TCP/IP vs shared memory thing, Windwalkr has claimed that this is not a bottleneck in Trainz.

http://forums.auran.com/trainz/showthread.php?103371-Two-Suggestions-64-Bit-Support-IPC-DB&p=1172819#post1172819
 
Last edited:
One thing it definitely is, it's a bloody time waster, sitting there for tens of minutes just validating crap that it's already installed and committed.

Regarding the TCP/IP vs shared memory thing, Windwalkr has claimed that this is not a bottleneck in Trainz.

http://forums.auran.com/trainz/showthread.php?103371-Two-Suggestions-64-Bit-Support-IPC-DB&p=1172819#post1172819

TADDaemon is not wasting time. The purpose of a daemon is to run in the background and monitor events and act on them when needed. This is a common application, or process, found on many systems and especially those that use a database, and is called by different names including a daemon. This terminology is more common on Unix and Linux then it is in the Windows Environment. The bigger database systems such as SQL, and Oracle 12g run a daemon, or background process, for many of the same reasons that we run TADD, and stopping those would have the same effect as it would if we were to stop TADD from running.

Remember our Trainz content is kept in individual files that need to be organized so Trainz its self and us can find easily. Even TRS2004 had a database. In that version, items were loaded and cached in the old .CHUMP files which would need to be deleted manually periodically or when items were modified. When content was repaired, or created and added, the .CHUMP files would have to be manually deleted prior to running Trainz. When the program was started up, it would then reload the content from disk and populate the menus we have in Surveyor. The old creation process, as simple as it sounds, was quite risky with manual file deletion. With the way the things are now, TADDaemon does that work for us as it checks and loads the content on the fly once the content has been installed into Trainz using Content Manager.

If you were to run the command consoles, you'll see TADD doing lots of transactions in TS12 SP1 and up. Perhaps these processes were going on all the time and the Service Pack has made this more verbose so we see what's going on. When a DB Repair has been done, and then Trainz is started afterwards, you'll see TADD verifying content and decrementing a number while individual TrainzUtil processes open up and close.

John
 
TADDaemon is not wasting time. The purpose of a daemon is to run in the background and monitor events and act on them when needed. This is a common application, or process, found on many systems and especially those that use a database, and is called by different names including a daemon. This terminology is more common on Unix and Linux then it is in the Windows Environment. The bigger database systems such as SQL, and Oracle 12g run a daemon, or background process, for many of the same reasons that we run TADD, and stopping those would have the same effect as it would if we were to stop TADD from running.

Remember our Trainz content is kept in individual files that need to be organized so Trainz its self and us can find easily. Even TRS2004 had a database. In that version, items were loaded and cached in the old .CHUMP files which would need to be deleted manually periodically or when items were modified. When content was repaired, or created and added, the .CHUMP files would have to be manually deleted prior to running Trainz. When the program was started up, it would then reload the content from disk and populate the menus we have in Surveyor. The old creation process, as simple as it sounds, was quite risky with manual file deletion. With the way the things are now, TADDaemon does that work for us as it checks and loads the content on the fly once the content has been installed into Trainz using Content Manager.

If you were to run the command consoles, you'll see TADD doing lots of transactions in TS12 SP1 and up. Perhaps these processes were going on all the time and the Service Pack has made this more verbose so we see what's going on. When a DB Repair has been done, and then Trainz is started afterwards, you'll see TADD verifying content and decrementing a number while individual TrainzUtil processes open up and close.

John

Agreed, that was why I put in one of the suggestions why not another DB engine? One that you could actually do some diagnosis with outside of N3V/Auran tools? Something a little more open that provides documented mechanisms for troubleshooting and analysis. It seems the lowest common denominator in problem resolution is "rebuild the database" or "re-install" and I'll have to admit I haven't seen much granularity in between. Maybe somebody else can point me to some other documentation on what to do when things go pear shaped with the DB or the asset directory tree?
 
Last edited:
Hi John,

If you could so kindly further enlighten me as to some behavioral aspects of TADD, that would be great. I'm perfectly alright with the database doing what needs to be done (ie. validating and organizing assets), heretofore referred to as "essential stuff". The "wasting time" part comes when I have left CMP alone all day to take it's own sweet time to perform essential stuff. It appears to be done because it's just sitting there doing nothing. I try to get back to work on some content or maybe apply a search filter, CMP jumps back into action and does that validating thing again. In the meantime, nothing can be done. Why can't it just settle the essential stuff at one go, however long it takes? This I do not understand.

Similarly, let's say a bunch of trains are installed. CMP appears to be done with essential stuff. So the user fires up trains and open up the route and tries to open Railyard. Oh no. Trainz appears to have frozen! And there it is, TADD doing that validating crap again. Stays this way for at least 20 minutes. Something that could've been gotten out of the way during the process of installing or committing and it had all day to do it, but noooo, as usual Trainz is bent on wasting as much of the users' time as practically possible.

TADD may not be wasting time, but TADD doing things at all the wrong times is sure as hell wasting everybody's time.
 
Last edited:
..

TADD may not be wasting time, but TADD doing things at all the wrong times is sure as hell wasting everybody's time.
I've not noticed this and my major Trainz activities are building in Blender and testing in a Trainz session. So I usually have CM running, Blender, AssetX, Context and probably an image editor as well. Not to mention the "usual suspects" of Chrome, Outlook, etc.

Whenever I paste an amended asset into CM it reacts instantly. Committing assets is not slow unless the asset is very large. I was testing another user's asset that had several very large image files and it was taking nearly two minutes to commit. My assets commit quite quickly.

Windows has lots of background processes so, as far as I am concerned, TADDaemon is just another.

But I will watch TADDaemon to see what it is up to. Right now it is regularly putting out a message roughly every 2 mins and 2 seconds. But I don't know if it is actually doing anything or just saying that it is "alive and well".

For wkwood. Thanks for another interesting post. It seems the same issues and topics are still being discussed after I first got involved with such things in the early 80's. :)
 
Hi John,

If you could so kindly further enlighten me as to some behavioral aspects of TADD, that would be great. I'm perfectly alright with the database doing what needs to be done (ie. validating and organizing assets), heretofore referred to as "essential stuff". The "wasting time" part comes when I have left CMP alone all day to take it's own sweet time to perform essential stuff. It appears to be done because it's just sitting there doing nothing. I try to get back to work on some content or maybe apply a search filter, CMP jumps back into action and does that validating thing again. In the meantime, nothing can be done. Why can't it just settle the essential stuff at one go, however long it takes? This I do not understand.

Similarly, let's say a bunch of trains are installed. CMP appears to be done with essential stuff. So the user fires up trains and open up the route and tries to open Railyard. Oh no. Trainz appears to have frozen! And there it is, TADD doing that validating crap again. Stays this way for at least 20 minutes. Something that could've been gotten out of the way during the process of installing or committing and it had all day to do it, but noooo, as usual Trainz is bent on wasting as much of the users' time as practically possible.

TADD may not be wasting time, but TADD doing things at all the wrong times is sure as hell wasting everybody's time.

This is exactly what I was explaining. It will sit there idle until it's called to action.

The verification process is due to a database action of some sort such as an install and perhaps a database repair. This usually doesn't happen until the database is called to action such as clicking on a menu in Surveyor, or starting up Trainz its self. Yes, TADD and even TrainzUTil will kick into action when Trainz is running and not just while using Content Manager.

During this time, especially if the system is busy, it can cause performance problems with Trainz. I'm not saying this is a good thing - just saying that it's doing it this. Once the verification process completes, the performance is usually okay. If you look at the messages in TS12 SP1 HF3, you'll see the number become less and less as the process completes along with TrainzUtil popping up messages windows and closing as it completes a verification of a some assets.

John
 
"Called to action", "kick into action" what? Verification? Is there a good reason why this cannot be done when installing or committing the asset? I didn't "call" it into action just as I'm ready to start Trainz, I expected it to be done with said action a long time ago. This is like bringing my car into the shop to have new headers or rims put in, leave it with the chaps for a couple of days and they finish the job in one afternoon. Then when I go to pick it up they say "oh sorry mate we've not put the engine back in or the wheels back on so please sit in that corner and wait".

Yeah.
 
Well last night I received an e-mail from Felix at @ auran support. Anyway, he had me try a couple of things with the Murchison problem but while re-installing the CDPs there was TADDaemon screaming away on local ports. I turned off TCPView after it was over 32MB of exchange between Trainz.exe and TADDaemon.exe. So while it's "not a problem" it's sure slow.
 
This is exactly what I was explaining. It will sit there idle until it's called to action.

The verification process is due to a database action of some sort such as an install and perhaps a database repair. This usually doesn't happen until the database is called to action such as clicking on a menu in Surveyor, or starting up Trainz its self. Yes, TADD and even TrainzUTil will kick into action when Trainz is running and not just while using Content Manager.

During this time, especially if the system is busy, it can cause performance problems with Trainz. I'm not saying this is a good thing - just saying that it's doing it this. Once the verification process completes, the performance is usually okay. If you look at the messages in TS12 SP1 HF3, you'll see the number become less and less as the process completes along with TrainzUtil popping up messages windows and closing as it completes a verification of a some assets.

John

First, why verify something that's already been verified before? If you've validated an asset, then there's no need to reverify it? I'm talking about a non-repair situation here.

Second, You can watch TADDaemon using any tool you want, TCPView, PROCEXP etc and it's not idle, it will start up start churning away and start making Internet connections on your behalf. I have four assets right now with missing dependencies and oh, I hit a great new bug but I'll share later, and none of them are in the routes or sessions I'm using with Trainz, so why all the activity? If you open the process windows and watch it, there's a constant stream of validation going on and that just sniffs of a problem trusting the previous validations it's already done.
 
Back
Top