Asynchronous Route Streaming

For clarification as I have not yet purchased TRS 2019, has Asynchronous Route Streaming been implemented in TRS 2019 or is this still to be a future step?

I understand that calls related to it have been active since T:ane SP2 and I have been looking at updating ATLS etc, which will fall over if Asynchronous Route Streaming is active. As will, I suspect much of the way I use Trainz!


Thanks,

Boat

boat said:

For clarification as I have not yet purchased TRS 2019, has Asynchronous Route Streaming been implemented in TRS 2019 or is this still to be a future step?

I understand that calls related to it have been active since T:ane SP2 and I have been looking at updating ATLS etc, which will fall over if Asynchronous Route Streaming is active. As will, I suspect much of the way I use Trainz!

Click to expand...

* T:ANE SP3 provides all of the necessary APIs but does not offer route streaming itself. Scripts which are built for route streaming will work correctly, but will not experience significant performance benefits.

* TRS19 enables route streaming by default in key built-in routes, and allows developers to manually enable route streaming and related techniques across the board. We expect that a reasonable amount of heavily-scripted third-party content will experience compatibility issues with route streaming until it has been appropriately updated by the content creator. We expect that the majority of built-in content will be fully updated, though it wouldn't surprise me if we find and fix a few edge cases in eventual service packs.

* We expect to enable route streaming by default in the next release of Trainz.

* We expect to make route streaming mandatory for the second-next release of Trainz.


As with any future-looking statements, please take this with a grain of salt.

chris

Finally managed to glean more information on this subject, thanks to the OP. I've had this question in mind since first coming upon the topic last year. It seems to me that ARS assumes that a route will be traversed by a managed consist from A to B, so that it is probable that the route can be streamed board by adjacent board. (Which would apply even if the criterion for loading boards is based on draw distance, rather than position of the consist.) But what about the situation where the player (point of focus) moves abruptly, say to another consist in another part of the route, outside the present draw distance? This would mean that in effect that the existing stream would have to be abandoned at once and a whole new stream would need to be loaded at very short notice. Would there be a noticeable gap in the gameplay here?

I can see how ARS is viable where an episodic Session is being played, where much of the player's activity will be determined by the "plot" of the Session. But what about the situation where a Session is open - ie not plotted - where many AI consists are following predetermined schedules and the player could jump from consist to consist without warning?

I don't raise this question as a kind of game-breaker, but simply because ARS as I understand it at present could perhaps negatively affect my experience (and that of many like me who like to wander about in their own mash-ups) of Trainz.

boat said:

For clarification as I have not yet purchased TRS 2019, has Asynchronous Route Streaming been implemented in TRS 2019 or is this still to be a future step?

I understand that calls related to it have been active since T:ane SP2 and I have been looking at updating ATLS etc, which will fall over if Asynchronous Route Streaming is active. As will, I suspect much of the way I use Trainz!


Thanks,

Boat

Click to expand...

ATLS seems to be non functional in the latest TRS19 Beta..... will continue investigating.

Still waiting on a change list.

phimat said:

It seems to me that ARS assumes that a route will be traversed by a managed consist from A to B, so that it is probable that the route can be streamed board by adjacent board. (Which would apply even if the criterion for loading boards is based on draw distance, rather than position of the consist.) But what about the situation where the player (point of focus) moves abruptly, say to another consist in another part of the route, outside the present draw distance? This would mean that in effect that the existing stream would have to be abandoned at once and a whole new stream would need to be loaded at very short notice. Would there be a noticeable gap in the gameplay here?

Click to expand...

You're probably overthinking the word "stream" here. The route is broken into tiles, and we load tiles on demand. There's no penalty to any particular load order; the only gotcha is that there is a small latency between requesting data from a specific tile, and receiving that data. The simulation as a whole does not stop during that period, which is why existing scripts can run into problems - they expect that the data is either present, or that it does not exist at all. They don't expect to have to WAIT for the data to be loaded.

In most scenarios, we're able to predict what data is required well in advance of the user caring about it, so the user will likely never notice streaming occurring.

chris

Wndwalker – Thanks for that. At least I don’t need to panic immediately!

Clam – Perhaps I should panic immediately!

The 2 issues which worry me most when ARS is implemented are firstly concerning ATLS and similar. When an ATLS Controller is placed in Surveyor, the first thing it does is Broadcast a message to all other Controllers to ask them what Channel they are using. The latest placed Controller will then offer the next available Channel to the user. This obviously won’t work in its current form as some of the Controllers may not be loaded. (I presume ARS will operate in Surveyor too?). A solution might be to call for the loading of all ATLS Controllers if I can find them via GetGameObjectByID() or it may be possible to set ATLS to work through a Library where info can be stored. The latter being a kinder option!
There may be more of an issue when ATLS is run in Surveyor as I’m not sure how far Triggers etc would be loaded if the observer is some distance away. Can a train cross a Trigger that has been unloaded and thus not register? ATLS does a lot of message sending between map objects at the moment. Changing it to be a Rule or operating through a Library may reduce these instances but I’d still need to be sure the Triggers were all registering passing trains and not being unloaded or I can’t see how it would work.
Clam is seeing a problem. But from what you say Chris, that would only be due to streaming and unloaded assets in certain routes. Maybe there are other issues. I will get a copy of TRS 2019 soon.

The 2[SUP]nd[/SUP] issue is the way I play Trainz. I don’t just have one train I am watching to follow a pre planned scenario. I write my own AI for a route so lots of trains drive on their own. Thus I might be (say) driving a train in one area of the route and I would expect maybe another 100 trains elsewhere on the route to still be running, controlled by that AI. These trains may be hitting Triggers that again may not be there and trying to talk to items on the route that also are not there. So when I jump 50k to another part of the route, instead of things having run happily while I’ve been away, they are now going to be in a mess. Is that assumption correct or will script still run and trains still be plotted in remote areas, just without graphics?

Mike

Hi Mike.

A few comments from my own experience currently preparing EIT to support asynchronous object streaming :

1 - Broadcast messages have been declared obsolete since Tane SP2 and are currently no longer supported in their initial API, but N3V has added a new API obsolete LegacyBroadcastMessage(…) call to help scripters needing still to send broadcast message during the transition period. Broadcasted messages should be definitely no longer supported in one of the next versions of Trainz. So if you use broadcasted messages, you will need to update your scripts to continue operations …

2 - The easiest way to support new APIs is most often to create a library that all your assets (driver commands, rules, scripted trackside objects, … ) will use to request any operation on any object. When referencing an object, use only the new GameObjectID object reference, which is persistant during all your session from surveyor to driver mode, and your library can ask the system to know if an object referenced by its GameObjectID is currently loaded or not loaded. In case it is not loaded, and the library needs to retrieve or modify some of the object properties it can request the system to load asynchonously the object to be able to read or update its properties.
Another interesting point in using libraries is that libraries are gameobjects that supports threading, and you can under a library easily launch a thread (or request an already launched thread) to do asynchonously the processing needed on an object.

3 - about when an object is resident (loaded in memory) and when it might not be loaded :
=> remember first that all objects displayed on screen are by construction loaded ; the system can only render and display objects loaded. so objects visible are by construction currently loaded.
=> trains are always loaded, as the system needs at each simulation tick to adjust its position and may need to send or receive messages from or to the trains.
=> tracksides objects nearby any train are probably currently loaded. But as you cannot be sure, you will need to check the object is in memory and if not and you need to access its properties, you will need to request an asynchronous load …
So to answer one of your question, no a train cannot cross a trigger without loading it. When it approaches the trigger, the train has to send a message to the trigger and you can only send messages to objects currently loaded … So a train will load any objects that it will encounter along its way. But … after leaving the object, how long will the object remain in memory before being unloaded is uncertain. If you are running a compact route with a lot of memory available, probably the object will remain in memory quite a long time. If you run a huge route with some memory shortage, may be the object will be unloaded to make some place for other objects needing to be loaded ...

Globally, modifications are not so complicated for script programmers used to threads and asynchronous programming techniques. May be more difficult for people not having done any asynchronous thread processings.

If you need some help or have any questions, may be the best is that you open a new thread under the script section of the forum on this topic, where we can discuss about the new APIs and how to do common operations using the new asynchronous API calls.

Regards.
Pierre.

pguy said:

=> trains are always loaded, as the system needs at each simulation tick to adjust its position and may need to send or receive messages from or to the trains.

Click to expand...

An excellent summary, Pierre. I would only draw attention to this one point. While it is currently accurate, there is no requirement that it remains true in the future. There are a variety of possible internal implementations where I could foresee this assumption breaking:

* We might choose to only stream in Trains which are active. This doesn't really change anything in the current environment since an inactive train generally isn't interacting with its environment.
* In a multiplayer environment, we might choose to only stream in Trains which are relevant to the local player. We don't need to simulate the state of the other trains, because we know that we can download the latest state from the server when required.
* We might continue to keep everything loaded, but may accept a small delay between gameplay starting and some of the more distant Trains being loaded.
* In addition to streaming causing trains to load and unload, obviously the user can add and delete trains at runtime, as can Portals, etc. This isn't quite the same thing, but since you'll need to handle this, you'll already be doing a lot of the working needed to support streaming.

I'm sure there are other examples as well. My point is simply that the above assumption, while correct at the current time, is likely to become incorrect in the future and you should not rely on it. The APIs to access Trains are asynchronous, just like everything else, and you should assume that they can come and go.

Finally, I should point out that Trains are not Vehicles, and that it's possible to have a Train streamed in while the Vehicles are not yet fully streamed in. This is a bit of an edge case, but it can certainly happen (albeit briefly). Don't make the blind assumption that a Train is always fully described. The game handles this kind of "not quite loaded" state by briefly pausing parts of the simulation until the necessary data is available - ie. there's never an opportunity for a Train to roll backward down a hill because its loco hasn't been loaded yet, or etc.

chris

boat said:

A solution might be to call for the loading of all ATLS Controllers if I can find them via GetGameObjectByID() or it may be possible to set ATLS to work through a Library where info can be stored. The latter being a kinder option!

Click to expand...

As Pierre indicates, the library approach is a good option. The main thing that I would add is to point out that most scenarios don't actually require you to keep objects loaded once you've performed any initial setup. A route with 1000 signals might only have 50 in active use. Certainly, those 50 might be spread out over the entire route, depending on the location of the various Trains, but that's still a massive saving.

I can envisage such a library working like this:

* It is created in Surveyor.
* Upon creation (NOT just any initialisation), it begins (asynchronously) searching for specific object types.
* If necessary, it requests that the objects are loaded so that it can query information from them. This should only be performed where strictly necessary, because it will be time-consuming. You should expect that your ability to stream in objects will be rate-limited. In many cases, knowing that the objects exist may be enough, and you can postpone the detailed queries until a later time.
* Once you're done querying the objects, you store the GameObjectIDs where necessary, and release any references to the objects themselves. It's important to note that holding a reference to an object will NOT prevent it from being unloaded while your code is sleeping, so this is more for your protection than to optimise streaming.
* You use the async search APIs to monitor for changes, and update your stored data as necessary.
* You save and load your data in the library properties, rather than recreating it on each load.
* If you need to monitor existing object types, you use Sniff() via the GameObjectID rather than trying to use object references which may or may not exist, and may or may not remain valid.
* If you need to monitor custom object types, you might consider having them place library calls directly rather than passing messages, but either approach may be reasonable.

The 2[SUP]nd[/SUP] issue is the way I play Trainz. I don’t just have one train I am watching to follow a pre planned scenario. I write my own AI for a route so lots of trains drive on their own. Thus I might be (say) driving a train in one area of the route and I would expect maybe another 100 trains elsewhere on the route to still be running, controlled by that AI. These trains may be hitting Triggers that again may not be there and trying to talk to items on the route that also are not there. So when I jump 50k to another part of the route, instead of things having run happily while I’ve been away, they are now going to be in a mess. Is that assumption correct or will script still run and trains still be plotted in remote areas, just without graphics?

Click to expand...

Trains will still run wherever necessary, and the simulation will stream in any required data to make this happen. There are various triggers for streaming; this is build specific, but for example:
* We stream in the data immediately surrounding each Train.
* If a TrackSearch is performed and terminates due to reaching unstreamed data, streaming is triggered. This means that future TrackSearches will eventually receive the required data. In the case of Signals, the system is smart enough to re-evaluate the Signal logic when this occurs.

One thing that the streaming system does NOT know about is random objects placed around the route which might attempt to influence things far away. For example, if you create a tree scenery asset with a script which causes all Trains to stop moving when it is initialised, it is undefined at what point that will occur. In short, don't do that. Any logic which needs to be globally active should be in a Library or Rule instead.

chris

Pierre/Chris,

Many thanks for all that info. It is very useful.

So long as all trains still run as you say, and stream in the immediate data around them, then both ATLS and my AI, ASB’s should work albeit with changes. In Driver, the sequence for an ATLS/ASB event starts with a train hitting a Trigger. So long as trains approaching a Trigger will automatically load any relevant unloaded Triggers then a modified system should work. I’m a bit concerned about not all of the train necessarily being loaded but so long as the Triggers still detect a ‘train leave’ event then it shouldn’t matter.

ATLS (and ASB) need to have a train complete it’s sequence though the Triggers or the system will not re-set. Trains are currently identified using GetId() which has always meant an ATLS sequence had to be completed before using a Portal due to the Id change. The new GameObjectId which don’t change will actually be an advantage. As you say, working ATLS through a Library will solve a lot of the other obvious issues too.

So, a lot of script to be changed before ARS is fully implemented. I don’t have as much time for Trainz as I did 10 years ago so this may be a while! Still, it will keep me busy on rainy days.

Mike

boat said:

So, a lot of script to be changed before ARS is fully implemented. I don’t have as much time for Trainz as I did 10 years ago so this may be a while! Still, it will keep me busy on rainy days.

Click to expand...

Good luck! Keep us up-to-date with how you get along - I'm more than happy to give suggestions if you're having trouble with some aspect of this, and (given the relative immaturity of the system) it wouldn't surprise me if you run into some kind of edge case that we haven't yet provided an API for. The only way this kind of thing gets added is if we have a known use-case to support, so the sooner we find out, the better.

chris

clam1952 said:

ATLS seems to be non functional in the latest TRS19 Beta..... will continue investigating.

Click to expand...

Mike, you can stop panicking, bad route import was the cause, deleted and re-imported it from 96000 where it was working and all now working properly in Beta 96191

WindWalkr said:

Good luck! Keep us up-to-date with how you get along - I'm more than happy to give suggestions if you're having trouble with some aspect of this, and (given the relative immaturity of the system) it wouldn't surprise me if you run into some kind of edge case that we haven't yet provided an API for. The only way this kind of thing gets added is if we have a known use-case to support, so the sooner we find out, the better.

chris

Click to expand...

Hi Chris.... Well one thing that has quickly come to my attention is if/when the Broadcast message calls go, there will be a problem. While I can probably do the basic ATLS assets via a combined Library, there are an awful lot of 3rd party ATLS assets which rely on those messages. The system was always meant to be available to 3rd party assets and it didn't need to know they were there. (Lights, barriers etc)
Does the LegacyBroadcastMessage Pierre mentioned have any chance of becoming permanent? (And where do I find the full API list for 4.5 and above?)
Admittedly if these assets are not loaded it won't help but at least they might be able to sort themselves out when observed.

Thoughts?

Mike

Yep, you should definitely plan for the removal of broadcasts as a general concept. There are too many problems with them to keep them around long-term.

The good news is that you can support both library calls and broadcasts as a starting point, so your new code can still interoperate with the old assets.

chris

WindWalkr said:

* TRS19 enables route streaming by default in key built-in routes, and allows developers to manually enable route streaming and related techniques across the board. We expect that a reasonable amount of heavily-scripted third-party content will experience compatibility issues with route streaming until it has been appropriately updated by the content creator. We expect that the majority of built-in content will be fully updated, though it wouldn't surprise me if we find and fix a few edge cases in eventual service packs.

chris

Click to expand...

Hi,
Can you tell me which are the Key Routes that have streaming enabled? Kickstarter 2 seems the most likely candidate as it’s the largest TB 4.6 Route but when I load it into Surveyor I see no noticeable difference to a non streaming route. I’ve updated a large route of mine to 4.6 but again I detect no streaming in Surveyor. (What would I see anyway? MiniMap with no objects in the distance?)

I am close to wanting to test my re-written ATLS on a streaming route. Am I missing something obvious or is streaming more seamless than I was expecting?

I have Compatibility Mode set to ‘Stream Objects’. Is there something else I should do?

Thanks,

Mike

boat said:

Can you tell me which are the Key Routes that have streaming enabled?

Click to expand...

The "included" TRS19 routes all have streaming enabled regardless of user settings. In the current Early Access build, I believe this is basically limited to KSC2.

.. I see no noticeable difference to a non streaming route. I’ve updated a large route of mine to 4.6 but again I detect no streaming in Surveyor. (What would I see anyway? MiniMap with no objects in the distance?)

Click to expand...

Ideally, you should see no difference from any other route from the user perspective.

I have Compatibility Mode set to ‘Stream Objects’. Is there something else I should do?

Click to expand...

This will force streaming for all routes, not just the included routes. Nothing further is required. You can turn the settings up higher if you want additional improvements and/or debug feedback - this will assist you with diagnosing scripts that haven't been updated yet and so on.

chris

Hi Mike.

About testing streaming mode, a few comments from my own experience of testing EIT in streaming mode (next EIT version v54 that will be uploaded to DLS soon will support streaming mode ) :

1 - You need to test with a route of at least medium size or huge size, so that the system uses loading/unloading on track objects. If you test on a too small route with sufficient memory available on your computer, all objects are loaded and you will not detect any streaming load/unload operations. And you will not be able to test your scripts with unloaded objects on such configuration.

2 - With medium size or huge size route, you may need to wait a little to see some streaming operations with some unloaded objects. At initialisation, there may be some objects initialisation that will need to load some other objects to access its properties, and this may lead to have all objects currently loaded. After a while, you will see that unused objects not nearby any train will become to be unloaded. This may be particuliar for Interlocking Towers (on which I worked) as at initialisation an interlocking tower takes ownership of all its path objects, which needs to load them to take ownership on them … So,when I look at my IT Runtime monitor, even on huge route, at initialisation all IT objects are first loaded and later I see that IT objets no longer used by any train will be progressivly unloaded keeping only active IT objects in memory.

3 - I have not seen any way to enable streaming on only some specific routes. The performance option can be set to "compatibility mode" or to "stream objects" seems to be a global setting. But may be there is a way I do not know to enable streaming for a route or not for another one. But I don't know if it is realy usefull as the supporting streaming on one route means supporting streaming in all scripts the route is using (rules, library, … ) and these components are also global to a configuration.

Regards.
Pierre.

Chris/Pierre,

Many thanks. Much useful information. Looks like ECML would be a good test route!

I have done much head scratching over my new scripts over that last week. Largely because I wrote ATLS so long ago I can’t remember how it all works. There are still some bits I have no idea why they are there but I’ve left them in hoping I knew what I was doing when I wrote it!

Anyway, the Trigger, Controller and Slaves now seem to work as planned with the new Library in the basic ‘Level Crossing Mode’. I fixed the final problem last night so next is a script clean up, testing of the more obscure features, and seeing if it does what it should in a streamed route, plus eventually updating my other add-on ATLS assets.

The Library currently has a legacy ‘Message Broadcast’ feature to keep 3rd party ATLS assets happy for the time being , (barriers/signals etc) but these will need updating by their authors at some point.

One point to note is that any routes which have been built with pre-existing ATLS will have to have those assets updated in Surveyor before the new ATLS will work. This is just to add a ‘name’ to each ATLS asset. Needed because ‘AsyncObjectSearchResult’ requires a name to return a result. Consequently the ATLS Library will presume any ATLS asset without a name has been deleted and will ignore it.

I’m unavailable for the next couple of weeks but will continue after that.

Regards,

Mike

Chris/Pierre,

OK, I have now returned from a two week holiday and a week of jet lag!

Updating all my ATLS assets will take some time but I do now have a re-worked Controller, Slave and Trigger just about ready. These are the 3 ATLS assets ‘built-in’ to TRS19. Plus a new ATLS Library.

However, final testing in streaming mode is proving difficult. For the life of me I can’t see any evidence that streaming is working on my TRS19 copy, (build 96000)

I am using the ECML route. I have built a basic ATLS set-up with train, trigger and slave in London. The associated Controller is placed in Edinburgh.
Surely, when focused in London and the system set to ‘Stream objects’, the Controller should not be loaded? However, even using old ATLS and its broadcast messages, the system works fine with the Controller 400 miles away.
In Surveyor mode, a controller in Edinburgh will still communicate with a controller in London using Message Broadcast.
I have even left the session for 15 minutes to see if anything would unload but it still works.
So if I can’t get it to unload, you can see I’m having trouble testing to see if my new script will load it when needed.

Also, I thought the advantage of streaming was quicker load times. On my system it takes 1’:30” to 1’:40” to load an ECLM session. It’s the same time whether set to ‘Maximise Compatibility’ or ‘Stream objects’.

Any ideas?

Thanks,

Mike

boat said:

I am using the ECML route.

Click to expand...

To start with, are you testing in Driver or Surveyor, and what trainz-build number is this ECML route? Have you tried re-saving it so that it's actually in the streamed route format?

Secondly, try displaying the appropriate performance HUD for whichever module you're testing in (via the in-game settings). You should see details relating to streaming.

chris