PDA

View Full Version : one for the programmers: file format parsers



WindWalkr
July 5th, 2015, 11:39 PM
For the programmers in here; there's an experimental project here (http://www.typhoonsystems.com.au/formal-manual/) which can peek inside some of the Trainz file formats, and is extensible. Definitely not suitable for typical end-users at the current time.

chris

BuilderBob
July 6th, 2015, 01:27 AM
For the programmers in here; there's an experimental project here (http://www.typhoonsystems.com.au/formal-manual/) which can peek inside some of the Trainz file formats, and is extensible. Definitely not suitable for typical end-users at the current time.

chris

Interesting. Chump and BMP work, but texture throws an error.

WindWalkr
July 6th, 2015, 01:38 AM
Make sure you're pointing it at a non-payware TANE texture file. TS12 binary textures are not currently supported. Payware textures will never be supported, for hopefully obvious reasons.

chris

pcas1986
July 6th, 2015, 03:56 AM
Like BuilderBob I managed to get it working with a chump file I found. It didn't like one particular bmp file I chose.

It would be nice to have an IM and KIN file grammar file for the reasons I mentioned a while back.


My virus checker really doesn't like this program and wants to keep quarantining it. It thinks it is exhibiting suspicious behaviour.

BuilderBob
July 6th, 2015, 03:58 AM
Make sure you're pointing it at a non-payware TANE texture file.

That makes a difference.

So is this a potential basis for a standalone asset analysis tool?

WindWalkr
July 6th, 2015, 05:00 AM
Not really. At least, not in the sense of "bring up the asset on screen and show off the performance metrics" - that will need to be done in-game.

If you're talking about "look deep inside the file to see what actual data is present and whether something unexpected is happening", then sure- that's what it does. It's a bit above the level of a hex editor.

chris

RPearson
July 6th, 2015, 09:47 AM
Chris I wasn't going to bring up file formats right away since it wasn't an urgent topic specific to TANE but I did intend to start a separate thread at some point in time in this forum on this subject.

However since you've brought it up, back in 2009 you discussed the TrainzNativeInterface.dll as a possible wayN3V could provide read and possibly write access to various file formats used in Trainz. Since then I've heard nothing more about it although there is such a dll included with the distribution. The TrainzWiki gives no info on what's in it (see quote below) or the header or other source info needed to access it. Is this the dll that was invisioned back in 2009 for 3rd party access to formats and other parts of the game? Can we obtain the sources necessary to use it?


The TrainzNativeInterface is a helper DLL and associated protocols which allow communication between Trainz executables and third-party DLLs. This page describes a work-in-progress and all details are subject to change.

Linkage

The trainznativeinterface.dll is provided in the Trainz bin folder, and is hard-linked by relevant executables. It is expected that any plugin DLLs will also hard-link the trainznativeinterface.dll such that a single shared instance of the DLL will exist in the parent executable's process space.

Any plugin DLL must use be built in MSVS 2005 (or compatible) and use the "Multi-threaded DLL" runtime. All Auran-supplied interface source is provided in C++ header format, however the trainznativeinterface.dll uses POD classes and global functions such that other languages should be able to link against the DLL if an appropriate interface is built.

Plugin DLLs should not make assumptions about their host environment. The TrainzNativeInterface is not limited to use by the Trainz game executable.


As you may or may not be aware I'm most interested in mapfile formats. Sometime in the distant past they were made available to 3rd party programmers. For whatever reason I've not been able to obtain a complete set since the 1st version of TRS06 was issued and since then all my requests for them were put off by the helpdesk in one way or another. TransDem is the best example of what can achieved using this info. Dr. Ziegler started some time after me but he's managed to hack out the mapfile changes with every Trainz version since TRS4. I gave up at TC1. It would certainly be much easier and safer (for programs writing the formats) if the formats were made available either in written form or thru software such as that once proposed for the Trainznativeinterface.dll.

Any comments or thoughts on this would be appreciated.

Thanks for the link to formal program.


It's a bit above the level of a hex editor.
I use winhex and have been happy with it. But this is clearly setup to work with Trainz to read various formats. I haven't gone thru the docs yet but it appears to be very extensible. Building the cx files seems to be the key. It worked with a .tzarc file and the chump and other files contained in the compressed file.

Thanks,

Bob Pearson

WindWalkr
July 6th, 2015, 11:03 AM
However since you've brought it up, back in 2009 you discussed the TrainzNativeInterface.dll as a possible wayN3V could provide read and possibly write access to various file formats used in Trainz. ... Is this the dll that was invisioned back in 2009 for 3rd party access to formats and other parts of the game? Can we obtain the sources necessary to use it?

Short version: yes. Longer version: we have a number of tasks ahead of us before that can happen. Exposing native interfaces is simultaneously very powerful, and can be very problematic. Specific issues that we need to address:

* Creating necessary interfaces to the various Trainz components, making sure to given enough access to be useful but not so much access that we can't upgrade the game without breaking the interfaces.

* Ensuring that we maintain a sufficient degree over control over what can be plugged into the interface so that we don't end up with a raft of compatibility issues that we can't do anything to resolve, or worse, introduce substantial performance, stability and security holes.


Physics, Input, and the Surveyor Interfaces are obvious candidates here.




As you may or may not be aware I'm most interested in mapfile formats. Sometime in the distant past they were made available to 3rd party programmers. For whatever reason I've not been able to obtain a complete set since the 1st version of TRS06 was issued and since then all my requests for them were put off by the helpdesk in one way or another.

To put it bluntly, we don't maintain file format documentation in the sense that you're looking for. We don't have interoperability with other products to worry about, so the code serves as its own documentation. Good documentation that's usable by outsiders is difficult to maintain- it really needs to be a two-way communication, with faults identified and reported. It is my intention to use cxlang/Formal for this because it is self-testing (ie. if the grammar is incorrect, it will fail in obvious ways when used against a real route.)




Thanks for the link to formal program.

I use winhex and have been happy with it. But this is clearly setup to work with Trainz to read various formats. I haven't gone thru the docs yet but it appears to be very extensible. Building the cx files seems to be the key. It worked with a .tzarc file and the chump and other files contained in the compressed file.

cxlang/Formal is nothing to do with Trainz, it's just a side aspect of a pet project of mine. It's just a fairly easy way to take a peek inside the file formats.

chris