PDA

View Full Version : What does ! LoadE2MeshObject> 32 chunks in IM.txt file: bb_bogey_lod2.im mean?



pcas1986
June 5th, 2015, 03:59 AM
I'm getting a bunch of warnings of this nature and, since we know that warnings can become errors, it would be useful to know what it means and what we can do about it.

p.s. a similar question was asked in one of the TANE related threads but there was no response to this particular warning.

clam1952
June 5th, 2015, 04:56 AM
That's one I'd like to find out about as well, rather solve it sooner than have to do major surgery somewhere in the future.

WindWalkr
June 5th, 2015, 06:18 AM
Okay, this is an easy one.

"chunks" are a collection of polygons which are rendered with a single Trainz material. There are upper bounds on the number of vertices per chunk. A material is not "t.bumptex" but rather a specific material instance which includes lighting settings and textures. (I think that these correspond to sub-materials in Max parlance, but I'm not a Max user so somebody who knows better can correct me..)

There is a performance overhead per chunk (per *anything* pretty much, but chunks are one of those that we care quite a bit about.) A 30k polygon mesh in a single chunk is actually fairly fast to draw in most cases, whereas a 3k polygon mesh spread over 20 chunks is quite slow to draw. If the mesh is subject to stitching and there are many instances in the scene, you may make some of this back through stitching, although the effort of performing the stitching will still be higher than necessary.

The simple version is that you don't want to have more than one chunk per 30k polygons (and since some people have been picky about this lately, I'll reiterate that the only polygons Trainz uses regularly are triangles. If you used a higher-order polygon in Max, it was converted to triangles when you exported.) It's not always possible to achieve this, but that should be your goal.


Places where excess chunks come from:

* Too many polygons.
* Too many materials (typically correlated with two many textures, although technically these are separate issues.)
* You can also have too many chunks by using multiple IM files, each with a smaller number of chunks.


This isn't an error condition (within reason; please don't try to use hundreds of chunks ;p ) but it is a warning so that you know your performance is likely to be very low.

chris

clam1952
June 5th, 2015, 06:42 AM
Thanks, that makes sense to me. Some of my earlier stuff, most not on the DLS thankfully does suffer from too many materials / textures, caused by looking at what others had done and following bad practices, sometimes really does pay to RTFM first! Was a few years back now but In the process of redoing them properly.

pcas1986
June 5th, 2015, 06:56 AM
Thank you, although I'm going to need to read your response a few times to fathom out a way of solving the problem. Here is output of a "view errors and warnings" of a bogey.

http://i1188.photobucket.com/albums/z410/casper131/Ashampoo_Snap_2015.06.05_21h42m04s_005__zps2x3ai9t o.jpg

The second lot of warnings seem to repeat the first. The 500 poly warning I'll ignore until your fix for that comes out and if it is still there then I'll think about it. The low level mesh does have a very high poly count but the bogey does get culled. The 20% warning I can address.

This is a 10 wheel bogey so the poly count is always going to be high but I'll see what I can trim. The LOD0 version is 12,602 tris. There are two materials: a tbumpenv and a reflect. The reflect I can probably discard.

I guess what we need in simple terms is how to address the problem.

WindWalkr
June 5th, 2015, 08:06 AM
I guess what we need in simple terms is how to address the problem.

Stop using so many materials per mesh file?

Sorry if that sounds obvious, but.. there's nothing really fancy going on here. The warning indicates that the mesh is getting excessive with the number of chunks, and the solution is to build in a more efficient manner.

For example, where you previously used a separate texture for each wheel, and each component of the bogey, you should probably create a single (probably larger) texture which represents everything. The only scenarios in which this approach doesn't really work are:

* Cases where you need to tile the texture both horizontally and vertically. In most cases, this isn't actually necessary. Small amounts of tiling can be accomplished with geometry. Larger amounts of tiling tend to look bad anyway.

* Cases where the overall texture size is simply too large. Options here include reducing your texture resolution, texturing more efficiently, or splitting into some number of smaller textures. If you have a case where you really need a *lot* of texture detail, but need almost no geometry detail, then in theory you may not be able to resolve this warning. I don't expect such cases to actually happen.

chris

WindWalkr
June 5th, 2015, 08:10 AM
I should also note that older style (rigid body) animations may also potentially blow out the chunk count; you may end up with an additional chunk per bone on top of the above requirements. For scenarios with only two or three bones, this is poor form but I don't think you'll trigger any warnings. If you're talking about large numbers of bones, this can be substantially below ideal.

chris

WindWalkr
June 5th, 2015, 08:14 AM
This is a 10 wheel bogey..

As per the above, if you're using rigid body animation, this is probably adding to your overall counts quite substantially for no gain.



The LOD0 version is 12,602 tris. There are two materials: a tbumpenv and a reflect. The reflect I can probably discard.

With two materials and skinned animation, 12.6k polygons, we should be looking at 2 chunks. Anything above that is pure wastage.

chris

pcas1986
June 5th, 2015, 09:16 AM
As per the above, if you're using rigid body animation, this is probably adding to your overall counts quite substantially for no gain.
...
With two materials and skinned animation, 12.6k polygons, we should be looking at 2 chunks. Anything above that is pure wastage.

chris

Now we're getting somewhere. The exporter log has this line:

". Info: Forcing Rigid Skeleton due to material types used"


I've used a tbumpenv and a reflect - which of those is causing the problem? Or is it both? What is skinned animation?

There are 8 armatures and 16 bones in the animation.

VinnyBarb
June 5th, 2015, 05:00 PM
This is interesting stuff, I get similar "chunk" warnings with my TS12 created locomotives and bogies. I will be waiting for more info of how to solve all this "chunk" warning/error stuff.

Would one solution be, to get the chunk warnings down, to break up the main mesh of a locomotive into several smaller meshes and assemble these via attachment points? At the moment, I do have quite a few sub meshes included in my main mesh, just moved into place where these should be but not attached to any other mesh in 3DS Max but these are just differently named meshes. Most are texture mapped from only 2 texture maps and their corresponding normal maps used with possibly only 2 to 3 different material types for these textured meshes used. I do have separate texture maps for night mode and alpha numbering, which I have to use and I guess can not be included on any main texture map.

Meshes like the platform mesh, fuel tank, air bottles, engine box etc., which are not attached to/with each other, just moved in the right place where these should be in 3DS Max. Then there are the night mode mesh(es), alpha number meshes etc. and finally all attached meshes via attachment points like animated fans, drivers, cab and others. Even then I get these "chunks" warnings although some of these non attached sub meshes are small. Then cumulative exported as individual .IM meshes, one for the main locomotive, separately exported as night mode, auto numbering and so on. Like I guess every content creator might be doing similar.

Anyway, I will wait with any alterations and upgrades of the above to TANE 4.2 until I know more of these warnings and how to get rid of these and you people bring out a reasonably performance improved TANE version to work with.

I am still wondering why all this tough crackdown on all these things like above is really necessary, you programmers MUST have serious performance optimization issues with TANE or you would not now do all this very strict error checking and restrictions etc., IMHO sometimes looking like being too much penalizing us content creators for the sins of previous and current coders of Trainz or possibly a non capability of understanding of how to solve or at least get to grips with this optimization progress for TANE on your end. Why you might ask I say this? Well, it makes me wonder why TS09, TS10 and TS12 would work at all and run and perform as well as these still do without any such tough restrictions placed upon us content creators in the past with the same created assets we now try to get error and warning free working in TANE.

I am not knocking you people, I am just wondering.

VinnyBarb

Pencil42
June 5th, 2015, 05:27 PM
The exporter log has this line:

". Info: Forcing Rigid Skeleton due to material types used"


I've used a tbumpenv and a reflect - which of those is causing the problem?

Is this Blender? Torsten's exporter forces rigid skeletons on items with bumpmapped materials due to bugs with skinned animation and bumpmaps - so it would be the tbumpenv material. There was a discussion on this a while back, let me see if I can find it...

Curtis

norfolksouthern37
June 5th, 2015, 05:34 PM
i have seen this at least in a few items. one of our locomotives has visible opening and closing shutters in the cooling system, TANE says "! LoadE2MeshObject> 104 chunks in IM.txt file: mesh_body/shutters.im"

obviously due to animation limitations it has to be 104 animated parts. there doesn't seem to be a good way to have it both ways in this instance.

It should be noted that reliably making skinned animations is pretty difficult - the max exporter doesn't seem to be able to produce consistent results here. I tried in vain to get some chain animations working on the intermodal crane but could not get trainz to even play them (another dialogue was opened with trainzdev but went unanswered after some time). sometimes this animation method works, other times it produces no good results.

pcas1986
June 5th, 2015, 06:52 PM
Is this Blender? Torsten's exporter forces rigid skeletons on items with bumpmapped materials due to bugs with skinned animation and bumpmaps - so it would be the tbumpenv material. There was a discussion on this a while back, let me see if I can find it...

Curtis

Yes, but it is TrainzMeshImporter that puts out the chunk messages. I also vaguely recall a statement about tbumpenv and animation - it may be in the exporter code itself. Since the animation worked and looked OK then I didn't think it was an issue. Perhaps it is. OK, it is only a warning but I'd rather fix it now than later.




...Would one solution be, to get the chunk warnings down, to break up the main mesh of a locomotive into several smaller meshes and assemble these via attachment points? At the moment, I do have quite a few sub meshes included in my main mesh, just moved into place where these should be but not attached to any other mesh in 3DS Max but these are just differently named meshes. Most are texture mapped from only 2 texture maps and their corresponding normal maps used with possibly only 2 to 3 different material types for these textured meshes used. I do have separate texture maps for night mode and alpha numbering, which I have to use and I guess can not be included on any main texture map.

If you do the first (with attachments) then you will need a LOD system for each. In the second case surely the various meshes are still being combined in the same IM file and therefore subject to the chunk problem.

Chris made a comment a couple of weeks back suggesting, I think, that large assets, such as locos perhaps, could be constructed from mesh libraries. Unless I've completely misunderstood the comment, which is very possible, that seems like a rather large design problem. For example, if those separate parts used a common texture then managing the mapping within that texture would be very difficult.

WindWalkr
June 5th, 2015, 07:16 PM
Would one solution be, to get the chunk warnings down, to break up the main mesh of a locomotive into several smaller meshes and assemble these via attachment points?

It would suppress this particular warning, but instead of resolving the problem it makes it even worse. The correct solution is to use less chunks, not to distribute them across more meshes.



I am still wondering why all this tough crackdown on all these things like above is really necessary, you programmers MUST have serious performance optimization issues with TANE or you would not now do all this very strict error checking and restrictions etc.

Yes, having the content throw away a possible 10x performance gains for no reason is a big problem for us.

chris

WindWalkr
June 5th, 2015, 07:24 PM
It should be noted that reliably making skinned animations is pretty difficult - the max exporter doesn't seem to be able to produce consistent results here. I tried in vain to get some chain animations working on the intermodal crane but could not get trainz to even play them (another dialogue was opened with trainzdev but went unanswered after some time). sometimes this animation method works, other times it produces no good results.

Have you got any examples of this around? Was this with the current exporter versions?

The exporters are not my forte, but I can say that T:ANE converts everything to a skinned animation on load- there is no support for rigid animations except via conversion to skinned during the loading process.

chris

Dinorius_Redundicus
June 5th, 2015, 07:31 PM
Is there a way to check the chunk count within Max, gmax or Blender as we are making our models, rather than getting a nasty surprise when they are finally loaded into T:ANE?

WindWalkr
June 5th, 2015, 07:43 PM
Is there a way to check the chunk count within Max, gmax or Blender as we are making our models, rather than getting a nasty surprise when they are finally loaded into T:ANE?

No, however there are various assumptions that can be made:

* Each separate (sub)material will force any vertices using that material into a unique chunk.
* There is an upper size limit on chunks, probably in the range of 32k to 64k vertices. If you're getting up to that kind of size, then the exact number is probably irrelevant to you anyway- you can afford the extra chunk.
* Each rigid animation bone will introduce a unique chunk.


What this really means is:

* Go easy on the textures.
* If you're using rigid animation, go easy on the bone count. Ideally, avoid rigid animation. It's very much a legacy technique from the pre-shader days.

chris

pcas1986
June 5th, 2015, 07:51 PM
No, however there are various assumptions that can be made:
...
chris


That was a useful post. Thanks Chris.

Later, I will try and pick up the "bones" of this thread and put them into a WiKi page as I am reasonably sure this stuff is not mentioned anywhere.

These threads are proving very useful. :)

norfolksouthern37
June 5th, 2015, 08:00 PM
Have you got any examples of this around? Was this with the current exporter versions?

The exporters are not my forte, but I can say that T:ANE converts everything to a skinned animation on load- there is no support for rigid animations except via conversion to skinned during the loading process.

chris

I probably dont have any unless I can find the original file I sent to James and later to trainzdev. I used the latest exporters in max 12 at the time. Looking further up in the thread it may have been whatever issues come up from using a normal map material. I hardly use any other kind of material on my creations since 2009. As far as I can remember the max scene was configured correctly (using an example I got from Zec and another file that is a part from one of our locomotives that does function using this method) and it exported fine, but nothing would happen in the game when trying to 'play' the animation.

I have not tried any such animations in TANE as of yet but it will be something I revisit I am sure. I released the container crane as-is with some 30 rigid bones after giving up, but it works and as far as I know is the only such asset to function like this in Trainz. if TANE can prove to handle the animations there are many more ideas I have like this one :)

https://www.youtube.com/watch?v=xeR5Zn9k5Pw

WindWalkr
June 5th, 2015, 08:28 PM
I have not tried any such animations in TANE as of yet but it will be something I revisit I am sure.

Righto. When you get to this, let me know if you encounter problems.



I released the container crane as-is with some 30 rigid bones after giving up, but it works and as far as I know is the only such asset to function like this in Trainz. if TANE can prove to handle the animations there are many more ideas I have like this one :)

https://www.youtube.com/watch?v=xeR5Zn9k5Pw

Looks cool :)

There are upper limits to the number of bones actually influencing vertices. Something like this doesn't look like it would come close, but obviously it depends on how you go about modelling things. If you do want to exceed these limits, you can of course move to several meshes each with their own animations- but this isn't ever going to be cheap. Of course, for a one-off like a crane, that's not necessarily a problem.

chris

VinnyBarb
June 6th, 2015, 12:11 AM
It would suppress this particular warning, but instead of resolving the problem it makes it even worse. The correct solution is to use less chunks, not to distribute them across more meshes.
chris

How the hell will I do this "chunk" reduction in 3DS Max for a large locomotive with several meshes for it to export in one go? Please don't say you are not a 3DS MAX user, as this does not help us content creators. If anyone else might know how to do this, please feel free to enlighten me/us. I have a strong feeling, it is perhaps N3V/Auran's 3DS Max exporters which might be the cause of these many "chunks" creating and exporting problems, possibly by not being able to handle larger meshes in one go when exporting and thus breaking these meshes down into several or even many "chunks"? As the 3DS Max exporter I use (the latest version on the TrainzWiki available) tells me exactly how many "chunks" of a mesh it is exporting. What do I know, you might ask any of your own 3DS Max Exporter writers/makers what they did to create the Exporters? What about Blender users, do you get any of these too many "chunks" warning/error issues when exporting into TANE's CM?

I checked my 3DS Max Exporter in which one can PLAINLY see what your 3DS MAX exporter for Trainz does when exporting/creating an IM mesh. I use the latest/last version available for my 3DS Max:

http://www.vinnybarb.com/pics/chunk.jpg
I would not have the foggiest idea how to prevent or reduce or minimize any number of chunks in a mesh, 3DS Max does not tell me of any chunks, let alone of how many chunks there are in a mesh but your own 3DS Max Exporter, as seen above, does create these "chunks" when creating the .IM file to export. I have no idea how to eliminate/minimize these error/warning of "chunk" messages and I thought I knew a little bit of constructing for over 10 years or more with 3DS Max.

Other games DO NOT SEEM to have these 3DS Max "chunk" issues when creating meshes for these (when I modded some 3D Max meshes for Skyrim in the past), suddenly I feel like an outcast for daring to use 3DS Max for content creation in Trainz. Very depressing and frustrating to say the least and this issue possibly will make me give away all this "free content creating hobby" for good. As I am sure other 3DS Max content creators might feel very similar.

By now checking the 3DS Max exporter myself (as per the above picture), it seems you people shot yourselves in the foot, on one hand you advise us not to create extra but instead to reduce "chunks" in a mesh, on the other hand it seems to confirm in the picture above, your own exporters are creating this "silly chunk" situation.

What to you say?

VinnyBarb

WindWalkr
June 6th, 2015, 12:46 AM
I have a strong feeling, it is perhaps N3V/Auran's 3DS Max exporters which might be the cause of these many "chunks" creating and exporting problems

Yes and no.

Chunks are a concept used for realtime rendering, and not something native to Max. From the perspective of the Max UI, they are neither desirable nor necessary. Max would have a very similar concept within its own rendering pipeline, but it is hidden from the user and not relevant to this discussion.

It is absolutely the case that the exporter builds the chunks from the geometry and materials that you specify in Max. This is a necessary part of the export process, and is not problematic in itself.

Part of your job as a content creator is to ensure that the content within Max is built in such a way that this export process produces output which is efficient for realtime rendering. A number of suggestions have already been raised in this thread which detail the typical "trigger conditions" for the exporter to emit geometry to a separate chunk rather than as part of an existing chunk.



by not being able to handle larger meshes in one go when exporting and thus breaking these meshes down into several or even many "chunks"?

This is absolutely the case. If you export a one-million polygon mesh, you will end up with a lot of chunks. This is not avoidable, nor is this problematic except so far as a one-million polygon mesh is already problematic.

The key issue here is that in a lot of cases, content creators are taking actions which are leading to a lot of chunks being created where there is no such requirement. This leads to a loss of performance for no in-game benefits whatsoever.

There are two things that come from this:


1. We are now logging warnings when the chunk count rises to the point that there you are likely paying substantial performance penalties. As with all warnings, this is a suggestion that you look more closely at the way your content is formed, and it is not necessarily indicative of a problem. Having this warning available will both allow content creators to better optimise their meshes, and also allow us to better identify systemic problems which can be resolved through education or changes to our content pipeline.

2. In this thread, we are identifying the types of thing which are likely to introduce chunk count. If you are doing these things unnecessarily, then you are hurting performance to no benefit. In some cases, these will be trivial to avoid (eg. overuse of small textures) whereas in other cases the may take more learning (using skinned animation rather than rigid) or be unavoidable (simply having an excess of vertices, which is not automatically a bad thing depending on how they are being used.)





Other games DO NOT SEEM to have these 3DS Max "chunk" issues when creating meshes

"seem" is definitely the word here. We all use the same underlying GPU tech, and the same underlying limitations exist. Some games may simply hide the details from the user (as we have done in the past, in this area, and in fact we still only expose this when the count is high or if you're paying careful attention to the stitched mesh stats) whereas others may expose it but call it something slightly different ("draw calls" or "buffers" perhaps, which are each something slightly different but which amount to the same thing.)



kind regards,

chris

pcas1986
June 6th, 2015, 02:35 AM
... What about Blender users, do you get any of these too many "chunks" warning/error issues when exporting into TANE's CM?

...

VinnyBarb

Well yes, as described earlier with my bogie animation. But now I know where to look at I believe I can fix it.

But for my last two locos, including one still in progress, I found some interesting information in the TrainzMeshImporter log files.

The first is for my Big Bertha loco that has 45000 tris in the LOD0 mesh and that came out with 10 chunks which is about a third of the limit. (I think!)

The second, and my current project, has 64880 tris and has 8 chunks. I am curious why the second is generating less chunks.

More importantly, neither asset throws the chunks warning in T:ANE. :)

I believe it important to discover why your locos are throwing these warnings. Perhaps you could give us some statistics such as number of polys and the number of materials.





LOGFILE for TrainzMeshImporter (FINAL), at Tue Jan 14 13:35:09 2014

. Info: Process started. This may take some time. Please wait...
. Info: Reading input file: 'D:\Trainz-Development\Current\110001-5 BB-Loco-MR - 2290 - pending\bb_body_export_mr_lod0.xml'
. Info: Read Mesh
. Info: Read Materials
. Info: Finished reading input file
. Info: Filled Mesh Geometry
. Info: Filled Materials
. Info: Filled Attachments
. Info: Filled Skeleton
. Info: Filled Mesh
. Info: Converting formats and optimizing...
. Info: (internal progress) Building chunks
. Info: (internal progress) Optimizing chunks
. Info: (internal) Mesh limits exceeded, 7 chunks split into 10 chunks. (Not sure what this is about)
....
. Info: Wrote IM file 'D:\Trainz-Development\Current\110001-5 BB-Loco-MR - 2290 - pending\bb_body_export_mr_lod0.im' with:
numTriangles=45016
numVertices=41584
numChunks=10
numAttachments=30
skeleton = false (non-rigid will set 'false')
influenceArray = false


. Info: Process completed after 8.84 Seconds with 0 errors and 0 warnings






LOGFILE for TrainzMeshImporter (FINAL), at Thu May 14 14:50:45 2015

. Info: Process started. This may take some time. Please wait...
. Info: Reading input file: 'D:\Trainz-Development\Current\Me\HF-Me-CNJ 840 loco\110102 - 840 loco-body\body840-high-poly-lod.xml'
. Warning: Nonconformant name 'body840-30Apr15-all-LODs-v4' replaced with 'body840-30apr15-all-lods-v4'
. Info: Read Mesh
. Info: Read Materials
. Info: Finished reading input file
. Info: Filled Mesh Geometry
. Info: Filled Materials
. Info: Filled Attachments
. Info: Filled Skeleton
. Info: Filled Mesh
. Info: Converting formats and optimizing...
. Info: (internal progress) Building chunks
. Info: (internal progress) Optimizing chunks
. Info: (internal) Mesh limits exceeded, 3 chunks split into 8 chunks.
....
. Info: Wrote IM file 'D:\Trainz-Development\Current\Me\HF-Me-CNJ 840 loco\110102 - 840 loco-body\body840-high-poly-lod.im' with:
numTriangles=64880
numVertices=79849
numChunks=8
numAttachments=34
skeleton = false (non-rigid will set 'false')
influenceArray = false


. Info: Process completed after 10.4 Seconds with 0 errors and 1 warnings

WindWalkr
June 6th, 2015, 02:53 AM
The first is for my Big Bertha loco that has 45000 tris in the LOD0 mesh and that came out with 10 chunks which is about a third of the limit. (I think!)

The second, and my current project, has 64880 tris and has 8 chunks. I am curious why the second is generating less chunks.


This is probably something that we could help with by adding additional options to the preview window, displaying the hierarchy of meshes and chunks and allowing the user to toggle visibility of each (working out what geometry goes into each chunk would hopefully give some good suggestions as to why.) Might take a little while to put something together, but it should be worthwhile in understanding how your model is actually rendered by the GPU.

Aside from that, I'm pretty sure that there are some third-party tools around that can look inside the IM files and tell you what material parameters, chunks, etc. are present.

chris

paulhobbs
June 6th, 2015, 03:03 AM
Is it the case that having separate objects in the max file generates extra chunks even if they are using the same material? (assuming they are all being exported into the same im file)

Paul

andi06
June 6th, 2015, 03:04 AM
Aside from that, I'm pretty sure that there are some third-party tools around that can look inside the IM files and tell you what material parameters, chunks, etc. are present.
PEV's mesh viewer and the AssetX mesh preview will show you which bits of a mesh are textured with which material and provide a text based summary of chunk usage - its not bedtime reading.

pcas1986
June 6th, 2015, 03:35 AM
This is probably something that we could help with by adding additional options to the preview window, displaying the hierarchy of meshes and chunks and allowing the user to toggle visibility of each (working out what geometry goes into each chunk would hopefully give some good suggestions as to why.) Might take a little while to put something together, but it should be worthwhile in understanding how your model is actually rendered by the GPU.

...

chris

That sounds like a plus but I will have another look at PEV's preview in AssetX. At the risk of sounding like a broken record, perhaps there are some opportunities to provide Andi and Peter (PEV) with sufficient information to improve AssetX's utility for content creators. i.e. use the community to provide the tools that Trainz doesn't.

I sometimes think that creators don't realise just what AssetX can do for them.

WindWalkr
June 6th, 2015, 04:30 AM
Is it the case that having separate objects in the max file generates extra chunks even if they are using the same material? (assuming they are all being exported into the same im file)

I honestly have no idea at all here. It should be pretty straightforward to test this, however.

chris

WindWalkr
June 6th, 2015, 04:32 AM
That sounds like a plus but I will have another look at PEV's preview in AssetX. At the risk of sounding like a broken record, perhaps there are some opportunities to provide Andi and Peter (PEV) with sufficient information to improve AssetX's utility for content creators. i.e. use the community to provide the tools that Trainz doesn't.

I think that they already have all the info that they need in that direction, and I'm more than happy to provide additional info if they need something. Andi is in here already; I don't think PEV is but he's also more than welcome to join the party..

chris

andi06
June 6th, 2015, 05:29 AM
I think that they already have all the info that they need in that direction
True, except for encrypted textures and meshes which have appeared lately. I know that PEV has struggled with 'influence' chunks - I'm not sure where he is with that currently.

Where this is done to provide DRM I don't have any argument, especially now that the native Preview is available to look at the mesh files.

Encrypted *.texture files for non-payware assets is another matter and I'm not sure why this is being done.

Encryption 'by accident' caused a lot of issues in the beta team getting to the bottom of procedural-track and it makes reskins virtually impossible - a lot of potential content creators start off by reskinning assets.

WindWalkr
June 6th, 2015, 06:05 AM
True, except for encrypted textures and meshes which have appeared lately.

For obvious reasons, we can't help you with that one.



I know that PEV has struggled with 'influence' chunks - I'm not sure where he is with that currently.

I've spent a lot of time with the animation code over the last year so I should be able to help out there. Let him know to PM me if he needs a hand (or he should still have my email, hopefully.)



Encrypted *.texture files for non-payware assets is another matter and I'm not sure why this is being done.

It isn't. Only content that is flagged as payware is encrypted; you can check this in the config file. Perhaps you're confused by the fact that TANE and TS12 use a different file format for the internal .texture files. There's no great secret here, and I can help provide parsing details if it's required.

chris

andi06
June 6th, 2015, 06:27 AM
It isn't. Only content that is flagged as payware is encrypted; you can check this in the config file. Perhaps you're confused by the fact that TANE and TS12 use a different file format for the internal .texture files. There's no great secret here, and I can help provide parsing details if it's required.

We'll probably come back to you on that. If encryption is limited to assets with 'is-payware-content 1' then at least I can show a message rather than a blank viewer.

WindWalkr
June 6th, 2015, 06:37 AM
Yes, it's limited to that case.

chris

VinnyBarb
June 6th, 2015, 06:07 PM
What I have found with Google to get information about chunks in 3D meshes and files and what I have found and learned is posted below. I guess some or most of you content creators know this already, my post is intended for those content creators that do not know this yet. In essence it is this with chunks:

I am talking here specifically of 3DS MAX meshes and their files structure and setup and I think this is somewhat similar with other 3D file formats like Blender too. Any 3D file contains "chunks" which are the places to hold information of how a 3D mesh is structured and build after this is exported out of 3DS Max. There are at least 3 chunks for vertices, where one chunk contains the info of vertices of the x-axis in the mesh, the other chunk has got info of the y-axis and another chunk the info of the z-axis info of these vertices. Consider chunks as being the placeholders for the various information bits needed for a 3D mesh to be a correctly assembled and good looking mesh.

Other different and possible quite a few of these chunks contain the texture mapping info, where a part of a texture is on a part of the mesh, what colour, amount of specular, transparency and so on, other chunks again hold the different material types infos of the various material types being used in the mesh, one or more chunks per material type and so on. Animated .KIN files have animation info like key frames and other info contained within. As one can see, any 3D mesh must have a minimum amount of chunks contained in its mesh structure for the mesh to be used correctly somewhere else. Like in our case, exporting an .IM or an animated .KIN file out of 3DS Max to be imported into Trainz

Now you will ask and this was IMHO not quite and proper explained so far HOW to reduce the amount of chunk instances in a given Trainz 3D mesh (.IM and .KIN mesh), hence I had to look for all this "chunk" business on the Internet to see and learn from. One certain way to reduce chunks in a given .IM file is to reduce the meshes it contains in it, ideally needing one mesh only, where one has a certain amount of needed chunks in it. One can not go below the minimum amount of chunks needed for this mesh or the mesh will either be corrupted or might not show at all in Trainz.

I have been guilty too in the past, even with my TS 12 created stuff, to export together most different meshes needed for an asset contained within my 3DS Max file, not knowing\ the above about chunks and no one before pointing out why one needs to reduce the meshes in an exported .IM file to cut down on the chunk count in an .IM file. Looking at some of my past exported 3D Max locomotive meshes, I have parts of the loco build as individual meshes in 3D Max, say a fuel tank, a platform, the engine bay, the inspection panels, the lights and hoses and so on and so on but placed there where these need to be to make said locomotive. Perhaps 20 to 30 or so different meshes, all contained in the exported .IM file, each different mesh with each extra similar count/amount of chunks but this time 20 to 30 times as many chunks contained overall.

Now logic dictates, it is prudent or even mandatory to attach as many meshes together when exporting, so ideally now one only has ONE SINGLE mesh to export, which is not always possible. So the minimum least amount of meshes contained in the export file, the better for performance this will be. On to some more chunk culling. Check how many material types you really need for your exported mesh, again ideally one material type used would be the best but is really not ideal for Trainz. One might need a m.tbumptex plus a m.tbumpenv material and possibly another different material type to suit the asset.

Then on to the texture map(s) being used in an .IM file, the more texture maps there are used in an asset, the more chunks are getting created for each texture map, as by using more texture maps, extra info, meaning extra chunks are needed, hence more chunks are thus getting created. By placing all different textures needed for texture mapping onto a single or even two large texture maps of up to 2048 x 2048 pixels in size, this is often more than enough for texture mapping most assets for Trainz. Sometimes one needs an extra texture map, say, for transparent decal placings or such. If such is needed, it is indeed needed, nothing a content creator can do about that.

Plus some other creating methods will create extra chunks introduced into Trainz, hence we content creators need more information of how many chunks are in a given asset, preferably somewhere contained in the DevTools in TANE and how this might affect performance. I can see above where Chris posted of doing something about that. Good. Something like:" buddy, you already used more than enough chunks in your asset, hence your mesh will self destruct in 5 seconds" or such. <-joke

Hence error/warnings regarding chunk numbers should only show in CM, IF this is above a reasonable arbitrary top limit of chunk numbers in a given mesh or if it is known by us content creators, how many chunks in a given mesh are reasonable for performance in TANE for a given asset. As one can plainly see by the above, some chunk numbers are necessary for any given mesh. Not like it is now, just by giving an error/warning regardless of the amount of chunks included in said given mesh.

Finally, what I have learned that the number of vertices and polygons/triangles per mesh is limited to 65536 per any mesh. Anything above this number introduces again extra chunks, So beware if using heavy handed smoothing or heaven forbid, even using TURBOsmoothing for your mesh, this will increase the amount of chunks exponentially upwards, especial with turbosmoothing as this will blow the chunk count limit literally through the roof.

There can be more added to this thread, if anyone can add to the above, the more info we content creators know and have, the better this is. So please, if you know more, please post this here.

As said, as many and most of you already know the above posted, it is only posted here for those who still do not know this and who do not want to chase all over the Internet with Google to probably find or even not find what one needs to know of chunks in 3DS Max meshes. I hope this will give some of you a little inside of the inner workings of a 3DS Max mesh file setup and its structure and hence will lead to better content creating.

Have fun reading.

VinnyBarb

WindWalkr
June 6th, 2015, 08:14 PM
There are at least 3 chunks for vertices, where one chunk contains the info of vertices of the x-axis in the mesh, the other chunk has got info of the y-axis and another chunk the info of the z-axis info of these vertices. Consider chunks as being the placeholders for the various information bits needed for a 3D mesh to be a correctly assembled and good looking mesh.

I think you might be talking about IFF chunks or something similar here; this isn't related in any way to how we're use the term. All data for a given vertex will end up in a single chunk, and nothing except geometry data and its related attributes is stored in chunks.



Now logic dictates, it is prudent or even mandatory to attach as many meshes together when exporting, so ideally now one only has ONE SINGLE mesh to export, which is not always possible. So the minimum least amount of meshes contained in the export file, the better for performance this will be.

I honestly have no idea whether this is true or not. If you're just making assumptions, please test the assumptions before you go changing anything. If you're actually tested it already, then good on you- and please say so, to avoid others needing to do the same tests.



On to some more chunk culling. Check how many material types you really need for your exported mesh, again ideally one material type used would be the best but is really not ideal for Trainz. One might need a m.tbumptex plus a m.tbumpenv material and possibly another different material type to suit the asset.

Then on to the texture map(s) being used in an .IM file, the more texture maps there are used in an asset, the more chunks are getting created for each texture map, as by using more texture maps, extra info, meaning extra chunks are needed, hence more chunks are thus getting created. By placing all different textures needed for texture mapping onto a single or even two large texture maps of up to 2048 x 2048 pixels in size, this is often more than enough for texture mapping most assets for Trainz. Sometimes one needs an extra texture map, say, for transparent decal placings or such. If such is needed, it is indeed needed, nothing a content creator can do about that.

All good advice in general.



Plus some other creating methods will create extra chunks introduced into Trainz, hence we content creators need more information of how many chunks are in a given asset, preferably somewhere contained in the DevTools in TANE and how this might affect performance.

As above, there are definitely ways we can improve this, but as long as you're talking about stitched geometry you can already see this by watching the buffer stats in the surveyor/driver performance HUDs. Definitely not useful for all cases, but it allows you to test your theories on which Max changes cause more chunks to be generated.



Hence error/warnings regarding chunk numbers should only show in CM, IF this is above a reasonable arbitrary top limit of chunk numbers in a given mesh or if it is known by us content creators, how many chunks in a given mesh are reasonable for performance in TANE for a given asset. As one can plainly see by the above, some chunk numbers are necessary for any given mesh. Not like it is now, just by giving an error/warning regardless of the amount of chunks included in said given mesh.

No, that's exactly like it is now. We only warn when the chunk count gets pretty crazy.



Finally, what I have learned that the number of vertices and polygons/triangles per mesh is limited to 65536 per any mesh. Anything above this number introduces again extra chunks, So beware if using heavy handed smoothing or heaven forbid, even using TURBOsmoothing for your mesh, this will increase the amount of chunks exponentially upwards, especial with turbosmoothing as this will blow the chunk count limit literally through the roof.

You should always aim to reduce your vertex counts, since (assuming that the mesh is reasonably optimal in other ways) this is fairly directly related to rendering cost. As above, if your vertex counts are high enough that you're getting a lot of chunks, then it is not the chunks that you should be concerned about.


There's also a discussion of this stuff on the wiki that is probably worth your time reading. We might even want to update it with some of the outcomes of your discussion (ie. avoid too much rigid-body animation, etc.) Modeling Guidelines (http://online.ts2009.com/mediaWiki/index.php/Modeling_Guidelines)

cheers,

chris

VinnyBarb
June 6th, 2015, 10:31 PM
There you go again, you just generalize there without actually telling us where the problem really lies. The way you described "chunks", YOU only know how this affects TANE or Trainz in general. You NEVER specified what "type" of chunk you actually mean which is affecting performance, as you so preach above. I made an effort as a "layman" to find this out myself about this chunk business and I just posted my observations above with some TO ME logical guidelines of how to create content now for TANE.

How do I know and as an educated guess, some or most content creators who are not programmers per se don't know either and only do content creating as a hobby, how this "chunk" business affects as all WITHOUT giving specific ways of the how and where this intrudes into your sphere of programming and how to solve this successful from our end.

I can only go by what seemingly looks logically to me, not knowing until now, there are good chunks and there are bad chunks and what they all do to the game. To me, a chunk is a chunk is a chunk and that is how I will look at these UNLESS someone comes along and really explains this clearly from the view points of both sides of the fence we are dealing here.

As said before, talking of dealing with chunk reduction and then saying, this is/was all said before (the how to do this) will not tell me how REALLY to deal with this subject. Like you talk about vertex reduction being one way of dealing with this, tell me how many meshes really do exceed the vertex limit being in a mesh of 65536 vertices to create any extra BAD chunks if exceeding that limit and therefore create (like you so often point out) headaches for TANE? Again, how many normal created meshes will in reality exceed this 65536 vertices limit and therefore create extra chunks if exceeding this limit? Is it really, really causing all that mayhem as you suggest or say or is this just a generalizing and glossing over of what might be really the problem, which might even be unrelated to all this above, as it seems YOU DO indeed have some serious performance solving issues at the moment and my guess is you grasp at any piece of straw you get hold of and could lay the blame on? We all would understand if this is indeed the case, so why not tell us and can solve this together?

If one were cynical and could say, hey, now we have a lot of faster PCs with 4 core and 8 core CPUs, newer video GPUs outperforming what is actually needed to run well and display good any graphic intensive current game, therefore one might ask, what IS the problem? What is actually needed for a good and graphic intensive TANE to run on such PCs. Now we even have TANE running in Win 64 bit, with almost all of the installed memory getting used for this, not like before with WinXP, having only up to 2 GB of memory getting assessed including running the OS and any other background programs to run in the same memory space, with Win7 32bit, AFAIK, which can assess 4 GB of memory with the same OS and their background programs taking away some of its memory space. Is it because these older Trainz versions did not know their limitations, hence they did indeed perform as well as they did?

Makes me wonder why all of the previous Trainz versions, as said above, were even running on one's crappy PC in the past with no one seemingly really caring about creating efficient and proper content and just willy nilly uploading what one wanted to upload to the DLS. Trainz still operated somewhat reasonable in those days. Lucky for us ignorant content creators creating content in the past, not knowing of these chunk and vertices limitation issues and happily creating away ignorant then and as said, Trainz still kept on running on one's crappy PC in the past.

Fast forward to today, how things have changed regarding content creating. I am all for good and correct created content to be available but it seem, we somehow all get penalized because N3V Games still insists to using that very old content as build in content, the very same crappy performance robbing content, which is suppressed in TANE's code to run error free but is still robbing performance left right and centre which apparently now seems to bring TANE to its knees. As we all of a sudden have to think of lesser chunks and vertices limitations and god only knows what else will be soon dished out as culprits, without much explaining, for not how to create content now and not knowing how this all will affect TANE.

Who's fault is this? You people should have cut with the past and with its old crappy build in content for TANE, as we have now a different game engine, with different and I presume a good platform and criteria to operate new content in. You should have given us content creators at the very least some guidelines very early on of how to create now ONLY for TANE and leave all the old inefficient content to run only on the earlier Trainz versions. Which were ignorant of all this "faulty" and "chunky" content in the past and despite all these faults and now being told to be incorrect creating but they still kept on running and performing as good as they did then.

I know, you will say, if this were so done, we would have no content and no routes with such content presently in TANE, which isn't even finished yet, as this isn't yet a proper TANE release. Again, who's fault is this, you could have had, for example, your own Rob creating new content for a year already or you could have even hired an extra content creator for this. Wasn't this what Kickstarter was all about to help things along? Plus we content creators could have had at least a year's time already to create content for TANE, all it needed was to have for us available a simple testers copy of TANE to test content in, the way this will look lighting wise, environmental mapping and normal mapping wise. Plus some extra info of how to be able to create this differently for TANE. Hence we all wasted a good solid year of getting and creating any content for TANE, just because we did not know of how to do this, Well, we still don't know to some extent, people (other content creators) I converse with all say similar things and agree with the above.

We content creators were time and time again told, TS12 content will be OK for TANE but as we all found out now, apparently this is not the case with a lot of such TS12 content. Hence we still wait and wait and wait what ultimately is needed for correct and proper content creating for TANE. When will this be forthcoming, as we asked time and time again for this, only being told, DO NOT CREATE THIS WAY but not telling us what the correct way is, is not really helpful?

The other issue I am having with you saying, the 3DS Max Exporters created by AURAN/N3V, when chopping a mesh into many different chunks, when exporting a mesh out of 3DS Max, which they really do tell you, I can assure you of this, up to often of 30 or so chunks getting created, is a "good" thing and not bad one. Can you please explain why having different pieces of a mesh in game after exporting and installing this into Trainz, which I would still call "chunks", as the Exporter tells me it is so, is "not bad" and will not affect the performance of TANE or Trainz? Even if this getting reassembled somehow later in game, which I don't know if this is really happening, it would logically still need some CPU operation to do this and hence be a drain on resources.

My humble observations of the state of content creating for TANE.

VinnyBarb

pcas1986
June 6th, 2015, 10:37 PM
Chris,
The discussions in this thread have caused me to take a closer look at the information available to me in various logs and within AssetX. Part of the TrainzMeshImporter log includes this information for one of my bogies:



numTriangles=5248
numVertices=6907
numChunks=1
numAttachments=42
skeleton = false (non-rigid will set 'false')
influenceArray = true

numTriangles, numVertices and numChunks are obvious.

I've worked out that numAttachments is the total of all meshes and attachment points (i.e. "a.").

Presumably "skeleton = false" is preferred?

I have no idea what influenceArray means and what its preferred value might be.

WindWalkr
June 6th, 2015, 11:28 PM
*sigh* forums just deleted a large reply I wrote to this one. Can't summon the effort required to rewrite it just now, so quick version follows. Apologies for being brief.



There you go again, you just generalize there without actually telling us where the problem really lies.

I've discussed this issue in quite some detail in this thread. If you have questions on the specifics of any of the points I've raised, please feel free to ask.



Makes me wonder why all of the previous Trainz versions, as said above, were even running on one's crappy PC in the past with no one seemingly really caring about creating efficient and proper content and just willy nilly uploading what one wanted to upload to the DLS. Trainz still operated somewhat reasonable in those days.

As Trainz goes on, we ask more and more of the game in terms of capabilities. If you load up an older route into T:ANE, you will find that it runs blazingly fast. Modern routes are MUCH larger and MUCH denser, and each item of content is typically MUCH higher detail.

Yes, this problem can be solved by throwing faster hardware at the problem, but if we can improve things on the content creation end then that benefits everyone.



We content creators were time and time again told, TS12 content will be OK for TANE but as we all found out now, apparently this is not the case with a lot of such TS12 content. Hence we still wait and wait and wait what ultimately is needed for correct and proper content creating for TANE. When will this be forthcoming, as we asked time and time again for this, only being told, DO NOT CREATE THIS WAY but not telling us what the correct way is, is not really helpful?

I'll stick by our original claim; that correctly-formed TS12 content will run equally well in T:ANE.

Yes, that's a generalisation. Yes, there are a few cases where we're deliberately breaking this generalisation. Yes, there are a few cases where there are bugs which break this generalisation (just take a look at some of the threads on this forum where we're working to try and isolate the problems and get them resolved.) No, you don't need to rethink your existing practices, except where the existing practices were also problematic in TS12.



The other issue I am having with you saying, the 3DS Max Exporters created by AURAN/N3V, when chopping a mesh into many different chunks, when exporting a mesh out of 3DS Max, which they really do tell you, I can assure you of this, up to often of 30 or so chunks getting created, is a "good" thing and not bad one. Can you please explain why having different pieces of a mesh in game after exporting and installing this into Trainz, which I would still call "chunks", as the Exporter tells me it is so, is "not bad" and will not affect the performance of TANE or Trainz? Even if this getting reassembled somehow later in game, which I don't know if this is really happening, it would logically still need some CPU operation to do this and hence be a drain on resources.

More of anything has a cost. More polygons has a cost. More textures has a cost. More chunk has a cost (although for the most part you can ignore this, since it's typically a symptom and not a cause.)

The question is not "how to reduce X?" but rather "is X justified?"

If you have a 30k polygon LOD0 on a loco, I think most people would see that as justified. So it is with chunks. The key is not to get excessive with polygons, materials, meshes, bones, etc. If you are introducing new objects into the scene for no user-facing benefit, then you should consider carefully what you are doing. Sometimes you may have a good answer as to why it is necessary, and why the cost is justified, but if you don't, and especially if the percentage efficiency loss is high, then you are probably doing a poor job of optimising your asset.

chris

Pencil42
June 6th, 2015, 11:34 PM
Note also that some of this increased power is getting eaten up by higher resolutions and other video processing techniques (Anti-Aliasing, for example). Back in the '04 / '06 days, I was running 1024x768 resolution max with no AA; but am now running 1920x1280 and 8x AA; an increase of ~2.5x pixels + the AA overhead.

Curtis

WindWalkr
June 6th, 2015, 11:36 PM
..TrainzMeshImporter log includes this information for one of my bogies

Okay, keeping in mind here that this is simply a log of what it is doing and is not intended as a commentary on whether the content is "good" or "bad"..



Presumably "skeleton = false" is preferred? I have no idea what influenceArray means and what its preferred value might be.

Both of these come down to the animation techniques in use. I'd have to look into the code to determine exactly what it's referring to, but assuming that these are general terms, which may or may not be a safe assumption:

* A skeleton is a collection of bones (obviously) and its presence simply indicates the use of animation.

* An influence array is a per-vertex list of bones and the extent to which they influence that vertex. Skinned animation uses this technique, so if you're involving a complex animation, you probably want this to be present. Rigid body animation uses a single bone per chunk, which works well for small numbers of bones or if the chunks are already full, but (as discussed earlier in this topic) probably causes the chunk count to blow out rapidly if you are using a lot of bones and each bone has only a small number of associated vertices. Doubly so if the vertices are also used for multiple materials.

chris

WindWalkr
June 6th, 2015, 11:43 PM
Note also that some of this increased power is getting eaten up by higher resolutions and other video processing techniques (Anti-Aliasing, for example). Back in the '04 / '06 days, I was running 1024x768 resolution max with no AA; but am now running 1920x1280 and 8x AA; an increase of ~2.5x pixels + the AA overhead.

Yeah. This is actually an amusing one when people complain about how "software was so much faster in the old days".

Let's take a 40-column text display. Say, 40x24x1B. That's roughly 1KB of memory that needs to be processed in order to refresh the screen.

Now compare to a modern 5K display, say, 5120x2880x4B. That's roughly 59MB of memory that needs to be processed in order to refresh the screen.

We're asking the computer to do fifty-nine-thousand times as much work, and then wondering why, even with a modern CPU, it doesn't seem any faster.

chris