MicroDem / HOG giving grid lines even though TGA has none..

[treblesum81]

Hi All,

Just like the title says, when I'm importing MicroDem maps into Trainz 2009 using HOG, I get faint grid lines (long, thin, straight ridges and troughs) showing up on my layout. Thinking that I didn't turn off the grids in MicroDem, I made another DEM import making sure there were no grids and still they show up. So I opened up the TGA in photoshop and zoomed in on some of the places where the lines supposedly are, only to find that there are absolutely no lines other than the color gradients in those locations. Is this a shortcoming of MicroDem / HOG, or am I doing something wrong??

Thanks,
Greg

EDIt: If this is a deficiency of the programs, is this something I can get away from by moving to TransDem?

 

[Jerker]

G'day treblesum81,

I would not expect any of the software you are using to create the effect you are getting. I suspect that the issue is being caused by some artefact in the actual DEM data. You have not mentioned the source for this...

...if this IS the case, then TransDEM would not overcome the problem (it would simply reproduce it)...

Also, a careful look at the image you have provided there causes me to have some concern over the contour ridges (or what we refer to in the trade as 'terraces'), created by the rise in the terrain from one contour to the next. This can be minimised by using a greater value in your 'smoothing' filter (but that is another issue) and not the one over which you are concerned.

What area are you (DEM wise) working with there, so I can check the DEM tile with my own source?

Jerker {}

 

[treblesum81]

G'day treblesum81,

I would not expect any of the software you are using to create the effect you are getting. I suspect that the issue is being caused by some artefact in the actual DEM data. You have not mentioned the source for this...

...if this IS the case, then TransDEM would not overcome the problem (it would simply reproduce it)...

Also, a careful look at the image you have provided there causes me to have some concern over the contour ridges (or what we refer to in the trade as 'terraces'), created by the rise in the terrain from one contour to the next. This can be minimised by using a greater value in your 'smoothing' filter (but that is another issue) and not the one over which you are concerned.

What area are you (DEM wise) working with there, so I can check the DEM tile with my own source?

Jerker {}

The area is actually around Kingman, Arizona, which is a relatively flat desert surrounded by hills and mountains. The DEM data is coming from the USGIS free database and is uniformly 10m resolution. I'm not sure whether there are artifacts there or not, but I've not been able to locate any of the ridges in the color gradations of the targa file anyway, which makes me wonder why they are showing up... HOG needs differences in color to create such features, yet they are showing up without said gradations...

As for the "terracing", I would like to know more about this smoothing filter of which you speak. I've still got all of the DEMs downloaded, so I can remake the map within a few minutes.

 

[Jerker]

G'day treblesum81,

Once again, I see issues created from the use of DEMs other than those provided by N. A. S. A.. Would you indulge me by re-doing the route with the appropriate DEM tiles supplied by those from the N. A. S. A. S. R. T. M. (which, for the continental U. S. are of 1 arcsec, 10 meter resolution)?

It has been a long time since I have used MicroDEM (having switched to Global Mapper many years ago and more recently TransDEM) and although I do still have it installed on my sytem (version 8, mind you), I was unable to find the 'setting' to which I refer (primarily because it wouldn't load any of the DEM files I have - for reasons unkown) but I am absolutely certain it is an option. Try looking for it in the instructions...

Jerker {}

 

[treblesum81]

G'day treblesum81,

Once again, I see issues created from the use of DEMs other than those provided by N. A. S. A.. Would you indulge me by re-doing the route with the appropriate DEM tiles supplied by those from the N. A. S. A. S. R. T. M. (which, for the continental U. S. are of 1 arcsec, 10 meter resolution)?

It has been a long time since I have used MicroDEM (having switched to Global Mapper many years ago and more recently TransDEM) and although I do still have it installed on my sytem (version 8, mind you), I was unable to find the 'setting' to which I refer (primarily because it wouldn't load any of the DEM files I have - for reasons unkown) but I am absolutely certain it is an option. Try looking for it in the instructions...

Jerker {}

If you can point me to where I can get free STRM data (I always thought it was pay-only), then I would be more than happy to use it (I personally prefer it much more than this USGIS stuff, not only does it cover the whole world, but it also comes in larger chunks... this 7-10 mile tiles are getting tiresome...). Also, please tell me about Global Mapper... is it free? is it better than MicroDem? I've only been using what I have because its what I've been directed to, not because I'm particular to one or another, though, at this point at least, I don't think I'll be putting any money on the line just yet, so if it isn't free I won't use it...

As to the smoothing filter, I went straight to the instructions as soon as I read what you had said and was unable to find any reference to such a setting.

 

[Jerker]

G'day treblesum81,

Get your appropriate S. R. T. M. data from this link..

ftp://e0srp01u.ecs.nasa.gov/srtm/version2/SRTM3/North_America/

...where you must obtain the tile with the name corresponding to the south west corner of the area you require by adding the value 1 to the required longitude. For example if you need a tile to cover N40 W78 (south-west corner) to N41 W79 (north-east corner), then you must download the tile labelled N41W78. The S. R. T. M. tiles are *.zip tyoe files that extract to *.hgt type files but MicroDEM will 'open' them from the *zip format directly without the need to extract them (unless you feel so inclined)...

Global Mapper is very definately a superior product compared to MicroDEM but unfortunately it is not freeware (the link wil give you all the information you require)...

...Hmmm, you seem to be correct regarding the presence of anything to do with the 'smoothing' I mention. Perhaps I am confusing Global Mapper with MicroDEM (but then I was absolutely certain)...

There is one other option I haven't mentioned yet and this was something that was discussed a long time ago in this forum (actually in one of it's previous incarnations) and that is the use of the XYZ Grid export for the DEM into HOG, instead of the standard Chrome Depth Colours. You may want to try this (I was advised that the height is exported with much greter accuracy). As far as I am concerned, this works superbly in Global Mapper and should do the same in MicroDEM (at a guess, you would need to change the 'view' from "Chroma Depth Colours" to "XYZ Grid" - if that is an option)...

Jerker {}

 

[treblesum81]

For $300 it had better tie my shoes for me... I'm glad you turned me onto the SRTM (it took me a day to download only 75% of a tile that took me 30 seconds to do here), but I think I'm still going to have to mess with MicroDem. On that topic, XYZ Grid doesn't appear to be an option... are there any other tricks you can think of?

 

[treblesum81]

Ok, so I've decided I may be in trouble here... I've spent the evening trying every permutation I could with MicroDem to get it to show smooth terrain, all without any luck. I've downloaded the trial of Global mapper, but its super complicated (not that I can't eventually figure it out, but its a pain), and from the looks of things not going to output much better data than Microdem (still limited by the 255 color gradations that produce the 1m terraces). I've tried the ASCII XYZ method, but I can't get the height map to convert properly (rather than a "psychedelic" image, I get hundreds of diagonal lines of dots and dashes, so no actual map can be made), and even if I could get it to work, the route that all of this is for is going to cover 35* of longitude and 10* of latitude (don't say it... I know its big), so the distortions brought into place by resizing the TIGER maps for each tile are going to add up to a mess... I know I can probably use I-Portals to cut down on the trouble, but as it stands, I can't even get one smooth DEM layout.

Help and advice would be greatly appreciated here about how to get the ASCII XYZ method to generate a proper Targa image, or also how to make GeoTIFFs work properly as they supposedly have 1mm resolution, and hence, absolutely no terracing... I might be interested in learning how to get Global Mapper to work properly, but at $300 I would only use it until the trial lapsed and then I'd be done...

 

[Jerker]

G'day treblesum81,

I knew you would benefit from the use of the S. R. T. M. data, it's unfortunate that you can't tell me, yet, if it solved your issue...

To get the XYZ Grid output from Global Mapper to work with HOG, you have to set up HOG up to accept the input but before you do that, you need to convert the XYZ Grid data into the otherwise 'missing' *.tga file. There is a utility in HOG called "Piglet" hehe which will do this. Piglet is accessed through the Tools menu; Tools>Convert>ASCII XYZ to 24-bit height map..., which opens a dialogue box for you to browse to the location of the XYZ file and when that is selected, the data gets 'converted' (and placed automatically in the same location) - you may want to have something else constructive to do in the meantime, it may take some time to do the conversion, especially with a large map (and be prepared for a constant "out of memory" error the more you try to do in any one sitting). Once you have the *.tga file, you must make sure that the "vector".tga file, the "minimap".tga file and the "DEM".tga file (the converted XYZ file) are ALL the same size, otherwise, this will produce an error in HOG. The rest of the process is as you do it now, although when you 'pick' the DEM.tga file in the Elevation Data section of HOG, make sure you change the setting for the 'Data Format' input to "24-bit height coding", leaving the remaining settings in that section as is - though you can 'check' the 'interpolate' option box if feel so inclined).

Much easier than all of this (and certainly much cheaper than Global Mapper) is TransDEM, which will basically do all of this 'invisibly' - find your various bits of data (the same as you do now) and have a map the size of the one you are discussing finished within a few days depending on how much you wish to embellish the original map images with user supplied 'vector' data...

Jerker {}

 

[treblesum81]

So now I'm running into trouble again . Working with the trial, I exported the XYZ file and imported it into HOG. The resulting map was a "forest" of points, which didn't clear up even after I redid the ground file with "interpolation" selected. This is what it looked like:

Going back into Global Mapper, I realized that the XYZ file was being outputted with a resolution of about 100m per point, which I think is just too much for HOG / Trainz to smooth. So I increased it to the 10m resolution that the SRTM is supposed to come in, and now the XYZ file is nearly 4GB and I can't get HOG to work with it... Where am I going wrong now?

 

[Dermmy]

I am a bit late into this, but referring to your original screenshot:

1. The narrow 'ridge' will be along a baseboard join. There is an issue with the way HOG handles two of the four baseboard 'edges'. Two edges are referenced twice, once for each board, and terrain irregularities are fairly common along them. It is a nuisance, but a quick hit with the Surveyor 'plateau' tool fixes it. AFAIK there is no fix. There will also be issues with the appearance of ground textures on these edges in TRS06 (and I believe later versions).

2. The 'terracing' seems to be a 'feature' of 10m quad data in 'flattish' areas. Oddly enough you don't get it with 30m data. My Clovis Sub (New Mexico desert) suffers from the same and I couldn't fix it with anything I tried in MicroDem. The good news is that after texturing it is hardly noticeable.

I'd be living with the admittedly less-than-perfect (but usable) first effort...

Andy

 

[treblesum81]

Switching from SRTM3 (the original link) to SRTM1 V2 which gives better resolution, produced the following results:

 

[Dinorius_Redundicus]

Greg

Just regarding the terracing - I've also had that problem with the chroma method. I used ASCII-xyz to overcome it on a very big map (1300 boards). I know it hasn't worked out for you yet, but not sure why. In this thread;
http://forums.auran.com/trainz/showthread.php?t=3832

are some notes I made as to exactly how I created and processed the xyz. They may just be old news, but I refer you to them anyway on the chance that they may assist.

~ Deane

 

[Jerker]

G'day treblesum81,

Ooops! Indeed, I completely forgot about the resolution issue (the beauty of Global Mapper is that once a setting like that is changed, it remains so - and thus forgotten). Sorry 'bout that!

It would appear that if you read through the thread to which Deane has pointed, you, you WILL find a great deal of useful information...

Thanks to Dermmy for clearing up the issue with the troughs...

...but I have no idea of what is happening in the images you show - I never obtained results like that at any time...

...as far as the huge file problem is concerned, I found that dividing the route into smaller portions with which HOG could work made things less harrowing but care needs to be taken to ensure that proper alignment is then obtained when the route portions are merged in Trainz...

...once again, this is something that TransDEM can do with little or no effort on the user's part...

Jerker {}

 

[treblesum81]

G'day treblesum81,

Ooops! Indeed, I completely forgot about the resolution issue (the beauty of Global Mapper is that once a setting like that is changed, it remains so - and thus forgotten). Sorry 'bout that!

It would appear that if you read through the thread to which Deane has pointed, you, you WILL find a great deal of useful information...

Thanks to Dermmy for clearing up the issue with the troughs...

...but I have no idea of what is happening in the images you show - I never obtained results like that at any time...

...as far as the huge file problem is concerned, I found that dividing the route into smaller portions with which HOG could work made things less harrowing but care needs to be taken to ensure that proper alignment is then obtained when the route portions are merged in Trainz...

...once again, this is something that TransDEM can do with little or no effort on the user's part...

Jerker {}

The more and more this goes on, the more I think that TransDEM is the way to go... none of this crazy hassle with who imports what into where with what resolution and settings to get the failed attempt....

I am curious whats causing the "forest" effect. Is it because I need to set the output to 10m resolution (or 5m in the case of the new TRS2009 ground types) in order to get the thing to work appropriately?

 

[Jerker]

G'day treblesum81,

I am curious whats causing the "forest" effect. Is it because I need to set the output to 10m resolution (or 5m in the case of the new TRS2009 ground types) in order to get the thing to work appropriately?

...indeed, the resolution must match and since I have always set it so, I must presume that if you're not doing so, that may well be the cause of your woes...

...far be from me to be "pushy" (and this is certainly NOT a paid political announcement of any type) nor do I have anything to gain from a purchase of TransDEM by anyone else (other than the personal satisfaction that I may have made life 'better' for another 'Trainzer' - two, perhaps, if you include "the good Doctor")...

Jerker {}

 

[RPearson]

...
I am curious whats causing the "forest" effect. Is it because I need to set the output to 10m resolution (or 5m in the case of the new TRS2009 ground types) in order to get the thing to work appropriately?

I don't believe the resolution will cause the effects you've posted. And forget the 5m grid - HOG still outputs the same file format used back in UTC days. Still works though just won't make 5m baseboards for the initial .gnd file. I don't think Roland has the 5m resolution worked out for TransDem yet either.

If you're converting the DEM data to an ASCII xyz dem format you can use 10m resolution if you want. The program will have to interpolate the data to generate it unless you start with 10m resolution. Actually STRM data is on lat/long grid so you also have a conversion to UTM grid which means some interpolation regardless of initial resolution in the DEM data.

As to the forest, iirc the important thing with the ascii xyz dem format for HOG is to select the following options for the coversion to XYZ DEM:
XY format - UTM
Z format - meters (float)

Other options give crazy output.

Bob Pearson

PS FYI the only programs I use are MicroDem to create ASCII XYZ DEMs and HOG/Piglet to create the .gnd files. I gave up on chroma files long ago. I don't use TransDem though I think it's an excellent program.

 

[treblesum81]

As it turns out, it was indeed the resolution that was causing the forest. The datasets I've been working with have all be 1 arc sec or more in resolution (less is better here), which translates over to about 30m. When trying to convert that native resolution over to a Targa file, HOG can't properly interpolate the data since its only capable of working with exactly 10m resolution, so it just fills in the missing data points with empty elevations, resulting in the forest. The actual tops of the pillars do conform to the DEM, but every space around each is at elevation 0. The problem was corrected in my case by forcing the output ASCII xyz file to 10m resolution, leaving me with a nice smoothly contoured layout, however, even a 20 mile section of track produced a 600mb xyz file at that resolution, leading me to believe that I'd be constrained to ~40mi at a time of route, a very small amount when thinking about rolling out 2,300 miles.

Because of this (TransDEM works a little better with memory and should be able to process closer to 100mi at a time), and because of the spline laying feature, I've decided to go ahead and order TransDEM. I'm hopeful that it will help me produce this mess a little faster and more smoothly, though time will tell.

 

[geophil]

[OT]

I don't think Roland has the 5m resolution worked out for TransDem yet either.

Oh yes, 5m is working in my alpha version. But I won't support NED 1/9 arc sec DEMs. If sampled to a 5m grid they would quadruple memory usage compared to the 1/3 arc sec DEMs at 10m. But 5m is nice for ground texture overlays. It will yield a significantly sharper representation of USGS 1:24k topo maps. See here for screenshots. (There was a colouring error in the minimap which has been fixed since.)
[/OT]

 

[markbin]

, leading me to believe that I'd be constrained to ~40mi at a time of route, a very small amount when thinking about rolling out 2,300 miles.

There are only 2 programs that can add, separate made parts of a route, automaticly together at the correct coordinates; thats mapmaker and transdem .
but mapmaker ( free) works only with the 3 arcsec dems .
USA has better resolution 1 arcsec dems .