NEDF, a step forward with Australian DEM data

[geophil]

Until now, Australian Trainz routes were build either with 3 arc sec SRTM data or 1 arc sec ASTER GDEM. SRTM 3 arc sec is somewhat low on resolution, particularly in mountainous regions. ASTER, on the other hand, nominally has a higher resolution, but comes with a coarse surface. Hence, both are far from being regarded as the ultimate DEM data source.

Now, however, a gradual improvement seems on its way. It's called NEDF, National Elevation Data Framework, is operated by Geoscience Australia, and has opened its portal at http://nedf.ga.gov.au/geoportal/catalog/main/home.page sometime last year. (A TransDEM user pointed me to this address.)

They offer - for free - SRTM derived data. The derivatives are based on SRTM 1 arc sec (!), not 3 arc sec, and come in three flavours: regular, smoothed and hydrological.

Unfortunately, TransDEM 2.3 cannot yet read the data format, which is called ESRI Binary Grid, but I am working on it and achieved first results in my lab today. With a bit of luck, the task may be finished before Christmas.

Let us compare a few samples. They are taken from the Blue Mountains, NSW, west of Lithgow. The clipping is 4.5 x 3 km.


SRTM and ASTER, the familiar sources

SRTM 3 arc sec, blurry by nature.


ASTER GDEM 1 arc sec, sharper, but rather bumpy.



Introducing NEDF

NEDF 1 arc sec, regular


NEDF 1 arc sec, smoothed:


NEDF 1 arc sec, hydrologically smoothed:


In my opinion it looks best in "smoothed" flavour.

Is it accurate? One of the Open Street Map renderers has contour lines, I don't know what the source is, but our NEDF comes close.

NEDF 1 arc sec, smoothed, with OSM cycle map overlay:


To me NEDF looks like being worth a try.

 

[ElStoko]

Thank you for bringing our attention to this. I have just now registered to access the data.
Will you be announcing your progress with enabling TransDEM to read ESRI Binary Grid in this thread, or elsewhere?

 

[geophil]

Will you be announcing your progress with enabling TransDEM to read ESRI Binary Grid in this thread

Oh yes, I'll keep you updated here.

 

[geophil]

I have released TransDEM 2.3.1 today, which adds ESRI Binary Grid as a supported DEM format. When you unpack the NEDF download package, you'll find number of .adf files in one of the subdirectories. Select any of the .adf files in TransDEM to open the DEM.

 

[geophil]

I have posted a tutorial in the TransDEM forum on obtaining 1 arc sec DEM data via the NEDF portal and processing it in TransDEM: NEDF tutorial for TransDEM.

 

[ElStoko]

Opening NEDF 1 arc second DEM data in TransDEM 2.3.1

When attempting to open NEDF 1 arc second DEM data from Geoscience Australia I find that only rarely, after many repeated attempts, does TransDEM 2.3.1 open successfully. For well over 90 percent of the time, I receive the message "TransDEM has stopped working" when an open request is made. I do not experience this problem with earlier data sets (e.g ASTER 1 arc second DEM data). I am wondering if others have experienced a similar problem. Roland (Geophil) does not.

 

[geophil]

Jerker reported a problem in the TransDEM forum but that had to do with unpacking of the zip file not working properly.

I did some further analysis to find a reason for TransDEM to crash. The only thing I detected was a possible failure with the meta data extraction. Firstly, it should not happen and I'll fix this. The underlying library does not check for a null pointer which has to be done at the caller's level. TransDEM then would no longer crash but report an error message instead. It still won't open the DEM.

The ESRI Binary Grid format is one of most complex formats I have seen so far. It consists of 6 related files, all .adf type but very different contents. Most are binary files, one is text. They always should be left untouched. The way TransDEM reads them is by directory, not by file. You select one of the adf files, then TransDEM will parse the path to that file and use the containing folder as the address.

The first step is extracting meta data. For ESRI Binary Grid, this consists of georeferencing plus file layout. The call to the underlying library may fail. I provoked this by removing one of the 6 .adf files. If it fails, it currently causes a crash, in the destructor of the wrapping class.

If the meta data extract succeeds, the second step is to retrieve the data. And here, a strange thing can happen. I have seen this in two NEDF examples, but never in any of the USGS DEMs which use the same format. In these two cases, the layout retrieved by the meta data claims a much larger file than needed to fulfill the georeferencing specification. And it turns out that this claim is unjustified, those extra parts do not exist. Trying to read data that does not exist in the file results in an error and TransDEM aborts the read loop.

In one of the two cases, the one reported by ElStoko, all required data has already been retrieved, in the other case invalid blocks are encountered much earlier and no useful data can be read. In the future TransDEM will ignore all invalid blocks and continue reading to the bitter end. But these block reading errors would not lead to a crash.

I'll continue trying to reproduce the crash with Leigh's files but have not been "successful" so far.

 

[ElStoko]

Thank you for the details you have provided, Roland. Is there anything you would like me to try that would help you divine why my load failure rate is so high? As reported earlier, I have TranDEM installed on 2 computers, and both experience the same problem. We have eliminated the possibilities that I had not extracted the files correctly and that I may have separated the 6 ADF files into different folders.
Leigh

 

[geophil]

I have tested on a 32bit XP system now and could reproduce the crash there. I will set up remote debugging and identify and hopefully fix the problem, but it will take a few days. I am working on TransDEM 2.4 at the moment and this would be a fix to be applied to 2.3. With the help of Subversion and its magic functionality I'll try to fork off a branch at the 2.3.1.2 tag without messing up the whole source code environment.

 

[geophil]

Update:

I could reproduce the error in a remote debugging session. And it is indeed related to the false meta data information and data blocks that don't exist. Sometimes, the underlying library throws an error and sometimes it prefers to crash instead. What I now do is to prevent even the attempt of accessing blocks that may not exist by checking against the overall dimensions of the DEM. It appears that the overall dimensions were always correct while the block information could be wrong. I would presume an error in the GIS software at the NEDF site used for generating the tailored DEMs. Anyway, my work-around seems to do its purpose.

The patch for TransDEM 2.3.* is in the making.

 

[SharkNose]

Thanks for the update, geophil. I'm looking forward to the new versions! :wave:

Andrew

 

[geophil]

I have uploaded the patch, available on my website.

 

[ElStoko]

Patch is successful

With the patch installed, I have loaded twice each of three NEDF ESRI Binary Grid folders containing 6 related ADF files. Whereas previously TransDEM would crash nearly every time, the patched TransDEM v2.3.1.3 opened the ADF file set successfully every time. Thank you Roland for your effort to speedily resolve this problem.
I have not exported any of the loaded NEDF files from TransDEM to Trainz, so I can't as yet comment on any improvement over the ASTER 30m data I have been using.

 

[Jerker]

G'day geophil,

...as above, Roland, we thank you for the extra effort you put in to overcome these problems as we find them (...above and beyond..., I am sure). Although I have downloaded the new patch, I have yet to install it but as you noted above, I have not encountered this issue in 'my travels', as yet. One thing that has occurred to me is whether or not anyone has thought to advise Geoscience Australia of these issues? I do not know if we are using the data in a way they had not envisaged or whether the issue is 'ingrained' but I am sure that they deserve to be advised that something is sometimes not quite right...

Jerker {}

 

[geophil]

One thing that has occurred to me is whether or not anyone has thought to advise Geoscience Australia of these issues?

A couple of years ago I told Land Information New Zealand about a formal error in the WMS meta data and a few weeks later the service was down for good. So, I don't know whether that's such a good idea.

Well, I guess Geoscience Australia will be using ESRI software to generate the DEM clipping while I am using low level GDAL functions to read it. It could be some kind of misinterpretation on the GDAL side but I doubt it. The values are plausible and appear to make sense, if taken for themselves.

I have experienced all kind of oddities with the geo data of this world and I am not taken by surprise any more. :wave: