Absolutely you can and should trim the DEM before exporting to Trainz. Here is how:
Once the dem is in TranzDEM, use the simple route editor (under the route menu) to draw polylines approximately where tracks will be to tell the software to only create Trainz baseboards within X baseboards (I usually enter 5) either side of the polyline.
Also, be sure also to create 10m grid baseboards, not 5m grid. Once the route is in Trainz, you can then change baseboards to 5m only where needed for better texturing.
It's much easier to avoid creating unneeded baseboards in TransDEM than it is deleting baseboards in Trainz.
I agree with the 10m suggestion as it should give you a smaller gnd file size. And I strongly suggest the route editor whether you draw it in TransDem or as I do making my own cad drawings and import the route data from them. But note if you are using TRS19 the gnd file you get is a single gnd file in TANE SP1 format. When you bring it into TRS19 it will be split into 1000's of gnd files when you 1st edit it - 1 per baseboard as Mutton says plus 1 extra.
My route statistics are:
# Trainz Baseboards total = 38640
# actually created Trainz Baseboards after filtering = 4092
The gnd file from Transdem, exported for an older version of Trainz and 10m grid, is about 198,000KB. In TRS19 I get 4093 gnd files about 161,000KB total but the file for each baseboard is only 41KB. If I used the unfiltered map I'd be looking at around a 2 GB gnd file - about half the size you're working with.
If you can get topo maps of the area - in the US most everything is available digital nowadays and georeferenced so you can load the raster topos into TransDem and design a rather crude route using TransDem's builtin route editor using the topos as background. Save the route (vector data) when satisfied with it and then load it back in after you load the dem data into TransDem. You can filter the route based on that specifying as many baseboards each side of the track line as you think is necessary. I suggest loading the georeferenced topos without the DEM because it takes a lot less memory but you can have the dem and topos loaded when you used the route editor. Just make sure to save the route for future use.
If you're unsure of certain areas just add in some polylines with the route editor and you'll get additional baseboards exported in that area. I'd start small with 3 or 4 as the filter width to keep the exported size down. You can also export the route vector data you created as track splines. It probably won't be exactly what you wanted but can give a ball park feeling of whether the route will work and where changes are necessary. I suspect it may be too crude since it's main purpose is to filter out unwanted baseboards. If you have some special "track" splines that you use as tools in track laying - I have some I use to help in setting alignments for grading - you can export them as the track for your "trial exports".
You can just paint the topos onto the terrain along with the track route you drew in - much better than the old Tiger lines we used to use in US both amount of info and accuracy. Also much less memory intensive that the 2D or 3D tiles you could make from the topo maps. You can set a larger width and use 5m grid near the track line for the final export when you're satisfied you got all the areas included. If you don't want TD to export track splines for the final DEM you can have it paint the terrain to define the route based on the vector data you created to filter the route or even export map tiles in selected areas.
You don't have to create the whole route at once with TransDem. You can break it up into several smaller sections loading just a few topo maps into TD and design the smaller route sections in the same way. Export the smaller sections as Trainz routes and check them out individually and make changes to the track route(s) as necessary with TransDem's route editor. Once you have the all baseboard areas you need as several separate track routes you can create a composite route by loading all the smaller saved route data into TransDem over your large DEM and use the composite route to filter the baseboards and export as 1 large map. No need to merge the smaller maps in Surveyor unless that's something you want to do - maybe working on the separate smaller routes in Surveyor would be easier for some reason.
Just a thought,
Bob Pearson
Edited a few points.