I have had merge route work intermittently in T12. Several times it would do exactly what afried is describing. I was told it is a "known issue" and that was about the last I heard. Fortunately I don't use it much.
When I wanted to merge large routes in TS10 - I had some trouble when the size of the route exceeded more than 550Mb. ( I got the Taddaemon error and had to shut down Trainz using taskmanager). In TS12 however ( after installing SP 1 and Hotfix 3 ), I can merge large routes ( one exceeds 1,2 TB ).
In your case - for TS10 - Using Shane's merging tutorial, should give you the result you want.
How about when merging 2 routes load up the main route and press merge route. Then pick the second route rotate to 180 degrees then set it on the spot to merge with first route. When i click merge and it's done the second route gets saved somewhere else on the new merged route, not the merged point that i picked. Is there still a bug in the merging process.