The hill building tips that have been given in a sticky can help.
The main thing to remember about mountains is that they are extremely large. The mountains outside my window have bases that can exceed 10 miles, and the peaks rise 5,000 feet over the valley floor or more. A route can look ok with mountains down to about 1/10 scale or so, but less than that and they approach the cartoonish, unartistic look that you describe.
I have found that building them slowly works best. Turn the dial on the right almost all the way down, and the dial on the left to as large a circle as possible. Then work on the mountain over at least a 4 board area, and try to keep the sides sloped less than around 60 degrees at most, and slopes of under 45 degrees work best.
To get the proper scale, the mountain should rise at least 400 feet above the base or "valley" level of the map. It may be that you need to expand the mountain base as you raise it in order to maintain a good slope within the range that looks natural.
If this scale seems excessive for your computer, build the mountain that way anyhow, and then cut the mountain off on one side by removing boards after it is built. before adding textures and trees.
Remember as you go that mountains are seldom symmetrical, and that their slope of ascent varies, with the slope gradually getting steeper as you near the peak. When you get very close to the peak, then exceeding 60 degrees will look ok.
I hope that gives you the steps you need to succeed.