I try to replicate how it would be in the real world, so I build my terrain first, then lay the track. This recreates the problems a real track-laying engineer would face.
In the real world, the track-laying engineer must balance the requirements of rail transport (no sharp curves or steep grades) against his budget. Major transformations of the landscape (cuttings, embankments) are costly, so he must keep these to as much of a minimum as possible while still making it possible for trains to traverse the route. For example, it might be cheaper to lay 20 extra miles of track to go around a mountain range rather than drive straight through it with cuttings, tunnels, viaducts and embankments all over the place. But if a mountain range is long, then there's no other choice.
In this scenario, lay your track to follow the contours of the landscape as much as possible while avoiding sharp turns (buttes and valley heads are particularly problematic here.) If you're crossing a range, find (or create if you're doing a fictional line) a valley that runs in as close to the watershed as possible, and have your track run along the valley walls, gradually rising to the top near the valley head, then along the watershed until you come to another similar valley on the other side, then down along the hillsides again. If along the valley run you encounter a butte (a length of hill that projects from the valley wall into the valley), this is where you start building cuttings and tunnels. Similarly, if you encounter a tributary (a branch valley bearing off from the main valley), this is where you start building embankments and viaducts.
Avoid building cuttings that are very deep, and embankments that are very high, as in the real world these would be prohibitively expensive. As a rule of thumb, if a cutting or embankment would be more than three or four times the height of the train, build a tunnel or viaduct instead. Think in terms of the amount of rock that has be dug out or trucked in and how much this would cost.
Once you've determined the optimal route for your track, smooth your track out so there are no bumps, by using the Spline Height adjustment tool (the button on the top left of the Advanced panel in track-laying mode) - just click the spline vertices with this tool so they turn yellow. Notice that where the track used to run roughly over hills and valleys when the vertices were white, the track now punches straight through the hills, or floats in the air over a valley. Now use the Spline Leveling tool (the button on the top right of the Advanced panel) to smooth the terrain to the track. This will automatically create your cuttings and embankments for you. Again, if as you're doing this you spot a very deep cutting or high embankment, undo the spline leveling (Ctrl-Z), cut out that section of track, and lay a tunnel or viaduct through the broken section.
Finally, some sections of your track will have too-sharp curves or "dog-legs" in S-bends. Use the Insert Vertex tool to break up dog-legs into smaller sections, then straighten the section with the Straighten tool where the dog-leg was to smooth out the curves. Where curves are too sharp, you'll often find the leveled embankment or cutting is wide enough to allow some sideways adjustment. Use the Move Track tool to drag the vertices in, reducing the sharpness of the curves wherever possible.
Here are some pictures to demonstrate the process:
Here are some examples of my work to illustrate further principles:
Hopefully this will give you some ideas about how to build your layout!
