Your speed will also drop if the line conditions are poor. This goes for both 56k modem as well as for broadband. If the client (you) and the server (ftp-site) have a hard time communicating, the service will fallback so there are fewer retries. This is called auto-negotiation.
The problem could be anywhere in the link. The line from your house to the pole, somewhere along the line, at the TELCO switch, and even at the ISP.
John