SSH on your server has a config file that is full of goodies. Be very careful editing this file! You can very easily lock yourself out!
The file on most distros (we'll assume RHEL based Linux here) is located at /etc/ssh/sshd_config. It can be opened with your favorite text editor.
Here are some of the default config options and suggestions:
Changing this will change the port that SSH listens on. Many hackers love trying to bash your server on port 22 to try and brute force. Changing this to a non-standard port can certainly help thwart attackers.
If you have a wheel user setup, then this can certainly be set to "no." That way, no user can login directly as root. Be sure you have a user added to wheel! This is crucial!
Rather than logging in using passwords, you can generate private and public keypairs and set the public key on the server to login you in automatically without entering in your password. If you use that, definitely keep this set to "yes"
As explained above, rather than logging in using passwords, you can set keys to login. If you switch this off, you will NOT be able to use a password to login. At all. Make sure your key works before disabling this and you're in as root!
More questions about the SSH config file? Type in "man sshd_config" from shell to open the manual!