SSH logins can be limited to only certain users who need remote access. The ssh server is configured by the file /etc/ssh/sshd_config.

If you wish you can restrict remote access to specific users. For example

To restrict remote access to John and Karen to login add the following:

AllowUsers john karen

and restart the sshd service.