Server begin to freeze after a short amount of time

Hello there .
I installed ISPConfig 3 using perfect server guide on Ubuntu 15 . My problem is that after some time host begin to freeze ssh connections. And i cant login and do anything . SSH with debug shows following . Tried to connect from different places and different open ssh versions. Got same result.

Code:

OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/user/.ssh/config
debug1: /Users/user/.ssh/config line 5: Applying options for yourhostname.com
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 102: Applying options for *
debug1: Connecting to yourhostname.com [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug1: identity file /Users/user/.ssh/id_rsa type 1
debug1: identity file /Users/user/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2

 

VPS Digital Ocean 512 mb Ram

Code:

MemTotal:     512909312
MemFree:     51724288
MemAvailable:     79982592