Is your panel a dedicated server in a multiserver setup? Then just set a firewall rule that blocks any traffic that is not "you" for port 80/443/8080 depending on your setup. If you want to only block the Panel itself but not block access to these ports you could add a "Require ip" block to the vHost for the panel
I need to prevent others from accessing the control panel. Via :8080. I need only me to be able to access it from my IP. For security reasons. But I need the 80/443 ports to be public. Because the server is running a client website.
Then just block the port 8080 for others in your firewall or add a Require ip XXX.XXX.XXX.XXX to the 000-ispconfig.vhost.