From what I understand, you need to register with an hosting service.

Create a subdomain called forum.domain.com at your hosting.

At your registrar, point all traffic from domain.com to the dns server of your hosting provider.

install forum. customize it.

You should be good to go.