Reading this I was also hoping the end result to be XenForo. IPB is great and all but I agree with the above, it feels heavy. XenForo seems light yet very powerful and at the same time, offers most of your requirements above (plugin or not).
I'll stick around no matter the choice though I think the best general direction for HD would be XenForo at this stage. Maybe some custom development is needed, maybe not. But it is a great system overall, probably one of the best if not the best.
Also, I haven't looked at the IPB development ecosystem but XenForos is huge, you can find pretty much anything you may need.