Saying “we need to talk” is a good way to get the message across of severity, importance, and seriousness which I’m sure relate to how you feel. It’s best to confront these things head on and be as clear and expressive as you can be with him. Don’t go overboard and scare him away, but you have to let him know how you feel.