You need to find out why you have these feelings of panic.
Lie down on a bed and let your feelings come. Do not stop yourself in any respect and do not suppress any feeling. Find out as much as possible about your feelings for this man. Also try to find out as much as possible about him. His past will tell you, whom he is. Would it be possible for you to see a psychotherapist in order to help you understand your panic feelings?