Oh Olivia, don't give pain to your heart and his heart as well.
If possible then better go there or tell him to be there with you. If you really love eachother and want to live whole life together then start plan for life somewhere together where it could be better for both of you.
Don't avoid sharing your real feelings because you are doing nothing wrong.
You love him and if he loves you so much the same way you do then I think he won't mind.
Wish you good luck. hope and wish you'll be together soon :)