What to do?

I'm at a loss as to what to do about the relationship between Johnny and I. Yesterday I had told him that I really wanted to spend some time with him last night. We don't see each other very often during the week because of work and school and last night since he didn't have to go to class I thought it would be a great opportunity for us to spend time together. Well after about an hour of hanging out he wanted to go play chess with a friend. I said that was find as long as he came back early. (I'm trying to be understanding and let him do his own thing so he doesn't feel suffocated.) He said he would be back around 10:30. Well at 10:30 he called and said that he would be a little late. I told him to hurry so that I didn't get too tired before he come home. He said he would. After we hung up I texted him telling him to please come home now because I really wanted to see him and spend time together. He said he was copying his friends notes from the class he missed. I went to sleep. Johnny arrived home at 3am. I'm upset of course. I'm crying because I feel like he doesn't want to be with me. Why should I have to beg for him to spend time with me? I told him that I feel like he doesn't care what I want or say. He said he feels the same about me and that I never listen to him or care what he is going through. I just can't believe he thinks that. I'm sacrificing so much for him and I'm trying to be so understanding but I need him to meet me halfway. I just don't know what to do anymore.