So I understand how you feel.
What are the reasons you can't understand?
1. An unfamiliar accent
2. The situation - listening to people over the phone, on a conference call, in a group or somewhere noisy are all much more difficult.
3. You do not know the context.
4. People are using a lot of slang or difficult vocabulary (such as phrasal verbs.)
5. You are having a bad day. (Yes, this real. If you are stressed about something else, tired or sick, you will find it much more difficult to understand English.)
What can you do?
1. Accents: The more you hear an accent, the easier it is to understand. You can prepare for this: if you know you'll be working with Irish people, prepare yourself by watching some Irish films. If it's a situation you can't prepare for, you'll have to just do the best you can.
The other day, I was watching a film set in Arkansas and the characters had very strong accents. I do not often hear this accent, and sometimes I couldn't understand what they were saying, even though they were speaking English and I am English!
Always remember: it's not that you're bad at English, even other English speakers can have difficulty if they are not familiar with the accent.
2. Situation: Prepare. Always prepare for a conference call, even if it's only 5 minutes - check what you will be talking about and check that you know the necessary vocabulary. You can prepare for a social event as well, by knowing how to ask questions and keep the conversation going. You can also prepare a few topics to talk about.
3. Context: again, preparation is important. If you are going to be in a situation where people might be talking about current affairs, listen to the news in English, or read a newspaper so you have some relevant vocabulary.
If you have no idea of the context, try to get a few of the words you hear and guess what the context could be.
Practise predicting vocabulary you will hear before you do a listening exercise (for example on this blog) and this will help you to understand what the context is in real life.
4. Slang and difficult vocabulary: If you are listening to native speakers talk, you probably won't understand all their vocabulary. Picking out the words you do know is important. Listening to songs and reading the lyrics is a good way of learning everyday language and slang. Phrasal verbs are everywhere. When you come across one*, find out what it means and then try to use it (DO NOT try to learn a list of phrasal verbs, it will only give you a headache.
*come across is a phrasal verb which means find something by accident, without looking for it.
5. Are you having a bad day. It's ok. Just relaaaaaax! This is the most important thing. If you miss some words, don't get in a panic. Keep listening, maybe you'll understand the next word. And do not get annoyed with yourself. There will be situations where you don't understand. Everyone has good and bad days. I have been speaking French nearly every day for 9 years, and there are some days when I my accent is terrible and I can't express myself at all. It's a part of language-learning, everyone experiences this. With practice and time, it will get better, I promise.
6. Very important - don't be afraid to ask. It's ok to ask someone to repeat, or to ask for clarification. It's ok to tell someone you can't understand when people speak fast and ask them to speak slower. Never be afraid to do this.
If it wasn't really an important situation, and you didn't understand anything, just let it go. Don't think about it again. You are not bad at English! You are getting better, it just takes time.
Remember: learning a language isn't easy, and there will be times - many times - when you don't understand. But the more you practise listening to English, the easier it will get.