Here is some information which may help you understand:
1. If you were overweight as a child it may be that you have a large amount of small fat cells in your body. They are very hard to get rid of by diet. Even if you temporarily manage to empty them, they easily fill up and you gain weight again.
2. Studies of overweight and normal weight tennis players have shown that the latter move around the court more. Can it be so that you tend to move less than those you compare yourself with? Small differences have a large effect in the long run.
3. If you eat food which gives you 100 kcal more than you need per day, your weight will increase by 50 kg in 10 years. You don't need to eat a lot more than others in order to gradually become overweight.
4. Those who are overweight often eat more fat than those who are slim. Consider a lunch where a slim person eats a very large plate of salad and you, as an overweight person only eat a few cheese sandwiches. You will have eaten far more fat producing calories, but probably think that you have eaten very little compared with the 'gluttonous' slim person.