Al took it personally.
He understands why his mom left his dad, since his dad wasn't there for her. But she in fact left also the young Al, and for that, you just can't forgive.
My mom had the same thing, she traded her hard work for her success. But the truth is, what she was trading in was also the shattered pieces of my heart. And for that, we shall always have our distance.