Why does my dog only listen to certain people?

If I'm gone more than 3 hours I leave my puppy with my grandma. At home when I take him out he will go pee and poo at his designated area but at my grandma house he just goes on the floor when I'm gone. I told her to say "go potty" when she take him out but he just won't go. However, when my husband picked him up from my grandmother's house yesterday he peed when my husband took him out. How come he only listens to my husband and I, and not others? I told my grandmother to be firm or what not but he just doesn't listen to her. My grandmother also baby him a lot, and let him do whatever, and I tell her she shouldn't but she does not listen. What can I do to stop him from peeing and pooping on my grandmother's floor?