Usually it's whoever is closest to the pin will tend and pull the flag.
That one doesn't nescessarily have an actually rule. Generally I think the person who is farthest and is about to putt SHOULD NOT grab the flag. Basically of the other three who ever is closest to the flag at the time (Person not ball) should grab it. It's one of those lesser ettiquette things that you shuoldn't beat yourself up over.