Sometimes, you just do need to be more literal. We experienced this with my own son when he was smaller - we directed him to "look both ways," and he did - but then he stepped into the street right in front of a car (I was holding his hand, so no harm done.) I later realized we needed to say "Look both ways to make sure the way is clear and there are no cars coming" this wound up meaning we'd have to wait for cars two blocks away, but eventually he generalized enough that he figured it out. Fortunately for us, we live on a one-way street!
In this situation, you may need to practice with your brother. Explain that there will be times when you need him to act, and don't have time to tell him exactly why. Find a word that works for the two of you. Also, I'd include more specific information on exactly what you want him to do - for instance, I would try something like "Danger! Run home fast and get inside the house!" "Danger! Get into the car quickly and lock the doors!" etc.
Explain to your brother that you couldn't figure out the intentions of this man, and that you were being cautious. While I think you totally did the right thing, do keep in mind that this type of predator - one who grabs children they don't know - is fairly rare (although somebody who doesn't back off the instant you start yelling is someone to be concerned about.) Tell your brother that you notice things that he doesn't, and you want him to trust you when you say "danger."