A Different Drummer

I was told that Henry David Thoreau said, "If a man does not keep pace with his companions, perhaps it is because he hears a different drummer. Let him step to the music that he hears, no matter how far away." I'm not sure if I have the quote exactly right, but I think that's the gist of it. Does anyone know if he really said this, and if so, in which of his works can it be found?

I personally have no idea, but I googled it and...

'If a man does not keep pace with his companions, perhaps it is because he hears a different drummer. Let him step to the music which he hears, however measured or far away.'
- Henry David Thoreau, Walden

Most likely he said that. Let's not forget that he was in agreement with Gandhi on civil disobedience.