I have often said these words. I am a young pastor who happens to have responsibility, among other things, as advisor to the Board of Youth. I am a pastor to the youth, but also to the children, and also to the adults.
What got me thinking about this was John Ledetroit's excellent post, "Youth Pastors Anonymous" at Beggars All. Check it out.
I will say, in addition, that because so many in the LCMS assign the title "youth pastor" or "minister of youth" to the Associate or Assistant pastor, I get the feeling some perceive me as such.