of a collar : having the ends fastened to the shirt with buttons
of a shirt : having a button-down collar
: having a very traditional and formal way of dressing, behaving, etc.
What made you want to look up button-down? Please tell us where you read or heard it (including the quote, if possible).