Kentuck Derby trip..has anyone been?

i was thinking of making a holiday-incorporating the Kentuck Derby in May. For 2 single lads. Has anybody been there? where would a man start? I was thinking, flying into Chicago..4 nights..drive to Louisville..3 nights, and then onto New York or Miami for a few nights to round off. Has anybody been part of such a trip, and is it worth doing iyo? cheers