Tolkien's works are sort of an odd duck when it comes to this sort of thing. A lot of what he wrote, he built from the ground up. Everything in the story was something of an original creation, from the languages spoken, the cultures, the way said cultures lived their lives. Thus, in order to make sure the audience understood what he was trying to get across, he had to make sure that what he was presenting made sense to his audience. He couldn't very well write a story called
The Hobbit without telling you what a hobbit was. That requires a bit of pre-planning and worldbuilding. But even then, Tolkien went the extra mile. He created a whole world, complete with mythologies, geographies, cultures, flora and fauna, a lot of which only came in snippets in his initial works or left out completely due to the constraints of the plot. I have a feeling that, had he been born in the 20th century as opposed to the 19th, he'd probably be game designer, rather than an author, simply because of the sheer volume of detailed background material.
I personally work on setting, then plot, then the characters. If I think there is something the audience might not understand, I try to write a concise explanation as to what it does and why it is there, plot and setting wise, then move on to the characters. Their motivations will dictate the plot, the setting will dictate their motivations. I sort of develop character archetypes, then work on the finer points of their personalities by thinking through how the setting would affect them. But, ultimately, the characters and plot are always the priority. You can't tell a good story unless you make the characters engaging. You may want to go into the full detail about a culture the characters visit, but you're going to lose your audience's interest in the characters and plot that drive the story. You can create as big a universe as you like, but you have to remember that the audience is going to be seeing a narrow focus on it through the confines of the plot as the characters interact with it.
Good lord, I hope that makes sense. I haven't had a lot of luck in getting my points across lately.
EDIT: Oh. One more thing. If you are making an original universe, something that always helps me is to write notes about the setting and characters, so I don't get my details mixed up.
"If the time should ever come when what is now called science, thus familiarized to men, shall be ready to put on, as it were, a form of flesh and blood, the Poet will lend his divine spirit to aid the transfiguration, and will welcome the Being thus produced, as a dear and genuine inmate of the household of man."
- William Wordsworth
: 
[ 
... ... 
]
)