When I was younger the difference between the Hugos and Nebulas to me was that the Hugos were the popular selection and the Nebulas were the professional one. Obviously professional writers would select based on literary merit and have a higher set of standards than just fans who show up at a convention, right? I've heard that sentiment many times; the Nebulas are supposedly the "literary" award.
I lost this view once I knew a bit more about the SFWA and their petty squabbles. If I hadn't been dissuaded of it before then reading all the Nebula winning novels would certainly do it.
The question remains for me, though, of which award really goes for literary merit. To that end I'm conducting a survey where I will be final judge of that very loosely defined term. I'm writing this paragraph before I perform the count so I don't know who will win but I'm betting on the Hugos. I am only counting novels since I'm still twenty-one stories away from finishing off the Hugo winning short fiction and I haven't dug too deeply into the Nebula winners.
For my purposes I'm judging literary merit partially based on if the author was attempting to expand the genres of science fiction and fantasy beyond the standards of the time. I also am looking at the prose so a clever and well written book might get the nod where one with a good big idea but pedestrian writing would not (few Nebula and Hugo winners actually have bad prose despite the efforts of Robert J. Sawyer to break this barrier) . So Dune gets marked as literay because it is a gothic novel in science fiction despite the fact that I don't like the book and just about every epic fantasy novel in the past thirty years has been in the gothic form (even if most of those authors couldn't tell a gothic novel from a gothic cathedral). I'll bring up some borderline cases after the poll and how I went with each for further examples.
This is not intended as a measure of quality; I can point out several novels in both that are self-important literary messes that
Ready? Drum roll please.....
Literary: 26 - 46.4%
Not Literary: 30 - 53.6%
Literary: 23 - 52.3%
Not Literary: 21 - 47.7%
Not what I was expecting at all. I found that I was judging mainly literary for both up until 1980 when a switch suddenly flipped on the Hugos and I started judging mainly not literary. The Nebulas underwent a similar shift at 1990 but it gave them an edge.
Here's some of the borderline ones:
Starship Troopers - While Heinlein may have created the template that military SF would follow after it was still following in other's footsteps and made it fall on the line for me. I chose literary mainly because of the political lectures; it made it seem like Heinlein was trying to pull in Ayn Rand's literary style at the height of her popularity.
The Falling Woman - I wavered a lot on this one but in the end I decided that the shift to including more women's perspectives within SF was well represented by this book and chose literary.
The Mars Trilogy - I thought long and hard on Red Mars and Green Mars due to the science aspects. In the end I felt it was an extension of existing Campellian forms rather than something distinctive.
The Moon and the Sun - I had to pause a second to think when I this this: how prevelent was historical fantasy before the late ninties? Then I realized it was very prevelent and shoved it under not literary.
American Gods - I may like Neil Gaiman as an author but after a few moments thought I decided that he was both just revisiting themes that he had already been over repeatedly and they weren't exactly unique when he did it. I'm a harsh critic sometimes.
The Diamond Age and Rainbows End - Both novels are part of the standard "Here's what a technology will do to us in the future!" form of science fiction and both are exceptional examples of it. I wound up saying yes to The Diamond Age because of its Victorian novel roots and multilayered structure but no to Rainbows End because it didn't really offer anything new in terms of writing.