@Dicrostonyx
Note that fair use is a system in which all four elements are examined collectively; it's not enough for something to be transformative, it also has to meet the other standards. Also, transformative in this context doesn't simply mean a new story, there are complex legal definitions in play.
This is not correct. You do not have to meet all four standards in order for something to be fair use. Indeed, Amazon has won 'fair use' rulings for decidedly commercial use specifically by meeting the 'transformative use' criteria.
Yes, all four standards need to be examined. However, you're misunderstanding the 'substantiality' test. The characters themselves are a tiny portion of e.g. Harry Potter. The world is another portion, the plot is another potion. If someone uses e.g. 20% of the characters and 20% of the world, along with 5% of the plot, is that 'substantial'? For that matter, if someone uses 100% of the characters (which are perhaps 5-10% of the work) and 100% of the world (which is perhaps 5-10% of the work), but none of the story, is that 'substantial'?
The answer is that we don't know.
The Star Trek case is interesting but not particularly relevant. The court ruled that it's not transformative (and, indeed, it likely isn't), because it is highly similar to the 'plot' (such as it is) of 'Oh, The Places That You'll Go'. It's a commercial work, and it fails to be parody.
It has very little in common with non-commercial fanfiction, which tends to be highly transformative (e.g. write the same story as 'Harry Potter And The Sorceror's Stone' with the same characters and that's a rip-off, not fanfiction) and wouldn't try to be claimed as parody.
I agree with the comment about findings of fact, with the important caveat that courts do look at similar rulings (otherwise all of these rulings being tossed about would be of no import whatsoever). If a court were to rule that non-commercial transformative fan-fiction was clearly protected, other courts would likely take notice of that.
I agree about zero-sum games. My point is that there is, in general, a strong disincentive for both traditional authors and fanfiction authors to test this in court. If there was a high likelihood that courts would find against fan-fiction, my guess is that some anti-fanfiction author would've bankrolled a challenge in order to get a ruling in place. At 50/50 or less, why would you make that attempt?
One other note: while it doesn't have any legal relevancy in a civil suit, juries would likely find the argument that author X allowed ten thousand fan-fiction stories to remain in place but singled out this work for a suit to be a reason to reject the suit, unless the work were particularly offensive in some way. Of course, if the work actually was particularly offensive in some way, that would play against it setting any form of precedent.
My main point in this discussion is simply to say that people who say that 'fanfiction is theft' or 'fanfiction is illegal' have no legal basis on which to make those claims. There is a very straightforward argument, supported by (non-fanfiction) case law and legal theory, in favor of fanfiction being legal. There is also a straightforward argument in favor of it being illegal. Since no US court has issued a ruling either way, that's as far as we can go at this time.
Nevertheless, there are a relatively limited set of things for which one can say 'X is illegal, but there has never been a successful civil or criminal proceeding against anyone who did X', beyond the sort of laws that show up in 'Weird Law' articles (many of which were successfully prosecuted 'back in the day' but are anachronisms now). The lack of any proceedings is at least indicative that those opposed to fanfiction have relatively little faith that they can win. This is especially noteworthy given that the most notoriously anti-fanfiction authors 1) have very, very deep pockets, 2) in some cases, have corporate patrons who also have very, very deep pockets, and 3) have a plethora of fanfiction authors writing in their 'universes' who aren't even being hit systematically with cease-and-desist letters, much less lawsuits.