There are so many challenges involved in writing a screenplay that solving any one of them feels like a triumph. Surviving the process requires that we embrace that feeling just long enough to keep us going, then take a step back to determine what other issues need to be tackled. Get too excited that you made it to 100 pages and you may not realize (or want to realize) that 15 of those pages are dead weight; spend too much time rejoicing about nailing your underlying theme and you could fail to see how clunky the plot is.
The final screenplay needs to balance all these things (and many more), and that in itself is potentially an even greater challenge. For all the elements of a script to exist in harmony, we need to make creative compromises at every turn; only they can never look like compromises. (Songwriters and poets have been familiar with this principle for ages -- in a good song or poem, you can never tell which words or phrases in one line were changed or sculpted to rhyme with the next one.)
Some of the most difficult compromises spring from the need to reconcile logic with emotion. A screenplay needs to proceed logically -- by which I do not mean that the characters need to behave logically, but rather that the characters' actions must have logical consequences. On the other hand, a screenplay also needs an emotional sweep to it; audiences are most engaged when the characters on screen run through a range of feelings. We don't go to the movies to see a sad person get sadder (Lars von Trier films excepted), or to see a happy person get happier. We want to see desperation turn into hope, mistrust turn into faith, pride turn into humility. And frankly, even that's not enough; that character's newfound feeling/outlook needs to be reversed before the end of the film, pushing the character even further back on the emotional continuum than he was at the beginning, and then, pushed beyond his limit, the character makes a final difficult choice that transforms him for the better.
All of which sounds great until you set out to craft a plot that makes these enormous changes possible. Good luck! Many have tried and failed. We've all seen movies that run the emotional gamut, yet don't make an ounce of sense because the writer left huge logical gaps along the way. Do we excuse the plot holes because the emotions are so powerful? Rarely, and only with the help of hugely talented directors and actors -- the benefits of which our lowly spec scripts do not have.
But if we instead embrace logic, focusing single-mindedly on ensuring that every scene proceeds inevitably from the last and the internal rules of our story are never defied -- then don't we run the risk of writing a flat, boring screenplay? Well, absolutely. Because merely taking a good idea and following it to its logical conclusion is nowhere near enough to make an interesting movie.
So this is where the rubber meets the road. To write a good screenplay, you have to incorporate everything I've described above -- the emotional sweep and the rigid adherence to internal logic.
The bad news is that (unsurprisingly) this is an incredibly difficult task. Really, it can't be overstated. Most people will not be able to do it. (I'm not being elitist when I say that, because as of this writing I am still comfortably ensconced in the category of "most people.")
The good news is that it's possible. How could it not be? As writers we have total control over every detail of our characters and every microscopic turn of the plot. As corny as it is to say, we really are limited only by our imagination. If the plot we've conceived is incompatible with the emotional journey we want to send our characters on, then we can change it -- or we can change the characters -- or we can tinker with both of them, making whatever small or large alterations are necessary to ensure an organic fit between plot progression and character arc, and hence between logic and emotion. We are never locked into any particular path; we only think we are because our brains are naturally lazy and hesitant to abandon anything that we've put any effort into. But we can and should feel free to get rid of anything that doesn't work -- anything that impedes the mission.
The great news is that the payoff is huge. Experiencing a story that delivers on both the emotional and logical levels is deeply satisfying. Watching a character put through the emotional wringer through a series of plot twists and turns that at first surprise us but make perfect sense in retrospect -- this is why we go to the movies. And a screenplay that can pull this off will go places.