Each year, Halloween gets me to thinking about the broad range of actual horror within the horror genre. What scares one person can make another role their eyes. What gives me nightmares (anything with snakes on people) might be an amusement to another reader. The "best all time scary story" has got to be one of the most subjective lists anyone is every going to attempt. That said, here are my tops in three categories:
Non-Fiction: Helter Skelter, The True Story of The Manson Murders by Vincent Bugliosi - Bugliosi may go over the top of the facts in his attempt to paint the Manson clan as evil incarnate, but it wasn't a very high leap. This is the book that convinced me that very bad things can happen to good people by pure random chance.
Short Story: The Lottery by Shirley Jackson - for pure creepiness, I don't think this story can be beat. You have no idea what's coming the first time you read it, and when you do know what's coming the story is even more horrendous.
Novel: Ghost Story by Peter Straub - this is everything a scary story should be in my opinion. It's got the foreboding thing down pat, ghosts with attitudes, and characters who aren't so stupid that you don't care that they're being haunted. This is one of only a handful of books I've ever read that had me hearing things and sleeping with lights on in my own house. (Straub's website is one of the most fun sites by an author that I'm aware of, too!)
I'm sure there will be other "best" lists published today, but one of the better ones I've found so far is seven years old. As I don't think there's been any truly classic horror published recently, I'd recommend this list from CNN if you're looking for some more great scary recommendations.