Is the community supported agriculture movement inevitable?

I was just listening to a group of food scientists and activists advocating a return to smaller-scale agriculture on NPR's Talk of the Nation, and I was struck by the number of times I have heard this exact same message, especially on NPR. I believe this goes hand in hand with the slow food movement, and a concept which I have dubbed time porn.
In an industrial society the price of most any 'luxury' item is cut down to affordable proportions. Most people can afford a stereo system, a fast car, a computer with relatively fast internet connection, a comfortable house and almost any food they could possibly want. A large portion of the luxury aspect of, say, a decadent piece of New York cheese cake is lost when anybody in Tulsa can go down to Von's and pull one out of the freezer. The physical pleasure of these things remains, but the exquisite exclusiveness of them is lost.
The people with money to spend face a challenge. How can they get added value if they already have everything they want? And what's worse, there's nothing special about having it. What is the one thing that is most valuable to hard-working modern people? In a world of increasingly busy people, the ultimate luxury and object of desire is the time to do things in inefficient but seemingly more satisfying ways. Let's say everyone has a warm factory-made wool sweater. A hand-made sweater has the warm coziness of any other sweater, plus it has the added value of all the labor that went into it. Super bonus points if it was made by either someone in America or Europe or at a humane, well marketed factory where all the women working there get to take courses for free.
The same thing is true of the small-scale agriculture movement and its proponents among the relatively well-off NPR crowd. In a world where everyone's going to Von's for that New York cheese cake, they want to go out to the country, where a rosy-cheeked farmer's wife will hand-make them a New York cheese cake with illegal raw milk and an extra helping of love. They leave with a 'superior' cheese cake, a story to tell, and the warm feeling in their hearts that they've helped support sustainable, environmentally friendly agriculture and America's rural culture. That's so much added value.
Not to say that the people behind this are wrong. If I were a farmer and I were concerned about the quality of my product, I would probably do exactly what these farmers do, organizing Community Supported Agriculture groups in which local people buy into the farmer's crop and then receive produce throughout the year directly from the farmer. I would, like the existing farmers, target exactly the NPR crowd, with their money and their desperate search for their next fix of added value. I have to say that I quite admire these farmers and their quest for excellence.
My qualms come when I hear people saying that this is a sustainable model for the entire food system. To reject large scale agriculture on the whole and expect the alternative to support the nation's or the world's food demands is naive at best and dangerous at worst. While it is clearly an important marketing point that these farmers and supporters claim that this system is viable for everyone, I fail to see how it can ever be more than the privilege of the well-off. It's like saying "People need to stop buying shoes from the big corporate shoe companies and start supporting local shoemakers. By doing so, we can meet the nation's demand for shoes in a cleaner, more environmentally friendly way, reducing the need for long-distance shoe shipping and at the same time supporting our local shoemaking community." Super bonus added value.