While getting a feel for the landscape, let's move on to excessive things that are widely respected. Take gardening. 10 hours of labor, watering, weeding, and finally the reward of 5 pounds of carrots. There are a few kooks who attempt to justify this on economic grounds. In particular, you can find eccentrics in the organic gardening community. Yes, there are non-economic grounds, but in order to maintain relevancy, advocates of course ground a significant portion of their appeal in economy.
One of the non-economic factors in both cases is authenticity. The argument over sport is an argument over authenticity. Gardening is seen as authentic. Where authenticity has fallen is a bit arbitrary. Dipping a rod with a hook in the water may be sporting, but the use of nets was quite common in fishing. As for agriculture, things are far more complicated. Gardening has never been egalitarian. Agriculture has most often been done by servants, slaves, or some combination of the above. Perhaps, the superintending portion of gardening in the part we wish to retain. More likely is that the authenticity we seek is the one of being upper class. Gardening for sport after all is to be identify with the elite of the past.