It's a cliché that gets trotted out whenever you put time into something that has no direct commercial value, or isn't something that society arbitrarily regards as valid.
It most commonly follows phrases like:
"Check out this light saber I built!"
"I finally got a Playstation 3 platinum trophy!"
"I finished my costume for (insert your favorite fantasy/scifi/gaming con)"
But the sentiment is so common among stupid jerks that you'll hear it in all sorts of situations. I think what it really means is:
"I am not capable of doing what you did, and would like to invalidate your work to make myself feel better."
Fortunately there's an easy comeback that cuts these people to the quick:
"Yeah, I should really be spending my time watching reality tv, like you and the rest of the boring people."
*And it's a stupid comment anyway. I don't sink my time into rediculous things because I have too much time, I do it because I'm crazy.