Aside from the obvious danger of allowing a stranger into your house, if you're not married or in a relationship, the risk is somewhat minimal. I'm more concerned about news getting back to my SO than anything else. A friend of mine almost got busted years ago when a provider mistakenly rang his neighbor's doorbell, got freaked out when a woman answered, and then stupidly walked to the correct address as the neighbor watched. Lucky for him the neighbor decided to scold my friend privately instead of telling his wife, but that scenario should be enough to scare any of you married folks.
That said, if you are single but have a lot of nice expensive stuff, a provider might decide to come back with her friends when you're not home. The same can be said for the pizza delivery guy or the refrigerator repairman or the greenpeace activist who knocks on your door and sees your 96 inch plasma television in the background. I'm not about to make up a statistic, but I'm willing to bet a good percentage of the supposed random home invasions are not so random. There could have been a contractor hired to do some work at one point, perhaps the cable guy saw something he liked, who knows. Like GMack and Muff said, better to be safe, right? As a general rule, its a good idea to minimize the number of times you allow a stranger to see the inside of your home, period.