There are three key variables in this decision:

1. Analysis of the idea itself, relative to other ideas
2. Estimation of market demand, by any means necessary
3. Introspection, individual and collective by the team, to figure out strengths/passions

Overall Process

There are formulas (they take skill to execute, not plug and play) for doing each of the 3 activities to get you about 60% of the way. But this is really about preparing your mind for intuition to work. When you've worked each of the 3 formulas to the 60% level (you'll know when you start spinning your wheels and hit diminishing returns), you set all your thinking aside and go do something else for as long as it takes, depending on the complexity of the thinking. Generally, if you spend 3 days on the formulas, expect to spend 3 more days with backburner simmering while you hike, watch Simpsons marathons or play video games.

Then you go back, review your output from the formula-sessions, and make an intuitive leap of faith to pick the idea that seems right. Often this idea will not be one of your original options at all, but just some elegant synthesis/Aha!/brainstorm triggered by the backburner simmering.

Skill Needed

The main skill needed is usually called "thinking through" ... a simple ability to take a large amount of ambiguous data, a weak skeleton of a process and just churn through, making stuff up as you go along, until you get somewhere interesting that has the signs of psychological closure.

The only real rule for this skill is take everything out of your head and put it into writing. Externalize all the thinking I am recommending below, even if it is a mess.

The Three Formulas

Analysis:

The basic tool here is a spreadsheet and lots of blank paper. One row per idea, as many columns as you need. Keep it simple, there will be no apples to apples comparisons here. The spreadsheet is more of a dashboard. Most of your cells will be empty or filled with speculative nonsense, but it will help you get a sense of both the important variables in the mix and your ignorance levels. Important things to include are: market size estimate, guesstimate numbers for launch budget (time+money), skills on board/missing, a "comments" section, branding/positioning, qualitative indicators, etc. One key column/variable is strategic value. Some ideas are self-contained market opportunities. Others are fertile ones that open doors to other markets. Make a note of these.

Switch back and forth between the spreadsheet and blank sheets of paper. It generally takes me about 10-11 sheets of paper/handwritten notes and diagrams to think through an idea at first order level, and about 1 hour of time. After that I have to do something else for at least 30 minutes before my mind is clear enough to tackle another idea. If you do this very intensely, a couple of days can yield about half a dozen ideas thought through at 10-page levels, and a summary dashboard in a spreadsheet.

But keep doing this iteratively until it feels like you are spinning your wheels and getting nowhere, that additional ideas have the same repeating DNA etc.

If you can't fuel at least 3 consecutive days of this type of thinking, chances are you don't have enough data in the hopper, and a very poor sense of the market. Go away and fill your head with TechCrunch for a few weeks before trying again.

Estimation of Demand

Your first release (an MVP if you are following lean startup ideas) is your first real test of demand, but you want to avoid stupid mistakes that you can WITHOUT building a product.

There will likely be very few market research reports (those are for mature markets). You should have a sense of demographic/psychographic profiles of your mainstream, but that's background information, and mostly irrelevant to the launch phase. Still, read what you can get.

This is basically grunt work. It is tedious, annoying to do, and an absolute must. For startup ideas, the single best way to estimate demand is to look for other startups trying to do similar things and trying to get a sense of how well they are doing. The nice part is that this is also competitive analysis.

And yeah, add another worksheet to your spreadsheet to capture this data. Things will likely work out so that each of your ideas is one of these categories:

1. No crowds: Nobody else is doing what you are, or anything close. If you aren't deluding yourself there are 2 possible causes: there is no market there, and you are an idiot, or you've actually come up with a true new idea. There's only one way to find out: build an MVP.
2. Crowd, no clear market: If there's a ton of very close ideas to yours, and all are struggling, it is a crowded ideaspace. If no clear winner has emerged, the market hasn't been created, but there is a good chance one exists (better than Case 1). You can join the fray, but your chances now depend on whether there is actually a market or whether it is a social-proof crowd around nothing. If there is a market, your chances of being the one to grab it are mostly random, if other things like talent and privilege factors are even (i.e. nobody has any decisive advantage like a big distribution agreement, a major backer, money in the pocket etc.). This is hard to accept, but is true in my experience. Most people in entrepreneurship are very smart. All commonly available information is known to all of them, and none of them make truly big blunders. Those who are doing things that reveal fatal levels of incompetence (like using Comic Sans as their main font and putting a "hit counter" and animated "under construction" gif on their landing page) can be safely ignored. Talent is over-rated. The decisive factors tend to be unknown ones at the outset. Talent matters after somebody stumbles upon an advantage.
3. Up for grabs: If 1-2 players seem to be pulling ahead by an order of magnitude (they've hit early product-market-fit, PMF in lean startup terms), but they haven't clinched a real market yet, and don't seem to be completely clear about what they are doing, it means there IS a market. The people who found it may or may not have the competence to actually hold on to it. You can win here by being a fast follower. Watch till they make their first mistakes and enter their post-hype slump, and attack where they are weak. A fool and his PMF money are soon parted.
4. Grabbed: Somebody has taken hold of the market effectively. They are 100x bigger than anyone else, growing faster than everyone, and the media treats them like they've arrived and that everyone is supposed to know who they are. Their brand has been established. Read up on disruption theory. If your idea is a disruptor with respect to theirs, you have a shot.

Introspection:

This is the easiest to describe, but the hardest to do. It takes a lot of self-awareness. For each idea, you ask yourself these questions:

1. Beyond sex, money, power and fame, why do you really want to do this? (not vague answers, if there's a real idea, there's usually some extremely specific hook for you personally, often something that is peripheral to the core of the idea).
2. How well could you actually do it?
3. Are you the right person to do this, or are there other very obvious candidates who could do it 10x better.
4. What do you actually know about the idea that the best "obvious person to do it" doesn't? If you have an idea for a new kind of CGI 3D animation technology, what do you know that George Lucas doesn't? (ILM will obviously know a gazillion things you don't, but all you need is actually that you know one crucial thing they don't).
5. If you pull it off, what's to stop a smarter, more powerful, richer party from simply taking away the candy you've found?

There is no rule for evaluating these answers. They should provoke thoughts. Think those thoughts through.

Now, simmer for a few days, review your notes, intuit away, and wait for the Aha! Take many showers and walks while waiting. They trigger the Aha! better than sitting around.