Most entrepreneurs are so excited about what they are working on they tend to see the market potential as like the ocean - vast and almost without limit. Many would be investors have heard countless times "if I can just get one percent..." If you use that line and you don't get an eye role from your audience, you sure deserve one. You must learn to "boil the ocean" and distill it down to your principle market and not more than one or two add on markets.
There really are two categories for new endeavors - incremental innovators and new products. An incremental innovator is taking a classification of product or products and trying to enhance the value of their offering in that space. How do you know if you're product adds incremental value instead of being innovative? One simple question gives the answer. Can other people use a substitute to get the same or similar value? You need to take a hard look in the mirror and really, really be truthful if other existing products are your competition. Why? It's critical. Instead of creating a market, you're entering into and trying to innovate within a defined market. This requires some different approaches. If your product's end result is coffee in a cup I don't care that you used cold fusion technology to produce the coffee, you're product is in the coffee maker market.
If you're product is the first ever then you have to take a different approach. In my example above if you invent a cold fusion method then you have a new product - forget about the coffee.
Putting You In a Bucket
You are going to hate this as an entrepreneur but everyone is going to try to take your offering and put you in an existing market/product category bucket. This is the psychology of the human mind. We take information in our environment and we try to associate it with what we know. Categorization is how the human brain works. Resist the urge to fight it. Instead, use that psychology to direct people to a market or category of products that presents what you're doing in a favorable light.
Boiling Down Incremental Products
I find a tool of large companies to be a great for entrepreneurs- market matrixes. Start by defining all the various existing markets your offering can apply to - consumers, commercial, scientific, government, hobbyists, etc. These are your market segments. In this exercise you want to define products that you might well be lumped into with each one of those segmented markets. You then want to pick the largest couple companies in that space and select the most "near to your product" offerings they have in the market. In this exercise you want to put down price, cost (if you can get it or estimate it) the top 3-5 key features/benefits, and overall revenue generation. Larger markets will have an industry classification you can look up. Don't be afraid to use government data because no one will question it.
Make a list of all the markets, companies, products and the product details. This will create a matrix. Now, place your product in that mix. Where will you fall in price, in cost, in features? What is the size of those markets. This is a market positioning exercise and it will go a long way in examining each market, its size and it's competitive landscape. This should help you pick your number one market to pursue.
Boiling Down New Products
New products that have no viable alternatives raise the task of looking at all possible markets and deciding which markets you will attempt to disrupt (cannibalize). Monster.com disrupted classified ads. Craig's List and LinkedIn are disrupting Monster.com.
Usually new products are disruptive to existing products. In this case I'd look for the most vulnerable markets. These would be markets with weak brand presence, many small players, large players with slow to no growth and slow growth industries. Why pick these "dogs" of a market? Because the companies that sell products in that space will struggle to fight disruptive products that come in. Markets that have powerhouse brands, huge companies and consolidated locked markets usually have huge switching barriers in place that hold their customers bound to them. You can take on these markets but it's going to require more time and money than the alternatives I've suggested.
A matrix of markets as listed above can be helpful but new products often create new markets. In this case you're inventing a market space. Gathering lots of information on alternative companies and products may not yield decision making information. A new product creating a new market often will mean every dollar of revenue you generate will be dollar for dollar market growth. You will define and build that market. It may be exciting but it can be quite intimidating as well. Trail blazers face many challenges (article here).
What to Focus On
The largest market is not necessarily the best market. If you're just starting out it doesn't really matter if the overall market segment is $10 billion or $100 million, either is plenty of market for you to start your efforts in.
Things that help in decision making is overall competition (you want weak competitors), sales complexity (you want fast sales cycles), ease of feedback (fail fast, learn quickly), marketing cost (lower the cost the faster your entry). The assumption here is you will initially have low success outcomes but huge learning outcomes when you first go to market. You want to fail fast and learn quickly. Large markets that have strong competitors and complex sales cycles often pose a large risk of spending a lot of money to learn with a long timeline. Avoid this. Often the smaller market segments provide the best initial opportunities for entrepreneurs.
Take the ocean of opportunity and boil it down to a swimming pool. There is still plenty of water to drown in. Best of luck in your endeavors.