How good is ChatGPT for competitive intelligence?

ChatGPT is all the rage at the moment. It can pass bar exams and medical exams and seemingly answer all sorts of questions in a way that has some people very excited about ChatGPT being a fundamental shift in computer intelligence.

Our understanding of the model behind ChatGPT is limited but it seems at first glance like it might excel at the fact-finding part of competitive intelligence (leaving aside the question of whether it can help with insights and decisions). So we asked ChatGPT the kind of questions that we get asked all the time (in fact, five questions we have been asked by real clients in the past few months, shown below except that they are cleaned for obvious client confidentiality reasons). ChatGPT was able to answer one of the five questions we asked it, superficially but acceptably. It failed the other four (which were answered by analysts at the time of asking over the past few months). And on a point of context, the data on which ChatGPT relies is about 18 months old, which may make it out of date even where ChatGPT can construct an answer.

Test question 1: Who leads the litigation team at [xxx]?

ChatGPT’s answer: As an AI language model, I do not have access to up-to-date information on the specific individuals who lead the litigation team at [xxx], as this information may change over time. However, you can visit the [xxx] website or contact their customer service for more information on their current leadership team and organizational structure.

Our comments: This is answerable via LinkedIn. At worst, ChatGPT might have been able to return [xxx]’s General Counsel.

Test question 2: What are the tiers of the [xxx] partner program?

ChatGPT’s answer: The [xxx] partner program has four tiers, each with its own set of benefits and requirements.

• Registered Partner: This is the entry-level tier for partners who are just starting with [xxx]. Partners at this level get access to training, sales and marketing resources, and a dedicated partner account manager.

• Gold Partner: To qualify for the Gold Partner tier, partners must meet specific revenue and sales targets. Gold Partners receive additional benefits such as co-marketing funds, joint sales opportunities, and a higher level of support from [xxx].

• Platinum Partner: The Platinum Partner tier is for partners who have demonstrated a high level of sales success and commitment to [xxx]. Platinum Partners receive the highest level of support from [xxx], as well as access to exclusive resources and opportunities.

• Premier Partner: The Premier Partner tier is the highest level of partnership with [xxx]. To qualify for this tier, partners must meet strict revenue and performance criteria, as well as demonstrate a deep understanding of [xxx]'s products and services. Premier Partners receive the most comprehensive set of benefits, including dedicated support from a [xxx] Partner Success Manager, customized training and enablement, and access to advanced sales and marketing resources.

Our comments: This is a good start. A client would expect more detail and depth, but we’ll count this as a question that ChatGPT was able to answer.

Test question 3: What is the pricing for [xxx]’s software?

ChatGPT’s answer: To get an accurate pricing quote for [xxx]'s solutions, you can visit their website and fill out a contact form to request a consultation with one of their sales representatives. They will be able to discuss your specific needs and provide you with a pricing plan that is customized to your business.

Our comments: While there are clues about this company’s pricing online, to answer this properly you’d need to make some calls. Possibly unfair to expect ChatGPT to be able to answer it then, but it’s just confirmation that ChatGPT is not the Oracle of Delphi - it relies on public, online data.

Test question 4: How many [xxx] customers pay more than $100M annually?

ChatGPT’s answer: Unfortunately, as an AI language model, I do not have access to [xxx]'s customer data or financial information. [xxx] does not disclose the number of customers who pay more than $100 million annually, so I cannot provide a specific answer to your question. However, it is known that [xxx] is a popular choice for large enterprises and government organizations, and many of these organizations may spend hundreds of millions of dollars on [xxx] services each year.

Our comments: There is data online that can be used to estimate an answer to this, and some of these customers are named publicly. A human analyst would at least build a model to come up with a sensible estimate. At the very least, if you know [xxx]’s revenues are $XB (which ChatGPT knows because it knows the company is listed), you’d know that at most [xxx] has $XB/$100M such customers. A good human answer would require calling around. On a tangent, the last sentence is impressive - not in itself but for the inference that the $100M is enterprise-level spend … it would be interesting to understand exactly how ChatGPT does that.

Test question 5: What are [xxx]'s revenues?

ChatGPT’s answer: As an AI language model, I don't have access to real-time financial information. However, according to [xxx]'s website, they are a privately held company and do not disclose their revenue publicly.

Our comments: If the revenues were disclosed publicly, we’d just have asked Google. One can estimate this company’s revenues from the number of employees, from the revenues of their peers, from knowing their pricing (which is partly public) and user numbers (likewise). But again, the best data would require making some calls.

Overall, ChatGPT is a long way away from being useful for competitor analysis. It’s not even as good as Google - ChatGPT speaks more eloquently than Google but with Google, we’d have answers to Q1, Q2 and the beginnings of answers to Q4 and Q5. Google may require a few more clicks and digging, but ends up being more useful.

Need a quote for competitor research?