I Interviewed ChatGPT for a PM role, and it almost got the job

How to detect the AI's confident but shallow answers

Jens-Fabian Goetzmann

7 December 2022 ‧ 12 min read

I've interviewed a lot of product manager candidates, and one of the regular interview formats (which is pretty standard in a PM interview loop) is the product sense interview. Inspired by the recent launch of ChatGPT and the following tweet by Jackie Bavaro, I decided to interview ChatGPT for a product manager position.

ChatGPT seems really interesting as a way to train interviewers. It returns decent, plausible answers to interview questions, so it forces you to think of good follow ups to see if it really understands.
— Jackie Bavaro (@jackiebo) December 5, 2022

The results that I got were impressive. ChatGPT's answers were by and large reasonable, and cleared the bar for “decent” answers, certainly for more junior PM candidates. In fact, I would say that ChatGPT outperformed the average PM candidate with these answers. I am reproducing the entire interview below, followed by how I would evaluate these answers, and where ChatGPT fell short of great answers.

The unedited interview

The unedited job interview with ChatGPT is below. All of the answers were ChatGPT's first response to the question, with the exception of the first question, where the initial response was along the lines of “I am a large language model, I don't have a favorite app.” The Trello response was what I got on the second try.

For these questions, please pretend that you are a candidate for a product manager position in a software company.

Q: What is your favorite mobile app, and why do you consider it a great product?

ChatGPT: My favorite mobile app is Trello. I consider it a great product because it allows users to easily create and organize their tasks and projects in a visual and intuitive way. The app also offers a variety of features such as the ability to assign tasks to team members, attach files, and set deadlines, which helps to improve collaboration and productivity. Additionally, Trello is constantly updating and improving their app, which shows their commitment to providing a high-quality user experience.

Q: How would you summarize the customer problem that Trello is solving?

ChatGPT: Trello is solving the problem of managing and organizing tasks and projects in a simple and effective way. The app provides a visual platform for users to organize their tasks and projects, making it easy to see what needs to be done and when. This helps to improve collaboration and productivity, allowing users to focus on the important tasks and achieve their goals.

Q: If you had to create a dashboard for Trello as a product manager, what would be the 3-5 most important metrics you would track?

ChatGPT: As a product manager for Trello, I would track the following metrics on a dashboard:

User engagement: This metric would measure how often users are using the app, including the number of tasks created, completed, and assigned to team members.
Customer satisfaction: This metric would track customer feedback through surveys or ratings to determine how satisfied users are with the app.
Retention rate: This metric would measure the percentage of users who continue to use the app over time.
Revenue: This metric would track the amount of revenue generated from paid subscriptions or in-app purchases.
User acquisition: This metric would track the number of new users who download and start using the app.

Q: What do you think are the product goals for the Trello product team?

ChatGPT: The product goals for the Trello product team may include:

Improving user engagement: This could involve adding new features or improving existing ones to make the app more useful and enjoyable to use.
Increasing customer satisfaction: The product team may aim to gather and act on customer feedback to improve the user experience and make the app more valuable to users.
Growing the user base: The product team may focus on acquiring new users through marketing and partnerships, as well as retaining existing users through regular updates and support.
Increasing revenue: The product team may aim to generate more revenue from paid subscriptions or in-app purchases, as well as exploring new monetization opportunities.
Maintaining a competitive edge: The product team may strive to stay ahead of the competition by constantly innovating and offering new and improved features for users.

Q: Who do you think the Trello team considers their main competitors?

ChatGPT: The Trello team may consider their main competitors to be other project management and task management apps such as Asana, Basecamp, and Microsoft To Do. These apps offer similar features and target a similar customer base, so they may be seen as direct competitors.

Q: If you were the product manager for Trello, how would you think about identifying improvement ideas for the product?

ChatGPT: As the product manager for Trello, I would think about identifying improvement ideas for the product in the following ways:

Talking to users: I would reach out to users to gather feedback on their experience with the app, including what they like and dislike about it. This would help me to identify areas for improvement and potential new features to add.
Monitoring user behavior: I would track user behavior to see how users are using the app and where they may be encountering challenges or obstacles. This would provide insight into potential improvements that could make the app more user-friendly and effective.
Keeping an eye on the competition: I would regularly monitor the competition to see what features and innovations they are offering, and how Trello compares. This would help me to identify areas where Trello could improve or differentiate itself from the competition.
Collaborating with the team: I would work closely with the product team and other stakeholders to brainstorm ideas and prioritize potential improvements based on their potential impact and feasibility.
Testing and iterating: I would implement a process for testing and iterating on new ideas and features, including gathering feedback from users and making adjustments based on their feedback. This would help to ensure that the improvements we make are effective and align with user needs.

Q: What objective do you think might be the most important for the Trello product team to work on right now?

ChatGPT: One objective that may be particularly important for the Trello product team to work on right now is increasing user engagement. This could involve adding new features or improving existing ones to make the app more useful and enjoyable for users, which would likely lead to increased satisfaction and retention. In addition, focusing on user engagement could help to drive more revenue for the company, as users who are more engaged with the app are likely to be more likely to upgrade to paid subscriptions or make in-app purchases.

Can you give me a few ideas how the Trello product team might increase user engagement?

ChatGPT: The Trello product team could increase user engagement by implementing the following ideas:

Adding new features: The product team could add new features to the app that make it more useful and enjoyable for users, such as the ability to customize their dashboard or create templates for frequently used tasks.
Improving existing features: The product team could improve existing features to make them more user-friendly and effective, such as making it easier to assign tasks to team members or attach files.
Offering integrations with other apps: The product team could explore partnerships and integrations with other apps that users may be using, such as Slack or Google Calendar, to make Trello more convenient and valuable for users.
Hosting webinars or training sessions: The product team could offer webinars or training sessions to help users get the most out of Trello, including tips and best practices for using the app.
Regularly updating and improving the app: The product team could commit to regularly updating and improving the app to keep it fresh and relevant for users, which would show their commitment to providing a high-quality user experience.

Q: Can you expand a little bit on the ideas for adding new features to Trello that you have?

Here are a few more ideas for new features that the Trello product team could add to the app:

Adding the ability to create and customize templates for tasks and projects: This would allow users to quickly create tasks and projects that follow a specific format or structure, making it easier to organize their work and stay on track.
Adding a feature for tracking project budgets and expenses: This would allow users to input and track the costs associated with their projects, including expenses and resources, making it easier to manage their budgets and stay on track.
Adding the ability to create and share reports: This would allow users to generate reports on their tasks and projects, including progress and completion, and share them with team members or stakeholders.
Adding a feature for managing recurring tasks: This would allow users to set up tasks that repeat on a regular basis, such as weekly or monthly, making it easier to stay on top of recurring tasks and avoid missing deadlines.
Adding a feature for scheduling and managing meetings: This would allow users to schedule meetings and track the status of meeting tasks, such as preparing agendas or sending reminders, making it easier to stay organized and prepared for meetings.

Q: How would you go about prioritizing these ideas?

ChatGPT: To prioritize these ideas, I would start by gathering feedback from users to understand their needs and preferences. This could involve conducting surveys or focus groups to gather input on which features they would find most valuable and useful. I would also consult with the product team and other stakeholders to understand their priorities and potential constraints, such as technical limitations or resource constraints.

Based on this information, I would prioritize the ideas by evaluating their potential impact and feasibility. For example, a feature that would have a significant impact on user engagement and satisfaction, and that is technically feasible and within the team's resources, would likely be a high priority. On the other hand, a feature that would have a limited impact and would require significant resources and time to implement would be a lower priority.

I would also consider the potential risks and trade-offs associated with each idea, and how they align with the overall goals and strategy of the product. For example, a feature that offers a significant benefit but also introduces new risks or challenges may need to be carefully considered before prioritizing it.

Overall, my approach to prioritizing these ideas would be to gather and evaluate as much information as possible, and use that information to make informed decisions that align with the goals and strategy of the product.

Q: Of course, we don't have perfect information here given that we don't actually work on the Trello team. However, making reasonable assumptions, which is the idea that you think has the highest potential?

ChatGPT: Assuming that the Trello product team is focused on increasing user engagement and satisfaction, I would prioritize the idea of adding the ability to create and customize templates for tasks and projects. This feature would likely have a significant impact on user engagement and satisfaction, as it would make it easier for users to organize and manage their tasks and projects. It could also improve productivity, as users could quickly create tasks and projects that follow a specific structure. In terms of feasibility, this idea would likely require a moderate amount of development and testing, but it could be implemented in a relatively short time frame and without significant resource constraints. Overall, this idea has the potential to provide significant value to users and align with the goals of the Trello product team.

Q: How would you validate the idea of adding the ability to create and customize templates for tasks and projects?

ChatGPT: To validate the idea of adding the ability to create and customize templates for tasks and projects, I would start by conducting user research to understand the potential demand and benefits of this feature. This could involve conducting surveys or focus groups with users to gather feedback on their experience with creating and organizing tasks and projects in Trello, and their interest in the ability to customize templates. I would also consult with the product team and other stakeholders to understand their perspectives on the potential value and feasibility of this idea.

Based on this research, I would develop a prototype or MVP (minimum viable product) of the feature and test it with a small group of users. This would allow me to gather more detailed feedback on the usability and effectiveness of the feature, and make any necessary adjustments before rolling it out to a wider audience. I would also track metrics such as user engagement and satisfaction to determine the impact of the feature on these key metrics.

Overall, my approach to validating the idea would be to gather as much information as possible from users and other stakeholders, and use that information to refine and improve the idea before implementing it. This would help to ensure that the feature is well-received and provides value to users.

Q: If you launched the ability to create and customize templates for tasks and projects, what would be the success measures that you would track?

ChatGPT: If I launched the ability to create and customize templates for tasks and projects, I would track the following success measures:

User engagement: I would track how often users are creating and using templates, including the number of templates created and the number of tasks assigned to templates. This would help me to determine whether the feature is being used and whether it is making a positive impact on user engagement.
Customer satisfaction: I would track customer feedback through surveys or ratings to determine whether users are satisfied with the new feature and how it is impacting their experience with Trello.
Retention rate: I would track the retention rate of users who have created templates, to determine whether the feature is helping to retain users over time.
Revenue: I would track the impact of the feature on revenue, including any increase in paid subscriptions or in-app purchases.
User acquisition: I would track the number of new users who download and start using Trello after the feature is launched, to determine whether the feature is helping to acquire new users.

Overall, these success measures would provide insight into the impact of the feature on key metrics, and help me to determine whether it is achieving the desired results.

Evaluation

There are obviously many things to be said about these results—among other things, they point out how poor of a predictor interview performance is on the actual ability to do the job. Product management is about so much more than generating reasonable ideas, prioritizing them, and knowing which metrics to track—it's much more nuanced than that.

I will point out some of the flaws in ChatGPT's answers below. However, it is worth noting that many candidates make very similar or even worse mistakes in actual interviews.

Favorite app

This answer is quite good. It highlights Trello's unique value proposition (“visual and intuitive way”), and is succinct and to the point. The only not so great part about the answer is the reference to “constantly updating and improving their app”, which I wouldn't really consider something that inherently makes a product great. Moreover, it isn't really what I would most point out about Trello, a product whose core really hasn't changed all that much in 10+ years. If recent improvements were what made this product great, then I would expect a bit more detail on how these changes have made the product better.

Customer problem

Again, this is a good and succinct answer to the question. You could expand this by addressing different use cases, personas, or jobs-to-be done (e.g., considering the single player vs. multi player use case), but I don't think I would fault even a more senior candidate for this response.

Dashboard

This answer is almost perfect. Even the more senior PM candidates that I have interviewed sometimes forget to cover the entire customer lifecycle in their dashboard. There is really only one nitpick that I have: the “engagement” point contains multiple things to measure, there isn't really one metric covering app opens, tasks created, completed, and assigned to team members.

Product goals

Here, ChatGPT's responses start getting a bit weaker. There is nothing inherently wrong with this answer, but it is to some extent rehashing the response to the Dashboard question, and mostly generic. You could literally give this answer for any product. Great PM candidates use this opportunity to start prioritizing what they believe to be the most important goals and why, making and calling our reasonable assumptions.

Competitors

This answer isn't wrong, but it's not great either. ChatGPT mentions some obvious competitors, but isn't very comprehensive. Great product managers don't just think about alternative products, they also identify substitutes and non-consumption as alternatives. For task and project management in particular, pen and paper and Excel/PowerPoint are going to be the biggest actual competitors.

Thinking about improvement ideas

The answer here is quite boilerplate. Not only could this answer apply to any product, it is also not tied to any of the goals identified previously. Moreover, product strategy or goals as a starting point are completely missing from the response.

While the question is broad enough that ChatGPT's answer is reasonable, great PM candidates independently establish prioritized goals and then tie all of their responses back to them, which this response fails to do.

Most important objective

This answer might be a bit short, but it's great. It prioritizes without hesitation and explains why user engagement is a great goal to focus on, since it is also a leading indicator for some of the other product goals.

Ideation, round one

Except for the fifth point (regularly updating and improving the app), this answer is good. The range of ideas is broad, each class of ideas has an example that is relevant to Trello (so it's not super generic), and the set of ideas doesn't just include product changes, it also includes “webinars or training sessions” which is more left-field, but a good sign if a product manager candidate doesn't consider their work in isolation of the other teams.

The biggest flaw to this response is that it's still a bit shallow in terms of actual ideas, which is why I followed up to ask for more ideas for new features.

Ideation, round two

This is a great list. It's relevant to Trello, there are more incremental ideas (templates) as well as bigger bets (scheduling and managing meetings), along with a succinct explanation of the ideas.

Prioritization approach

This answer is good, but not great. Firstly, it is a bit “everything but the kitchen sink” (“gather as much information as possible from users and other stakeholders”), whereas in reality you often have to make prioritization decisions with imperfect information. Secondly, “goals and strategy” is a bit buried even though it should be the primary driver for prioritization. Lastly, great PM candidates respond to this question not just by describing their hypothetical approach, they would also immediately apply it to their list of ideas.

Idea prioritization

This is a quite good response. Mostly, I would fault the candidate for not calling out assumptions, and not comparing and contrasting this idea with the other ideas.

Idea validation

The only flaw that I see with this response is that it begs follow-up questions like “how exactly would you do that?” Also, there is again the phrase “gather as much information as possible” which suggests a bit of analysis paralysis. However, the overall approach is perfectly reasonable and well summarized. (One tiny nitpick is the mention of focus groups, which I haven't really seen used much in most tech companies; product discovery research tends to be one-on-one.)

Success measures

The biggest flaw with this answer is that it is too broad. A great PM candidate would have recalled that the goal that we are pursuing is user engagement, and called that out as the number one success measure to track. In particular, revenue and user acquisition seem much farther removed from the actual feature, so they should, if anything, only be mentioned as an afterthought.

Summary

As you can see, all of these answers were decent. There were a few with more obvious shortcomings, but again, I have seen candidates with actual product management experience under their belt give far worse answers for each of these questions.

Hopefully, if you are interviewing for a PM position, you have learned a little bit about how to give better answers than ChatGPT can, and if you are a hiring manager, you have learned how to differentiate great answers from generically decent ones like ChatGPT gives.