LLM content discoverability experiment

Are you feeling pressure to improve your content discoverability with large language models (LLMs) like ChatGPT or Google’s AI summary, but are struggling to find a place to start?

I have a fun experiment to kick you off.

Put simply, the idea is to run ten of your most typical content questions through your choice of LLMs and observe the responses with the objective of making improvements and hyping up your team.

What are LLMs like ChatGPT and Claude saying about your organisation or brand?

Process

To start, set up your spreadsheet. Include enough variables to be useful, but not so many that it feels like an out-of-control chore. Try to find the Goldilocks zone. A good starting place is the following headings. What would you add or replace?

Spreadsheet headings

• Content question

• LLM tool

• Answered correctly? Yes/No

• Answered correctly? Quality

• Were we referenced? Eg is there a link to our site?

• Mitigation required? Yes/No

• Mitigation detail - what specifically will we change to get a better result?

• Date run - so you can compare results over time.

Example questions

Then you want to collect your content questions. Imagine the top phrases that your users are going to type into an LLM or Google.

Some examples include:

• What kind of insurance do I need for my small business?

• How do I register a beacon in Australia?

• How do I get an extension on my car rego costs in NSW?

• How do I return my Uniqlo order?

• How much money do I need to retire?

• How do I transfer my internet when I move?

• What is the phone number for the ATO?

• What is the best note-taking app for my iPad?

• What are the best ways to see the new years fireworks in Sydney?

• How can I get help looking after my sick mum?

• Where can I find the fines for buying black market tobacco?

• Are the new fish markets worth visiting?

• What Australian bank has the best mortgage rate?

• What channel can I watch the AO on?

• Where can I find a local tennis coach?

Guidance

• Don't make it too complicated - look for the sweet spot between detailed and actionable

• Structure your spreadsheet so you can review the results over time

• Structure your spreadsheet so you can pivot it

• Decide what the best time interval is - every month?

• Look for opportunities to act

• Look for stories to share with your team and stakeholders

• If you need help, refer to Gerry McGovern's top tasks to help you choose your questions

• Ask the question “Who should be doing this task?” Can it be spread among the team to increase exposure to the tools?

Tool ideas

Select from this list or add another option of your choice 

• Claude

• ChatGPT

• Gemini

• Google AI summary

• Google AI summary - show more

• Copilot

Next steps

Are you satisfied with how LLMs found or summarised your content? If not, your next step is to make some changes.

Start with page titles, headings and in-page summaries. Then run the questions again and see if you have improved the results. Ask yourself the question, how much time does it take for changes to be pushed through?

And finally, share these insights with your broader team. This experiment is a great opportunity to advocate for your business impact.