AI Issue 3 (2025-3) open access

My friend and I use the same prompt, how come the results are different?!

#outputvariation_alsomatters

5 min read
My friend and I use the same prompt, how come the results are different?!

It can be confusing or irritating, but it is normal. :)
It’s our expectations for "identical inputs must lead to identical outputs" which make the situation look awkward.

Remember the last time you asked somebody for directions on the street. Assuming those were correct, what are the chances that another passerby you’d stop would repeat the instructions word-by-word or at least choose the same way to direct you? And if you start thinking that it should depend on the question (in this case, on how far the points A and B are from each other and how twisted the road is), you are absolutely right!

To demonstrate what the process is like for LLMs, we conducted an experiment with Gemini and two questions, one requiring factual information and another one introducing a hypothetical scenario.