Show HN: Value likelihoods for OpenAI structured output
(arena-ai.github.io)
5 points by ngrislain 3 hours ago
Thank you! The number is the the sum of the logprobs from the token constituting the individual values. So it does represent the likelihood of seeing this value.
So yes OpenAI is super-biased as a random number generator.
We sampled other values from OpenAI and got other die roll values, but with much lower probs (5 has 8% chances ).
by ngrislain 2 hours ago
More precisely it represents the likelihood of seeing this value conditional on the tokens before it.
by ngrislain 2 hours ago
That said, it's concerning to see the reported probability for getting a 4 on a die roll is 65%.
Hopefully OpenAI isn't that biased at generating die rolls, so is that number actually giving us information about the accuracy of the probability assessments? by yodon 2 hours ago