[blackshoe]

thepossum, on 2021-March-08, 01:51, said:

Lets think. I know the number of hands dealt so far is a tiny fraction of infinity etc

"A tiny fraction of infinity" is a nonsense phrase. That tiny fraction is still infinite.

Besides we're not talking about infinity. There are only 635 billion possible hands.

[thepossum]

blackshoe, on 2021-March-08, 17:10, said:

"A tiny fraction of infinity" is a nonsense phrase. That tiny fraction is still infinite.

Besides we're not talking about infinity. There are only 635 billion possible hands.

The number of hands is finite but the issue of distributions and randomness requires consideration of infinity

I need to add a correction. My memory isnt that good. I think I had the number of possible deals wrong by about 10 orders of magintude. Its only around 5.3 x10^28 deals

But we are looking at the samples of hands, sequences, many players potentially over an indefinite period of time and much of the theory that is used also requires infinity

I still don't think the problem has been specified well enough. Are we allowing for other variables and differences in individual perception of a pattern or lack of randomness too

But most of do usually only see a tiny fraction of infinity do we not

Deleted some nonsense statistical calculations I think I totally mis-estimated the degrees of freedom. My brain is addled at the moment

[jardaholy]

IowaST8, on 2021-February-16, 05:59, said:

OK, Are the Bridge Hands on Bridgebase truly random? Or, are the hands sometimes padded to make the game more interesting?

Well, this is THE question!
I got PhD in probabilistic risk analysis and I earn my money in this area for almost 35 years.
I started playing daylongs at the begining of Covid-19 pandemia and played approximately 30 boards a day, i.e. more than 11000 boards by now so that the data sample is big enough to make some conclusions.
I absolutely do not believe that the boards are truly random.
During last several months, I met the hands with 6-6 distributions seven or eight times, that is far more than it should have been (just one example, other sick distributions as 7-5, 8-4 etc. are also much more frequent that they should have been).
Still, this is not about distribution.
In some discussion here, I could read very nice idea that the variance of the results in the boards generated in daylongs is what is much bigger that it should have been. Just because the boards are not flat, at least many of them.
That is even possible to do - I can imagine that the software for dealing boards for daylongs simply use, for the pool of boards, boards, which already were played sometimes and produced results with high variability (not allowing you to play board, you already played).
Of course, boards with high variance of results must be, in general, more demanding and more interesting than flat boards, where the variance of results is very small (everybody can manage such board and get average result).
The main problem I can see, if it is really the case, is that the boards with high variance of results are biased in some specific ways. For example, you can get high variance of results, if you offer the players a board with significantly sublimited game or slam (which also need good card play) You will not get such big variance in results if such game is impossible to be made. But I do not like that, because it has impact on my bidding and play and I am affraid, I will more frequently overbid the board later, in my bridge club, as soon as it is opened after Covid, just because of bad habit I got here.
If I have not been absolutely sure about what I am writing here right now, I am even more persuaded after playing one very specific board recently (I do not want to present it right now, because it could be still in play at BBO). This was almost a kind of once-per-life board, you should not meet once per month as here. But this one was even more special. The slam (bid by the robot on the opposite side) looked absolutely impossible to be made, but after several minutes, I found the way, it could be made. It needed just a small thing - the distribution 6-6 in the hand of one of the defenders. And it really was that. I am grateful to BBO for having this miracle experience (and a nice story for my bridge friends), but I am still not sure, whether this way of generation of the boards is good.

[hrothgar]

jardaholy, on 2021-March-10, 17:19, said:

Well, this is THE question!
I got PhD in probabilistic risk analysis and I earn my money in this area for almost 35 years.
I started playing daylongs at the begining of Covid-19 pandemia and played approximately 30 boards a day, i.e. more than 11000 boards by now so that the data sample is big enough to make some conclusions.
I absolutely do not believe that the boards are truly random.

All tournaments are not created equal

BBO give tournament organizers the ability to screw with the hands.
And, lots of individuals running tournaments choose to do so.

If you truly are getting way too many 6-6 pattern and the like, it probably reflects the tournaments that you are playing in rather than the hand generators itself.

In order to evaluate the whether or not BBO's hand generator is biased in some manner, you'd want to look at a limited set of tournaments (for example, non best hand tournaments run by the ACBL)

[pilowsky]

jardaholy, on 2021-March-10, 17:19, said:

Well, this is THE question!
...

What are the odds that sitting South you would pick up this hand AND be playing in a major tournament AND be the worlds best Bridge player?

---

And the lead was a ♠.
one in 2,722,762 apparently.

If the current pandemic has (re)taught us anything, it's that incredible 1/100 year events only happen once in a hundred years, but there are lots of them.
COVID-19, Texas freezing, NSW and California burning, A non-white woman being elected to be Vice-President.
The hand may be over, but the war goes on. http://bit.ly/WarGoesOn.

It turns out that the odds of being dealt a hand with 9-x-x-x or better are better than 1 in 1700. Last month I played only 750+ hands.
One dealing site has now dealt more than 57,527,618,400 hands

26,358,553 of them (0.045874812%) held 25 or more HCP!

I've said it before, common things do occur commonly, but uncommon things do occur.

Spoiler

http://bit.ly/Zia7D

---

[smerriman]

jardaholy, on 2021-March-10, 17:19, said:

I started playing daylongs at the begining of Covid-19 pandemia and played approximately 30 boards a day, i.e. more than 11000 boards by now so that the data sample is big enough to make some conclusions.
I absolutely do not believe that the boards are truly random.
During last several months, I met the hands with 6-6 distributions seven or eight times, that is far more than it should have been (just one example, other sick distributions as 7-5, 8-4 etc. are also much more frequent that they should have been).

I know you said your post is not about distribution, but I've done the same analysis numerous times over tens of thousands of daylong hands. And the results for shape are exactly what you would expect from random hand generation. Incidentally, if you had a sample since the start of the pandemic, why did you restrict your conclusion to 'the last several months'?

As for your "conspiracy theory" about BBO adding more variance to the hands, to be honest, I don't think they'd know how if they tried.

But if you wanted to quantify any numbers behind how you think they're doing so or what you think shouldn't be happening, I can attempt to refute them, like I have every other similar claim.

[pilowsky]

smerriman, on 2021-March-10, 18:35, said:

I can attempt to refute them, like I have every other similar claim.

It's true, he will - it seems to be a full-time job.

[gszes]

Instead of "defending" the results over time of your method of dealing, why not simply reveal the method. For EX: my method is simply to create a randomized 52 card deck then deal :from the top: NESW. Just so happens this coincides with the recommended dealing method for FTF play. There can always be slight wrinkles, for EX: always give the first card to the dealer and then deal clockwise for the remainder of the hand. Stop using boards that have previously been played. They are great for teaching but totally unnecessary in these days of faster than light computers (ok so technically maybe a tad slower than that). IF a tournament is going to be NOT random, the exact reason for the lack of randomness must be revealed prior to a person committing to play in it.

[mycroft]

And how do you "create a randomized 52-card deck"? That's the actual problem. How you distribute said {0..51} sequence among the hands is irrelevant if the sequence is truly random, all possible options are available with the same frequency, and given previous information, it is impossible to predict the next. And really, the only way to prove that is to try it, to calculate percentages and sequences, and so on. "Defend it", in other words.

Please note, it is a very hard problem, one that computer people have been getting better at for 30+ years, and finding holes in "perfect solutions" for the same length of time.

Having said that, with hardware TRNGs available for <$100 (now, debiasing them and keeping them debiased is an art and takes much expert time), the only acceptable solution should be "generate 96 bits of randomness, and look that number up in The Big Book (or throw it out if you go over)", repeat ad nauseam. That's basically BigDeal, with a TRNG in place of the entropy pool (which frankly is good enough for practical purposes, even Bermuda Bowl-level practical purposes).

But for BBO casual games, anything that isn't clearly biased or predictable is Good Enough, and almost certainly better than handshuffling.

[hrothgar]

mycroft, on 2021-March-17, 13:42, said:

Having said that, with hardware TRNGs available for <$100 (now, debiasing them and keeping them debiased is an art and takes much expert time), the only acceptable solution should be "generate 96 bits of randomness, and look that number up in The Big Book (or throw it out if you go over)", repeat ad nauseam. That's basically BigDeal, with a TRNG in place of the entropy pool (which frankly is good enough for practical purposes, even Bermuda Bowl-level practical purposes).

I am of the opinion that the PRNG is superior to a TRNG for generating bridge hands.

1. The benefits of using a TRNG versus a PRNG simply aren't salient. So long as you're using a good PRNG, you're fine

2. There are enormous benefits to being able to demonstrate that the seeding and the hand generation were done in a fair / unbiased manner. Hans van Stavern did some nice work creating a protocol by which folks who are running tournaments with Big Deal can prove that they aren't biasing the deals in some weird way. I don't think that you can do the same with a TRNG.

There may be hardware based RNG's that are capable of attestation.
(I'm not aware of any)

[mycroft]

Definitely a point.

The opposite point is that whatever allows repeatability or attestation is now the key to that hand set. That is potentially easier to pass off or acquire than the hand records.

My concern about TRNG is "don't roll your own crypto". There are people who spend their entire working lives making randomness sufficiently secure for cryptographic purposes, for things that are much more data-intensive and crack-lucrative than the Bermuda bowl. Using TRNG, you have to do all the debiasing (and run, or verify, the tests of the maker). The chance you will do that wrong (or me) is not insignificant. The chance that a tool-user who doesn't have a crypto background will do it wrong is much higher. And it doesn't take much. The chance that all those people making randomness safe have missed something available to [sponsor to be unnamed here], if they're not secretly working for the NSA or Mossad (and are willing to cheat to win the BB using tools that are top-of-the-top class government secrets) are very small.

Between those two, you are probably right. Still doesn't matter what the "dealing" method is.

[gszes]

mycroft, on 2021-March-17, 13:42, said:

And how do you "create a randomized 52-card deck"? That's the actual problem. How you distribute said {0..51} sequence among the hands is irrelevant if the sequence is truly random, all possible options are available with the same frequency, and given previous information, it is impossible to predict the next. And really, the only way to prove that is to try it, to calculate percentages and sequences, and so on. "Defend it", in other words.

Please note, it is a very hard problem, one that computer people have been getting better at for 30+ years, and finding holes in "perfect solutions" for the same length of time.

Having said that, with hardware TRNGs available for <$100 (now, debiasing them and keeping them debiased is an art and takes much expert time), the only acceptable solution should be "generate 96 bits of randomness, and look that number up in The Big Book (or throw it out if you go over)", repeat ad nauseam. That's basically BigDeal, with a TRNG in place of the entropy pool (which frankly is good enough for practical purposes, even Bermuda Bowl-level practical purposes).

But for BBO casual games, anything that isn't clearly biased or predictable is Good Enough, and almost certainly better than handshuffling.

My bad,,, simplest (not most efficient but at these speeds who cares) is the lottery method.
spades 1 thru 13 hearts 14 thru 26 dia 27 thru 39 club 40 thru 52
Pick a number btn 1-52 or 0-51 if you cannot stomach leaving 0 out of your programming.
that is card 1 now pick another 1 - 52 and compare it to the one(s) already picked and keep doing that until a unused number is picked and keep that up until 51 of the 52 possible are picked then assign the last card (checking the deck starting from 1 up to 52 and if a number is missing that is our last card.

This is "dating" me but the randomizer in any decent software is based on using the internal clock so the results will vary depending on the processing speed of each individual computer. This factor alone would make predicting the next deal nearly impossible (I suppose a quantum computer might someday accomplish this task).

[smerriman]

gszes, on 2021-March-18, 21:10, said:

My bad,,, simplest (not most efficient but at these speeds who cares) is the lottery method.
spades 1 thru 13 hearts 14 thru 26 dia 27 thru 39 club 40 thru 52
Pick a number btn 1-52 or 0-51 if you cannot stomach leaving 0 out of your programming.
that is card 1 now pick another 1 - 52 and compare it to the one(s) already picked and keep doing that until a unused number is picked and keep that up until 51 of the 52 possible are picked then assign the last card (checking the deck starting from 1 up to 52 and if a number is missing that is our last card.

This is "dating" me but the randomizer in any decent software is based on using the internal clock so the results will vary depending on the processing speed of each individual computer. This factor alone would make predicting the next deal nearly impossible (I suppose a quantum computer might someday accomplish this task).

I think you're still missing the whole point here - the *only* part of interest / relevance when it comes to bridge deals is how you 'pick a random number between 1 and 52' in a way that not only satisfies random distributions but is also not predictable based on previous random numbers.

A simple pseudo random number generator seeded with the internal clock is not even remotely close to good enough, and would be extremely easy to crack, and also wouldn't be sufficiently random, given the limited number of values the clock may take*

About 5 years ago the ACBL random number generator was cracked; given three consecutive boards, it was possible to determine every layout from then on.

You might like to read this article on how BigDeal works, and everything it had to consider. Especially interesting is how they were able to generate a seed from human keystrokes.

* Well, it may be good enough for BBO, given the immense number of things going on in parallel on the server, which nobody could really keep track of. But you'd still need to use a good PRNG that doesn't repeat as often as most basic ones would.

[hrothgar]

In the case of the ACBL hand generator, they were using a linear congruential generator which made it particularly easy to crack

The hard part about this was actually mapping from the random number to the deal.

Please note: The ACBL implementation only covered a very small fraction of the deal space.
You could, in theory, create a rainbow table enumerating the complete system.

[mythdoc]

For understandable reasons, I suppose, this discussion always is diverted back to a debate about generation of deals as opposed to delivery of deals to tables in events. It’s more familiar ground I guess, and the conversation is never going to go anywhere because those methods are still secret.

I, for one, do not suggest that the generation of deals is flawed in any meaningful way. I do not believe that hands coming from the generator are flawed distributionally or in some other measurable way.

I have raised, and continue to raise, a suspicion about the delivery of hands from the generator to the so-called “deal pool,” and from there to the specific events which use these hands. I had these suspicions for a while, and they only intensified when my friends and I, who play lots and lots of robot challenges and have tried every variant of challenges possible, played a series of “non-best-hand” “just declare” MP’s challenges followed by a series of “non-best-hand” “just declare” IMP’s challenges. The hands were immediately, obviously different from the one scoring method to the other. This raised in our minds the strong possibility that hands were delivered from the deal pool based on some kind of “measured variance” (as Jardaholy wrote above) in respective scoring methods from play at other tables before delivery to our table. Playing “just declare” hands was eliminating one huge set of variables — human bidding — enabling us to see a clear difference in the subset of hands that was being chosen for our challenges.

Once you permit the supposition that the delivery of hands may be subject to an algorithm, which computers everywhere are doing in all sorts of ways, then you encounter all sorts of important issues. Once the hands are in the deal pool, they go from table to table, provided that no one sees them twice. They can be analyzed in all sorts of ways, including scoring variance. They can be delivered to specific events (and even, though I don’t suggest this myself) specific players. It’s a lot to process, and since there is no transparency about the deal pool, we know nothing.

(I once found a reference on a BBO ad page that stated that “deal pool worked a little like the college SAT’s.” The link is in an earlier post of mine. Like the SAT’s ???? Sure would like to know what whoever said that meant)

As I’ve said twice previously, this exact same test (two sets of robot challenges described above) can be done by anyone who is interested in really investigating what I am talking about, as opposed to diverting the conversation back to the deal generator blah blah. So far, no one has mentioned they even tried it. That is disappointing but also predictable. Confirmation bias and unwillingness to explore beyond one’s comfortable beliefs can cut both ways.

[hrothgar]

mythdoc, on 2021-March-19, 09:37, said:

my friends and I, who play lots and lots of robot challenges and have tried every variant of challenges possible, played a series of “non-best-hand” “just declare” MP’s challenges followed by a series of “non-best-hand” “just declare” IMP’s challenges. The hands were immediately, obviously different from the one scoring method to the other.

Please provide a precise and testable hypothesis describing how the two sets of hands differ

Is one set of hands stronger than the other?
Are the distributions skewed in one set as opposed to the other?

BE specific

Once you do so, it's easy enough to test this with a set of deals that are yet to be dealt

[pilowsky]

Or, to put it another way, the drunk man searches for his lost keys under the lamp-post because it's too dark to see anywhere else?

[hrothgar]

mythdoc, on 2021-March-19, 09:37, said:

For understandable reasons, I suppose, this discussion always is diverted back to a debate about generation of deals as opposed to delivery of deals to tables in events. It’s more familiar ground I guess, and the conversation is never going to go anywhere because those methods are still secret.

The reason that people don't talk about the deal pools is that there is no reason to do so.

The deal pool system doesn't need to care about whether hands are biased with respect to strength or the distribution of various shapes or any of this sort of stuff. Presuming that the hand generators aren't borked, you can simply take a stream of inputs and slice and dice them into different pools.

Indeed, trying to add logic to look at hand strength (and tweak the allocation of hands accordingly) adds code, adds complexity, and increases the chances that something will get screwed up.

I trust BBO not to have screwed things up in this mind bogglingly stupid a manner.

[mycroft]

smerriman has it right - you're missing the only relevant piece of information. "how do you pick card 1? 2? 3?" And yes, that is in fact one of the most critical, most studied, and most difficult parts of computer science.

Anyone who doesn't realize that right away is almost certainly failing at it. I have post-graduate research in cryptography and random number generation, and I would almost certainly fail at it. And were I to succeed, people like Hrothgar, who have different priorities from me, will disagree with my methods. And he isn't wrong - his priorities are at least as valid as mine, in practise. We're just concerned about different parts of the distribution chain being attacked.

My crypto professor, basically day 1, made it clear that almost never is the actual crypto the source of failure - it's errors in process, or deliberate sabotage, or systems causing incentives to skip steps that seem unimportant. In this case, not paying attention to your source of randomness is an error of process - blindingly obvious to anyone who has tripped over it before, but frankly something that isn't even a thought to people who haven't yet.

So, were I to write something like this (which I wouldn't, because you Don't Roll Your Own Crypto), to keep Hrothgar happy, I would:

seed = secrets.randbits(SEED_LENGTH)
random.seed(seed)
for i in range(1, hands_in_set):
    while not hand[i]:
        hand_no = random.getrandbits(96) # I guess it's possible to use randrange() here and skip the "if".  Assuming randrange is safe (see below)
        if hand_no < POSSIBLE_HANDS:
            hand[i] = BigBook.hand(hand_no)
BigBook.create_dup_file(hand)
BigBook.print_hand_records(seed, hand)

But watch, it's easy to make mistakes here, too:

Quote

Changed in version 3.2: randrange() is more sophisticated about producing equally distributed values. Formerly it used a style like int(random()*n) which could produce slightly uneven distributions.

getrandbits uses randrange if the numbers are big enough, so it's sort of a self-fulfilling loop. If it would call randrange at 96, we'd probably have to build it as multiple internal integers smashed together.

To be potentially safer, I could random.getstate() after a secrets seeding, and store that pickled with the hand record uniqueid. All that would leave the computer was the uniqueid, insufficient to seed another implementation. You could call the program with a uniqueid, and if that had been generated on this machine, it would setstate() and rerun the generation, proving that the hand record hadn't been tampered with.

To be potentially safer, I could... That's why you don't roll your own crypto.

[mythdoc]

hrothgar, on 2021-March-19, 09:50, said:

Is one set of hands stronger than the other?
Are the distributions skewed in one set as opposed to the other

Neither of these. These questions (indeed your comment as a whole asking me to state an hypothesis) show your own assumption about “measurables.”

They reflect the factors that creat scoring variance according to the two scoring types. This has much less to do with distributions or strength as it has to do with the following:

MP’s capturing an over trick or conceding another undertrick; competing correctly on part score battles
IMP’s game or slam hands, or hands where a swing is formed by being set in a part score (particularly doubled) vs. making a part score. Overtricks not particularly important at all.

As I have said, you will see the difference most obviously in just declare hands where the bidding has been done by the computer as well.