[hanp]

Cascade, on Apr 28 2009, 03:21 PM, said:

Although we are happy to decide major championships by many fewer boards and some conclude that the obviously the better team won on much less evidence.

In this simulation you randomly deal hands and then let four GIBs who do some screwy stuff bid and play the hands.

In a major championship you randomly deal hands but then you duplicate them and let two groups of world class bridge players play them.

I think that it is clear thatt in the latter case the standard deviation is much smaller.

So it seems natural to me that if your first simulation suggests that 2S wins 56 IMPs against pass, to keep in mind that the actual expected value may be considerably smaller or larger, while if one world class team beats another world class team by 56 IMPs over 100 hands then it is more reasonable to assume that the winning team played better.

There are also other practical issues, obviously it is easier to let a computer simulate 1000 more hands than it is to keep the world class players for several more weeks.

[hanp]

[CODE]Overall while GIB does do some screwy things I don't feel or expect that it is particularly biased against one sort of action it seems that mostly its actions were ok on the hands that I looked at in more detail (that is more detail than +100 -50 +170). [QUOTE]

Your sentence isn't clear to me.

Do you mean to say that you don't have any reason to expect that GIB is biased in a particular way? Or do you mean to say that you expect that GIB is not biased?

I have some experience with bidding against robots on BBO. At the time I tried this (two months ago?) the robots were extremely reluctant to make a penalty double. It was incredible. We practiced bidding over 4H. Of course it was a big winner to bid 4S very aggressively, they would never double when you stepped out of line. Now I don't know if these robots ran on the same GIB engine, but if it did then I would keep in mind that the results of such a GIB simulation might have nothing to do with the real world.

If the GIB you are using is much more trustworthy then that would be good to know.

[whereagles]

what a revolting 2♦ opener!

but who are we argue with sucess?? LOL

[jdonn]

It's interesting to me that I think we should double 3♦ on the way back whether it's takeout or penalty (if it's takeout I would pull 3♥ to 3♠ since I think the double initially gives the wrong impression of our hand).

[Cascade]

hanp, on Apr 29 2009, 09:20 AM, said:

Cascade, on Apr 28 2009, 03:21 PM, said:

Although we are happy to decide major championships by many fewer boards and some conclude that the obviously the better team won on much less evidence.

In this simulation you randomly deal hands and then let four GIBs who do some screwy stuff bid and play the hands.

In a major championship you randomly deal hands but then you duplicate them and let two groups of world class bridge players play them.

I think that it is clear thatt in the latter case the standard deviation is much smaller.

This maybe true but I am not so sure that it is obvious.

I am currently running a little experiment to check (well at least get a feel for this) by having GIB play the hands from a recent World Championship Final and comparing the variance (or standard deviation) of GIBs results with the variance of the results at the two tables in the World Championship Final.

I will let you know the results.

[jdonn]

Cascade, on Apr 29 2009, 11:02 AM, said:

I am currently running a little experiment to check (well at least get a feel for this) by having GIB play the hands from a recent World Championship Final and comparing the variance (or standard deviation) of GIBs results with the variance of the results at the two tables in the World Championship Final.

I think this kind of experiment is very interesting and a much better idea than trying to use GIB to test the correctness of bids early in a competitive auction. I am curious to see the results.

[hanp]

Cascade, on Apr 29 2009, 11:02 AM, said:

hanp, on Apr 29 2009, 09:20 AM, said:

Cascade, on Apr 28 2009, 03:21 PM, said:

Although we are happy to decide major championships by many fewer boards and some conclude that the obviously the better team won on much less evidence.

In this simulation you randomly deal hands and then let four GIBs who do some screwy stuff bid and play the hands.

In a major championship you randomly deal hands but then you duplicate them and let two groups of world class bridge players play them.

I think that it is clear thatt in the latter case the standard deviation is much smaller.

This maybe true but I am not so sure that it is obvious.

I am currently running a little experiment to check (well at least get a feel for this) by having GIB play the hands from a recent World Championship Final and comparing the variance (or standard deviation) of GIBs results with the variance of the results at the two tables in the World Championship Final.

I will let you know the results.

Very interesting but of course it has little to do with my comment (which was poorly phrased I admit but a good reader only needs half a word as they say in the Netherlands).

Suppose you let a completely deterministic computer program play the same hand twice. The variance would be 0, no matter how well the computer program plays bridge.

In your simulation you forced the computer program to make 3 different bids with the same hand. This is entirely different.

To get a similar comparison with the world championship hands, force dealer at the first table to always pass, and force the dealer at the other table to always open 1X in their longest suit. See what variance you get then...

[hanp]

jdonn, on Apr 29 2009, 11:11 AM, said:

Cascade, on Apr 29 2009, 11:02 AM, said:

I am currently running a little experiment to check (well at least get a feel for this) by having GIB play the hands from a recent World Championship Final and comparing the variance (or standard deviation) of GIBs results with the variance of the results at the two tables in the World Championship Final.

I think this kind of experiment is very interesting and a much better idea than trying to use GIB to test the correctness of bids early in a competitive auction. I am curious to see the results.

OK I just wrote that it was interesting but now I am not sure why it is. This simulation tests whether GIB is consistent in its choices, not whether what GIB does is good bridge.

[hanp]

As for a constructive comment:

What would be more interesting (to me, as a bridge player) would be to compare GIB's results with the results obtained by the two tables and look at the variance of the differences.

[Echognome]

Just thinking out loud. Would it make sense to cross IMP GIB's results with, say, the results from the World Championship or something along those lines. I'm just thinking of the "Play with the Champions" type competitions.

[Cascade]

2008 World Mind Sports Final standard deviation was 5.88 IMPs per board.

GIB vs GIB playing the same hands the standard deviation was 6.17 IMPs per board.

Comments

1. GIB was slightly more variable than the players. So maybe it is true that world class players are less variable than GIB but I still think it is not obvious. GIB e.g. never misses a spot card, never gets tired, never goes on tilt etc etc all human failings that will contribute something to the variance.

2. When I have done similar calculations in the past my recollection is the standard deviation has been higher.

3. I actually forced GIB to play two different systems as I was worried that if it played the same system then that would reduce the variance (standard deviation). GIB played ACOL 4-card majors NS at one table and EW at the other table and Standard American 5-card majors in the other seats (other team).

4. For the record GIB Standard American beat GIB ACOL by 49 IMPs. Before you non Acolites get too excited GIB ACOL was actually leading before the start of the last set but GIB Standard American won 53 IMPs in the last set of 16 boards.

5. Given the difference between England and Italy was 30 IMPs (or 30/96 IMPs per board) and the standard deviation was 5.88 then if there was no difference between the two teams over 30% of the time Italy would have won by a larger margin. Basically this means the result is not statistically significant. In order for the result to be significant at a 5% level if the true difference in the teams was 5/16 IMP per board then around 1000 deals would need to be played. At 1% significance we would need something over 1900 deals.

As an aside someone once told me that when Terrance Reese was asked how many deals would be needed to conclusively decide which team was better he replied 10000 (or something similar). Does anyone have a reference for a quote of this sort?

6. The arguments about the variance aren't really important in my view. If the variance is higher then we just need to do a bigger simulation so that the randomness from the variance is overcome and we can make some conclusions about the skill factor.

7. Small simulations as I have reported in this thread seldom prove anything - although if you get differences of the order of 1.5 IMPs per board as I reported the results are pretty conclusive after a few hundred deals.

8. The question we would like to prove is "Is overcalling 2♠ superior to passing?". All we actually prove with simulations like this is that GIB bidding 2♠ did better than GIB passing. It is possible there is something other than what is inate in the problem in the GIB engine that biases the results one way or another so that the extrapolation from GIB to real world is completely invalid. However by and large despite the many anecdotes that suggest otherwise GIB plays a reasonable game and its judgement actions are based on simulations which even when not good will simply increase the variance so we will have to make a bigger simulation. Basically the GIB simulation produces odd actions when it gets a bad sample - this is just like a human player making a mistake. It is just that GIBs mistakes are different than human mistakes.

9. If you could get to a world championship final repeatable actions that are 1.5 IMPs per board better than your opponents will see you win comprehensively.

10. Probably some other things but I am too tired to think about them ... maybe more later ...

11. I actually did another 100 hands with replicated the nature of the results I reported above but in this third set of 100 hands I also considered overcalling 2NT and overcalling 3♣. In this set of 100 hands the order from best to worst actions was:

2♠ +560
3♣ +332
Dbl -95
2NT -256
Pass -541

The standard deviation in for example the head to head 2♠ versus Pass was almost 8 IMPs per board which is higher than before and is to be expected when you force a variation between the two auctions. That would mean that a larger sample was needed to draw reliable conclusions. Head to head 2♠ was up over 2 IMPs per board on the 100 boards in this last sample.

[Fluffy]

I loved reading this stuff Wayne.

Go canape!, always the major over the longr minor .

[shevek]

JLOL, on Apr 26 2009, 10:33 PM, said:

I would rather kill myself than bid 2S.

No need to get personal.

I pass too but you could extend ELC doubles to auctions like that. One ELC princiiple is that they only apply when partner bids a minor, such as
1♠ - X - no - 2♣ - no - 2♦ = 4-5 reds. Or
1♣ - X - no - 1♦ - no - 1♥ = 4-5 majors.

You could try extending to auctions like this, when
2♦ - X - no - 2♥ - no - 2♠ = 4-5 blacks.

Not 5-4 the other way because that hand overcalls. I quite like that but partner's never agree to it, pointing out that partner might bid 3♥ or 4♥.

These sort of flexible doubles are more use to strong clubbers.
Change it to a 19-count and a different auction
♠AKxx  ♥Ax  ♦Kx  ♣AJxxxx

1♣* - 3♦ - no - no - ?

I like double then 3♠ to show this.

[se12sam]

Wayne, The analysis is brilliant. Thank you very much.

[Cascade]

...

12. GIB is probably biased against four-card overcalls as I imagine it is programmed to assume a five-card overcall when doing its simulations from partner's perspective. Obviously it sees its own cards when doing the simulations from the overcalling side and judges accordingly.

[awm]

My observations have been that:

(1) GIB has trouble penalty-doubling partscores. It does not make these doubles very often, and when it does the reason for doing so often seems sort of random. It also removes penalty doubles of partials fairly often for no obvious reason. GIB also makes a lot of defensive errors against partscore contracts (more than it makes against games or slams). Since one of the big downsides to acting on this hand is the possibility of a penalty double from LHO, this tends to help bidding/doubling disproportionately over passing.

(2) GIB responds to takeout doubles in very strange ways. I have seen it bid three-card suits in preference to four-card suits (even when 4-3 in the majors over a takeout double of a minor). It will do things like insist on a four-card suit after partner's double for no reason, or convert a takeout double to penalty with no real hope of setting. This tends to make doubling look worse than it would opposite a player who knows how to respond to takeout doubles.

(3) GIB seems to have determined (rightly or wrongly) that we need to get into the auction on this hand. Subject to that determination, it is probably safer to bid right away than to pass and then balance into a four-card spade suit over 3♦ or 4♦. Thus we are not really comparing "bid 2♠ now" against "pass throughout" -- we are comparing "bid 2♠ now" to "pass now and balance into spades at the three or four level." It's not particularly surprising that bidding 2♠ now wins. Obviously I could be wrong about this, but you did mention several hands where GIB passed and then backed in with some ridiculous call later on.

(4) Obviously the results depend heavily on which hands are eligible for a 2♦ preempt and how aggressively partner will balance or pre-balance in auctions like 2♦-P-P and 2♦-P-3♦.

[mike777]

Interesting thread I hope to hear more.

[Cascade]

awm, on May 1 2009, 09:14 AM, said:

(1) GIB has trouble penalty-doubling partscores. It does not make these doubles very often, and when it does the reason for doing so often seems sort of random. It also removes penalty doubles of partials fairly often for no obvious reason. GIB also makes a lot of defensive errors against partscore contracts (more than it makes against games or slams). Since one of the big downsides to acting on this hand is the possibility of a penalty double from LHO, this tends to help bidding/doubling disproportionately over passing.

Here are all of the hands from one set of 100 where GIB hand 10+ HCP and four or more spades sitting over the 2♠ bidder and an analysis of what happened.

Draw your own conclusions.

Some observations:

1. I wouldn't have doubled with all of these hands. Especially at IMPs I am a bit of a wimp with the axe.

2. While the big penalties +500, +800 etc that were missed look bad there will be more -670s etc when you consider if you double with all of these hands you will also be doubling with similar hands when the overcaller has a five or six-card suit and therefore probably will be able to take more tricks.

3. GIB only doubled five times. The weakest with an 11-count. There are 19 times when it had similar hands and did not double. Not all of those doubles would have been successful.

Here are the hands that GIB doubled on after 2♦ (2♠) ...

JT98.KQT98.98.AQ -670 when 2♠ made.

QT84.KJT7.54.AKJ +800

QJT7.KJ73.Q5.KQ7 +200

QJT64.KT85.A6.K7 +1100 (There were at least 8 tricks available in hearts)

QJ954.KT7.A5.QT6 +500

Here are the hands were GIB did not double when it might have ...

QJ965.KJT.T7.AQJ GIB bid 3♦ +500 was available in 2♠X

QJ95.KQJ3.AQ.QT6 GIB bid 3♦ - insane not only did they not get the +1400 or so penalty but they didn't get to their cold game.

J964.974.QT96.AK GIB bid 4♦ and beat 4♠ by one trick.

J9864.KJ.T73.AKJ GIB bid 3♦ and doubled 3♠ for +500

T985.QJ43.AJ8.A6 GIB bid 3♦ and 3♠ made

QJ985.J64.Q93.KQ GIB bid 3♦ - 2♠ is probably 1-off (its not 2-off)

J9875.J86.4.AKJ6 GIB Passed - 2♠ went 2-off so 500 was available

Q865.Q93.765.AKT GIB bid 3♦ - 2♠ is no more than one-off

JT86.KQJ95.87.K6 GIB Passed 2♠ went 1-off

QT64.K984.73.KQ6 GIB Passed 2♠ made

J8654.KQ98.J4.K6 GIB Passed 2♠ made an overtrick

QT74.43.A64.AQ75 GIB raised to 3♦ NS can play 3♥ they got to 4♥ down one.

Q864.K7543.6.AK6 GIB passed 2♠ went 3-off

QT9865.K6.Q.AKQ6 GIB passed which became the final contract. NS can make game in hearts which they did after an initial double. 2♠ was also the final contract after a Pass with the hand in the opening post as GIB apparently can bid 2♠ non-forcing which it did in response to the weak two.

QT876.J7.T87.AKQ GIB bid 3♦ which made. NS can play hearts in a 7-2 fit.

J976.K9643.Q.AJ6 GIB passed 2♠ went 1-off.

QJ864.QJ3.AT5.T5 GIB raised to 3♦. NS can play hearts which they did - 4♥ 1-off (maybe 3♥ making after a penalty double.)

9765.KQJ953.A.A6 GIB passed. 2♠ went four-off

T876.KQT94.A3.Q5 GIB bid 3♦ which made. 2♠ probably down one maybe down two.

[awm]

Well, suppose a simple rule like "double if you have two spade tricks assuming RHO has all the remaining high ones, and additionally you have 12+ overall hcp."

Under such a rule:

GIB would not double on the hand where 2♠X made, where it had only one trump trick.

GIB would still double on all the hands where 2♠X failed.

GIB would double on five of the hands where it did not double. On all of these, 2♠X would go off (in some cases several tricks). On two of these hands GIB obtains a much better result than it did at the table. On one hand GIB actually got to double 3♠ by passing for a bigger number, and on the other two there is some superior resting place for the other side which may or may not be reached after the penalty double of 2♠.

So it is the case that GIB made at least one bad double and missed at least a few opportunities for good doubles.

[Cascade]

A quick look suggests that a 12 count with four trumps (including some trump tricks) will allow 2♠ to make somewhere between 35% and 40% opposite a standard 6-9 hcp range weak two.

The opening post suggested this pair were "frisky". Adjusting the range to 5-8 hcp increased the makeable 2♠ to nearly 60%.

In both cases sometimes with overtricks.

All of this in the case of a more normal 2♠ overcall with 5-6 spades and 11-16 HCP.

I mean it is not surprising that a frisky double works out when the overcaller turns up with only a 4-bagger.

The real boundary for light-ish doubles will depend on how they preform on the more normal and more frequent five-card or longer overcalls.