Better than guessing? Why the chance rate isn't 50%

The question

Jyotishya's quick check asks how yesterday went (thumbs up, so-so or thumbs down) and compares the answer with each system's call for that day. The obvious test counts the hits and asks whether more than half are right. That test is wrong whenever either side doesn't split 50/50.

Formula

If a system calls a day good with probability g, and a person answers thumbs up with probability u, independently of the call, they agree by chance with probability u × g + (1 − u) × (1 − g). It equals 50% only when u or g is exactly 50%. The same idea underlies Cohen's kappa, the standard measure of agreement beyond chance.

Worked example

In a simulation of 12 made-up birth charts over every day of 2026, the Vedic rules called 'good' on about 64% of the days they made a call, the Western rules on 57% and the Chinese on 59%.

Take someone who answers thumbs up on 70% of days, with no connection to any forecast. They agree with the Vedic call 0.70 × 0.64 + 0.30 × 0.36 = 0.56, or 56% of the time. Tested against 50%, that person would sooner or later see Vedic marked 'better than guessing', purely from the two base rates.

Two smaller traps

Small samples: 7 hits out of 7 has a two-sided binomial p of 0.0156, below 0.05, so a test with no minimum can declare a result within a week.

Repeated looks: re-testing after every answer and stopping at the first significant result raises the false-alarm rate. Armitage and colleagues showed that ten unadjusted looks turn a nominal 5% into nearly 20%.

How Jyotishya tests it

Each system is tested with Fisher's exact test on the 2 × 2 table of answers (up or down) against calls (good or hard), which compares the agreement with what both sides' base rates give by chance. So-so answers and Mixed days are left out. A system gets a verdict only once it has 10 calls of its own, and only at p below 0.05 ÷ 3, because three systems are tested at once. The page shows each system's chance rate instead of 50%. The quick check is still unblinded, since the forecasts are visible in advance, which is why the blind diary exists.