Have Geek Brain

Determination of the smallest number of samples for 99% accuracy

I am trying to compare 100,000 records in a local database (L) with 100,000 records in a remote database (R).

Basically, I want to know if an element exists in L in R. To determine this, I have to make a query against R for each L, which takes a long time (I know there must be a better way, no that I have an API) ...

So, I would like to test a small sample of L versus R and then output with some level of confidence how many of them are present in the whole R. How much should I test to have a 99% confidence level?

+2


a source to share


3 answers


If you test N records from your local database, and they are all in the remote database, you can estimate the likelihood that the local record will not be in the remote database as between 0 and 3 / N. This is called the "rule of three" in statistics. I explain it here .



The only way to know that all records in both databases are to check everything. But if you check 100 records, for example, you can estimate that the proportion of records not in both databases is below 3%.

+5


a source


I also suggest experimental design for estimating a proportion p

.

Suppose we are interested in estimating the fraction of p elements in L that also exist in R, and we would like to calculate 99% CI with a tolerance level (lvl) that is plus or minus 3%. A "conservative" estimate of the size of the random sample will be determined as follows:

n = (Za / 2) ^ 2 / (4 * lvl ^ 2)



In R

CI<-.99
lvl<-.03    
qnorm(1-(1-CI)/2,0,1)^2/(4*lvl^2)
[1] 1843.027

More about more

+3


a source


Is this a trick? Is it 99% right? After checking each of them, you will know with 100% certainty whether it is in the remote database or not, so if you want to check the entire database for 99% accuracy, you need to check 99% of the records (99,000).

0


a source






More articles:


All Articles