# More determining sample size

#### shmeggeroni

##### New Member
Hoping for some help and advice from statistical gurus....

Please forgive that this question has probably been asked a multitude of times (including once about an hour ago) -- I can't find enough similarity to comprehend an answer in the existing threads.

I'm trying to use websites like http://www.surveysystem.com/sscalc.htm to determine what my sample size needs to be, but apparently, I don't understand some basic requirements well enough to make this a simple calculation.

Finally, on to my question...

Our warehouse fills picking carts with 25 orders each with items that are picked from amongst 13,000 items. At the time the picking carts arrive at the packing stations, I want to have someone review the carts for accuracy (especially for newer pickers). What I'm trying to figure out is how many slots on each (or on how many carts) need to be checked in order for the reviewer to be 99% sure that there are no errors on the cart.

In case some other data is relevant:
13000 different skus
2300 pick locations with 1-sku/location
1500 pick locations with multi-sku/location
85 picking carts with 25 slots (1 per order) on each cart
50-70 items picked on each cart (2-3 items per order)

I hope you can help me find a little guidance...

Thanks.

-Matt

#### shmeggeroni

##### New Member
I'm hoping to sample the contents of some of the picking carts and then make an assumption about the accuracy of the rest of that picker's picking carts.

To clarify a little bit about the 13,000 different skus -- those are all on shelves, not on picking carts. There are 85 about 85 active picking carts, and about 12,000 individual items being picked every day. Those 12,000 items are distributed among the 85 carts. And, of course, the carts are used many times per day.

So, one picker might load 30 carts in a day. How many slots on those carts would I have to sample in order to determine that this picker is 99% (or maybe 95%?) accurate?

I hope that I'm making this a little clearer? I appreciate any interest I can solicit in coming up with a workable answer to this question.

-Matt

#### talkstatsdkf

##### New Member
Matt,

I've done an analysis below that I hope will be helpful.

---------------------------------------------------
30 carts per day
25 orders per full cart
750 orders per stocker per day

(assuming 2.5 items per order)

1875 items per stocker per day

12,000 items per day

12,000 / 1875 = 6.4

so about 6 or 7 stockers in the warehouse.

The basic model is that when a stocker fills an order that is a trial which has some probability of success (we'll assume that this probability is either .95 or .99)

So each stocker fills about 750 orders in a day, so one could in principle have a sample size of 750 where the probability of success for each sample is (let's say .99).

The basic test to use is called in Devore's book Probability and Statistics for Engineering and The Sciences (5th edition) "tests concerning a population proportion" and is on page 335.

The formula is:

H_0 : p = p_0
H_a : p < p_0 (in this case we would probably want a one-tailed test)

(p_hat - p_0) / sqrt( (p_0) * (1 - p_0) / n )

p_hat -- estimated probability from data
p_0 -- probability under the null hypothesis

the constraints for this test are that (p_0 * n) >= 10 and
(1 - p_0)*n >= 10

(10 is what Devore chose, in another book it is said that other Statistics texts have a lesser constraint that (p_0 * n) >= 5 and (1 - p_0)*n >= 5 )

With a null hypothesis of p=.99 the more stringent constraint of
(1-p_0)*n >= 10 is not met with a sample size of 750:

750*.01 = 7.5

However, it is met with a null hypothesis of p=.95 :

750*.05 = 37.5

in fact, if one wanted only to get, say 50% about the limiting value of 10:

15 / .05 = 300

so a sample size of 300 would be sufficient to meet the constraints of the test (300*.05 = 15 >= 10) , or a sample size of 200 would just meet the published constraints for the test. (200*.05 = 10)

By increasing the sample size it is also possible to check for a higher accuracy (the 99% accuracy) (see note at the end)*

15 / .01 = 1500
10 / .01 = 1000

so similarly 1500 samples or 1000 samples would be required to check for the higher accuracy.

It is also possible under the model to compute the probability of a type-II error. In this case that would mean that the picking accuracy is some specified value lower than what is sought for, but the test does not find it.

David

*This might mean combining data from different stockers, and goes to a related question which is whether different stockers have different accuracy rates, and if they differ by how much do they differ, or could also mean combining data from different days.

p.s. Here are some other resources that might be helpful:

http://en.wikipedia.org/wiki/Hypothesis_testing#Common_test_statistics
http://www.stat.berkeley.edu/~stark/SticiGui/Text/zTest.htm#z_for_percentage

#### terzi

##### TS Contributor
When obtaining a sample size, you want to estimate something (a parameter). I didn't completely understand what you are estimating here. Are you interested in the products or in the carts? Do you want to decide whether the products in cart are completely correct? What would you measure exactly?

#### shmeggeroni

##### New Member
Hi Terzi. I think I understand your question -- I want measure/estimate the accuracy of the picker who placed the products in the cart. -Matt

#### shmeggeroni

##### New Member
Hi talkstatsdkf. I'm still digesting...

Really, it sounds like our production volumes are too small to make strong estimates about the cart-by-cart accuracy of our workers. What I was hoping to learn is, e.g., that we can sample ~10 slots on each cart and then feel confident that the remainder of the cart is correct.

But, your analysis indicates that I need to do a bit more reading (thanks for the references) and plan on a longer term strategy with increased sampling sizes by rating pickers on a day-to-day basis or rating groups of pickers together.

Does it sound like I'm beginning to understand your analysis?

-Matt

#### talkstatsdkf

##### New Member
Hi Matt,

I've done a further anlaysis which may be more along the lines of what you are looking for:

http://www.dkfriedman.name/picker_calc.html

The calculator takes three parameters:

p - The probability that a picker sets a given slot correctly
n - The number of slots on a cart
m - The number of slots checked on a cart

and outputs the probability that the cart has no errors given that the number of slots checked has no errors.

The calculation indicates that with a picker accuracy of 0.996 with a 25 slot cart checking 12 slots yields a probability of about 0.95 that if all 12 slots are correct the entire cart is correct.

It would be good for me to post my analysis to the forum, and to also double check that the formula has been inputted correctly. I can probably get to that tomorrow.

David

#### shmeggeroni

##### New Member
Wow. Sweet!

This is a very interesting way to look at the predictability of our picking output. Thank you for all your work on this.

"Probability that picker sets a given slot correct" is our historical recorded percentage of correctly filled cart slots?

-Matt

#### talkstatsdkf

##### New Member
Wow. Sweet!

This is a very interesting way to look at the predictability of our picking output. Thank you for all your work on this.
you're welcome. I hope that it is helpful.

Matt, one question I have, is how difficult is it to check a slot for accuracy? How much time and effort does it take?

It might help to have some more context about the problem. Feel free to discuss your business and your process in more detail. If there is a company website feel free to post it. Here are a few questions that might help in working toward a solution:

-What happens if an incorrect order gets shipped? What is the cost to the company when this happens?
-How much time, effort and expense is budgeted for the project?
-Are there any specific time constraints by which the analysis must be done?

If there is any other information that you think could be relevant to a solution or helpful feel free to mention it.

"Probability that picker sets a given slot correct" is our historical recorded percentage of correctly filled cart slots?
yes. it could be estimated through historical data.

David

Last edited:

#### talkstatsdkf

##### New Member
hi all,

Here is my analysis.

Define two random variables:

EP -- Errors produced in picking
EF -- Errors found when checking

Define three parameters:

p -- the probability that a given slot is filled correctly
n -- the number of slots on a cart
m -- the number of slots checked on a cart

The probability that is sought is:

P( EP=0 | EF=0) = P( EP=0 and EF=0) / P(EF=0)

Errors found must equal 0 when errors produced equals 0:

= P(EP=0) / P(EF=0)

Errors found can equal 0 if there are anywhere from 0 errors produced to n-m errors produced. If errors produced exceeds n-m then it is not possible to find 0 errors.

= P(EP=0) /( P(EF=0 and EP=0) +
P(EF=0 and EP=1) +
P(EF=0 and EP=2) +
....
P(EF=0 and EP=n-m) )

The probability that there are no errors is p^n (i.e. P(EP=0)=p^n)

The probability P(EF=0 and EP=i) is the product of a hypergeometric probability and a binomial probability. The hypergeometric probability is equal to the probability that 0 of the i bad slots are checked and m of the n-i good slots are checked:

binomial(i,0)*binomial(n-i,m) / binomial(n,m)

the binomial probability is equal to the probability that i bad slots are produced in the picking process:

binomial(n,i)*(p^(n-i))*(1-p)^i

so the denominator is a summation going from 0 to n-m of these products:

n - m
====
\
> binomial(n,i)*binomial(n-i,m)/binomial(n,m)*binomial(n,i)*p^(n-i)*(1-p)^i
/
====
i = 0

it is possible to check this sum conceptually for i=0,1,2 etc.

for i=0 we can see that the expression yields p^n (the probability that no errors are produced)

for i=1 we have the probability that although there is one error we do not select it but pick all from the n-1 that are good multiplied by the probability that one error is produced.

similarly for i=2 we have the probability that although there are two errors we do not select from them but pick all from the n-2 that are good multiplied by the probability that two errors are produced.

the equation then for the P(EP=0 | EF=0 ) is:

this is what is implemented in javascript on the calculator web page:

http://www.dkfriedman.name/picker_calc.html

David

#### shmeggeroni

##### New Member
Matt, one question I have, is how difficult is it to check a slot for accuracy? How much time and effort does it take?
It takes too much effort right now. We're having people go to the cart, scan the order number (from the packing slip that's already in the cart slot) and then use a browser-based utility program to scan each UPC in the cart to make sure the product matches the item in the cart. If we really want to do this right, we need to make a better task-driven program whose purpose is to inspect a cart. How many slots to check for this cart (maybe factored by who picked the items in the cart?) would be one element that makes this analysis so important.

It might help to have some more context about the problem. Feel free to discuss your business and your process in more detail. If there is a company website feel free to post it.
We have a seasonally high-volume business of selling costumes. So, right now is the time when we realize all of our failures in planning for this year's growth. The high-volume of picking combined with some more "efficient" picking strategies has left us wishing for some more accurate picking. We're still maintaining an acceptable level of accuracy, but we're also still wasting a ton of money on making up for picking errors.

What happens if an incorrect order gets shipped?
An incorrect order might be the wrong item or it might be a missing item. Usually, when this happens, we don't know that it happened until the customer calls us on the telephone and one of the customer service reps hears words like moron, lame, and other unspeakable verbal catastrophes. At that point, we apologize and initialize a returns procedure that includes sending out a replacement item and hoping that the wrong item gets returned to us. We usually get the wrong item back, but people are slower shippers than we are, and if this error occurs after Oct 15th, we likely won't be able to sell the returned item for Halloween. Am I answering the question? We then ship the correct item with extra special care to do it right this time.

What is the cost to the company when this happens?
~?? customer dissatisfaction.
~$20 for us to reship (we usually reship with overnight service). ~$3 customer service time
~$2 repicking/repacking time ~$1.5 shipping materials
~$8 for return shipping to us (if we sent the wrong item). ~$? cost of the other item (if we sent the wrong item).

How much time, effort and expense is budgeted for the project?
Sometimes I wish we were big enough and corporate enough to have a budget for a project like this. Really, we just have to spend less money than we'll lose by sending out wrong or missing merchandise.

Are there any specific time constraints by which the analysis must be done?
We promise same-day shipping if customers place their order before 4pm (3pm right now at the height of the season). So, our time limit is just to do it in the flow of the order from picking to packing. We don't want to sacrifice our shipment guarantee unless we really, really have to.

If there is any other information that you think could be relevant to a solution or helpful feel free to mention it.
I wish I could think clearly enough to produce more information. We're looking toward the future for using this analysis. The problem arises at the height of the season because this time period reveals our biggest flaws. Thanks again for all your help so far with this. I'm going to get back to another long day right now...

-Matt

#### talkstatsdkf

##### New Member
Matt,

This being the peak season for your business you may feel that it is not the right time to invest in significant changes.

One suggestion could be to have the customer service reps keep a record or
log whenever customers call saying that their order was wrong. This wouldn't be an exact accuracy measurement because some customers may not call even if the order happened to be wrong, but still could be worthwhile information.

With the right methods in place accuracy can improve, and that you've got a large volume of sales is consistent with good performance and growth

I hope that my posts have helped, and that your business does well.

David

Last edited: