FOI Request LEX3389, Schedule of Released Documents [PDF 123KB] (pdf)
Download cached file | Download from AEC--- Page 1 --- Attachment A – Schedule of Relevant Documents Doc ref Document description Exemption Decision on Access Document 1 Extract of ABS sample size calculator set to risk rejection at 6% for rejecting a valid list Document 2 Advice from ABS Chief Methodologist on sampling and methodology used dated 29 July 2022 Document 3 Letter to Mr Tom Rogers, Australian Electoral Commissioner from Dr Gruen AO, Australian Statistician regarding non-parliamentary political party membership testing methodology dated 2 September 2022 - - - Document 4 ABS Party membership testing methodology table - Published on the Disclosure Log under LEX3388: https://www.aec.gov.au/information-access/foi/2022/ Published on the Disclosure Log under LEX3388: https://www.aec.gov.au/information-access/foi/2022/ Published on the Disclosure Log under LEX3388: https://www.aec.gov.au/information-access/foi/2022/ Published on the Disclosure Log under LEX3388: https://www.aec.gov.au/information-access/foi/2022/ Document 5 Advice from ABS Chief Methodologist on sampling and methodology used dated 19 August 2022 47F Access in Part
This document is a schedule of five documents released in response to an FOI request concerning the Australian Bureau of Statistics' (ABS) methodology for sampling and testing political party membership lists.
Summary of Document:
The document, titled "Attachment A – Schedule of Relevant Documents," lists five specific records:
* Document 1: An extract from the ABS sample size calculator, configured for a 6% risk of rejecting a valid list.
* Document 2 & 5: Two separate pieces of advice from the ABS Chief Methodologist on sampling and methodology (dated 29 July 2022 and 19 August 2022, respectively).
* Document 3: A letter from the Australian Statistician to the Australian Electoral Commissioner regarding non-parliamentary political party membership testing methodology (dated 2 September 2022).
* Document 4: The ABS Party membership testing methodology table.
All documents except Document 5 (which was partially accessed) are indicated as published on the AEC Disclosure Log.
Relevance to FOI Request:
This schedule directly confirms the release and identifies key documents pertaining to the FOI request's scope. It shows that the ABS provided specific advice and methodologies related to sampling and testing political party membership lists, particularly concerning:
* The statistical basis for sample size determination, including the risk of false rejection (notably the 6% threshold).
* Expert advice from the ABS Chief Methodologist on the sampling methodology used.
* The official methodology and associated tables used for testing non-parliamentary political party membership, which directly addresses the core of the request regarding the deregistration review of VoteFlux.Org | Upgrade Democracy! The document serves as an index to the specific records that contain the detailed information described in the FOI request overview, such as the requirement for random sampling from the whole list and the principles of sample size calculation.
LEX3389 documents [ZIP 865KB] (zip)
Download cached ZIP | Download from AECZIP Contents
Document 5 - Email from ABS 19 August 2022_Redacted.pdf (pdf)
Download file--- Page 1 --- --- Page 2 --- --- Page 3 --- --- Page 4 --- and I last week on Thursday 21 July. At that meeting we discussed the ABS Thank you for meeting with methodology for sampling and testing membership of political parties used by the AEC. You kindly reviewed some questions we had and considered some issues raised by applications for review of the decision to deregister VoteFlux.Org | Upgrade Democracy! (the Party). The purpose of this email is to summarise the key points raised in that discussion. I would be grateful if you could confirm that I have set out your advice correctly. If not, I would be grateful if you could edit the response or reply clarify any points. If possible, we would be grateful for your response by Monday 1 July 2022. We may provide this information to the Electoral Commission for the purposes of their review of the decision to deregister the Party. 1. What conclusions can be drawn in relation to the list of 4680 names in the Party’s list, following the delegate’s decision to test the top 1,650 names of that list (the sub-list)? With respect to the test conducted it is not the randomisation or not that is causing the false rejection (or false acceptance) rate to land outside the limits. It is the sample size that control the test conditions. If you had randomised you would have been able to say something about the whole list as the final sample (probabilistically) would have referred to the 4680 not just the 1650. Nevertheless you still would have needed a bigger sample size to get the desired risk rates. You advised that as the Party’s list was not randomised before the sub-list was made (as only the top 1650 names were selected) it is not possible to draw any meaningful statistical conclusions about the Party’s whole list of 4680 from the results of testing the sub-list. You can only say something about the selected 1650. This is because, without randomisation there is no chance of the other records being selected. You explained this by the analogy of attempting to sample from a deck of cards for aces. Having failed to first shuffle the deck, chunking the bottom half away, and then sample from what is left will not give you useful information about all original cards. Those thrown away were never in the running. The test done indicates a list with low proportion of eligible members among the 1650. DELETE As would usually be the case, as the test of the sub-list failed, it is possible to conclude (to the level of confidence to which the test is set) that the sub-list of 1,650 did not include 1,500 members. 2. What are the alternative options for testing the larger list? You explained that a larger sample size would be required to test a large list to the same degree of confidence. Current sampling is calibrated so that the probability of accepting an invalid list is less than 2%, and the probability of falsely rejecting a valid list is less than 6%. You estimated that the sample size required to test the full list of 4680 names would be over 300. You also advised that excel spreadsheet calculator provided to the AEC could provide information about the sample size required to test a list of 4680 names to different levels of confidence. We have now used the calculator (attached), to calculate that sampling a list of 4680 members, where the desired probability of false rejection is 6% and the desired probability of false acceptance is 2%, would require an approximate sample size of 564 and a maximum number of denials allowed of 399. 3. Does filtering names affect the error rates? You explained that in your view, the arguments made in paper with respect to filtering names increasing the error rate are without foundation. You explained that, provided the filtering process is done in accordance with the ABS methodology, filtering names works in favour of parties by removing from a list members who would not have been capable of meeting the requirements. Correct this would improve the ‘quality’ of the list and decrease the occurrence of finding denials (non-members) in the list sample. 4 section 47FS 47F --- Page 5 --- 4. Your general comments on paper hypothetical example. That example You considered that it was not instructive to consider in depth started from the premise that the party has more than 1,500 and sought to prove that a specific list could be rejected by the sampling methodology. You agreed with the general proposition that if the correct sampling size was not adopted in relation to a larger list, the likelihood of false rejection increased. This is shown by the calculator. We also discussed generally the rationale for requiring a smaller sampling size as a practical and fair method for testing party lists. We discussed the difficulties of testing a larger list. Since the requirement is minimum 1500 a party with a very large list that is ‘low quality’ in the sense that it contains a high percentage of non-members will require a very big sample size to control false rejection risks. This relates to the incentives of parties to keep good records of their members and provide the AEC with a high quality list. Providing large low quality lists should be discouraged. Legal Services Section | Legal & Procurement Branch Australian Electoral Commission DISCLAIMER: If you have received this transmission in error please notify us immediately by return email and delete all copies. If this email or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the email or attachments. DISCLAIMER: If you have received this transmission in error please notify us immediately by return email and delete all copies. If this email or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the email or attachments. 5 section 47Fsection 47FS 47FS 47F
This document is an email from the Australian Electoral Commission's Legal Services Section, confirming advice from the Australian Bureau of Statistics (ABS) Chief Methodologist regarding the methodology for sampling and testing political party membership lists, specifically in the context of the deregistration review of VoteFlux.Org | Upgrade Democracy!
Key points of the advice include:
* Statistical Validity: Drawing conclusions about an entire list (e.g., 4,680 names) requires random sampling from the whole list. Testing a non-random sub-list (e.g., the top 1,650 names) only provides statistical conclusions about that specific sub-list, not the total list, because records outside the sub-list had no chance of selection.
* Sample Size and Confidence: To maintain desired confidence levels (e.g., false acceptance below 2%, false rejection below 6%) for larger lists, a proportionately larger sample size is required. For a list of 4,680 members, the required sample size is approximately 564, allowing a maximum of 399 denials.
* Filtering Names: Filtering names in accordance with ABS methodology improves the "quality" of the list by removing non-eligible members, thereby decreasing the likelihood of finding denials in the sample and reducing the probability of false rejection.
* Methodology Rationale: The sampling methodology is designed as a practical and fair system. It encourages parties to maintain accurate membership records and provide high-quality lists. Using an incorrect sample size for larger lists significantly increases the probability of falsely rejecting a valid list.