FOI Request LEX3134, Schedule of Released Documents [PDF 62KB] (pdf)
Download cached file | Download from AEC--- Page 1 --- Request for: FOI REQUEST NO. LEX3134 • The report from Axiom Associates on the audit they conducted to assess the accuracy of the 2022 Senate scanning and digitisation process. SCHEDULE OF RELEASED DOCUMENTS Doc No. Description Draft – Senate assurance detailed report and assurance outcomes Draft – Senate collation of Stage 2 assurance exceptions Date 5/07/2022 5/07/2022
This document serves as the schedule of released documents for FOI Request No. LEX3134. It confirms the release of two draft documents dated 5 July 2022: a "Senate assurance detailed report and assurance outcomes" and a "Senate collation of Stage 2 assurance exceptions." These documents directly address the FOI request for Axiom Associates' audit report assessing the accuracy of the 2022 Senate scanning and digitisation process, indicating that the sought information, albeit in draft form, has been provided.
LEX3134 documents [ZIP 7.45MB] (zip)
Download cached ZIP | Download from AECZIP Contents
22FE Senate assurance detailed report and assurance outcomes_DRAFT (A1916202).pdf (pdf)
Download file--- Page 1 --- External statistical assurance of the Senate ballot papers Detailed report and statement of outcomes (draft) Australian Electoral Commission July 2022 --- Page 2 --- Australian Electoral Commission | July 2022 Contents 1. STATISTICAL ASSURANCE ACTIVITIES PERFORMED .............................................................................. 1 LEGISLATIVE CONTEXT FOR BALLOT PAPER ASSURANCE ............................................................................... 1 ASSURANCE PROCESSES UNDERTAKEN ......................................................................................................... 1 SAMPLE SELECTION ................................................................................................................................................... 1 ASSURANCE TESTING ................................................................................................................................................. 3 QUALITY ASSURANCE ................................................................................................................................................ 4 AEC REVIEW OF POTENTIAL EXCEPTIONS ................................................................................................................. 5 REPORT CREATION – STATISTICAL ANALYSIS ............................................................................................................ 5 2. FINDINGS, STATISTICAL ANALYSIS AND STATEMENT OF ASSURANCE OUTCOMES ................................. 6 APPENDICES ........................................................................................................................................... 9 APPENDIX A: MARKS ON BALLOT PAPERS NOT RECORDED ACCURATELY IN BPRS WITH NO IMPACT ........... 9 APPENDIX B: ABS ADVICE TO AEC ON SAMPLING METHODOLOGY ............................................................. 11 Statement of Responsibility © Axiom Associates (Aust) Pty Ltd 2022. All Rights Reserved. Axiom Associates (Aust) Pty Ltd documents and reports may be relied upon by the Australian Electoral Commission (AEC) for purposes only pursuant to the terms of our engagement outlined in the Services Agreement Contract Number: SRC001297 dated December 2021 and any subsequent amendments. We disclaim all responsibility to any other party for any loss or liability that the other party may suffer or incur arising from or relating to or in any way connected with the contents of our documents or reports, the provision of our documents or reports to the other party or the reliance upon our documents or reports by the other party. Liability limited by a scheme approved under Professional Standards Legislation. Document Control Version Number 1.0 1.1 1.2 1.3 Date Provided 30 June 2022 1 July 2022 5 July 2022 Initial draft copy provided to the Australian Electoral Commission for management comments Track changes to the report Report updated to reflect AEC feedback and statistics updated Detail External statistical assurance of the Senate ballot papers i --- Page 3 --- Australian Electoral Commission | July 2022 1. STATISTICAL ASSURANCE ACTIVITIES PERFORMED LEGISLATIVE CONTEXT FOR BALLOT PAPER ASSURANCE The Electoral Legislation Amendment (Assurance of Senate Counting) Act 2021 (the Act) sets out the 1.1 requirements for ballot paper sampling assurance throughout computerised scrutiny of votes in Senate election. 1.2 Section 273AC requires the Electoral Commissioner to arrange for statistically significant samples of ballot papers to be checked throughout the scrutiny of votes for the election to assure that the electronic data used in counting the votes reflects the data recorded on the ballot papers. The ballot paper sampling process is not part of the scrutiny in relation to the election.1 The ballot 1.3 paper sampling process may be inspected by the scrutineers at the counting centre where the scrutiny is being conducted.2 1.4 Before the polling day for the election, the Electoral Commissioner must publish on the Electoral Commission’s website: a methodology to be used for the ballot paper sampling process; and the process to be used for reconciling preferences.3 1.5 Within 14 days after the return of the writ for the election, the Electoral Commissioner must publish on the Electoral Commission’s website a statement setting out the outcomes of the ballot paper sampling process.4 ASSURANCE PROCESSES UNDERTAKEN On 16 May 2022, the Australian Electoral Commission (AEC) approved the methodology and 1.6 operational approach that would be used to deliver the external statistical assurance of the 2022 Senate ballot papers. The assurance process would consist of five components: sample selection, assurance testing, quality assurance, AEC review of potential exceptions and report creation. 1.7 The following sections outline how the approved methodology was implemented for the 2022 Senate ballot paper assurance activities and any deviations from the agreed methodology. It also details any risks of these deviations to the statistical assurance outcomes. SAMPLE SELECTION 1.8 AEC determined that sample selection must occur at the Central Senate Scrutiny (CSS) centres in each state and territory. A physical ballot paper was at the CSS and this was compared to the electronic data used by the AEC in the counting of the votes (preference data recorded in AEC’s Ballot Paper Reconciliation System (BPRS)). This approach to sample selection was based on the flow of ‘physical ballot paper to electronic data 1.9 record’. As a result, the assurance was designed to conclude on whether, for each ballot paper tested, there were matching electronic data used in counting the votes. The assurance was not designed to conclude on the completeness of the number of physical or 1.10 electronic ballot papers or the accuracy of the count. The approach to assurance testing was not designed to 1 Subsection 273AC(4) of the Electoral Legislation Amendment (Assurance of Senate Counting) Act 2021. 2 Subsection 273AC(5) of the Electoral Legislation Amendment (Assurance of Senate Counting) Act 2021. 3 Subsection 273AC(6) of the Electoral Legislation Amendment (Assurance of Senate Counting) Act 2021. 4 Subsection 273AC(7) of the Electoral Legislation Amendment (Assurance of Senate Counting) Act 2021. External statistical assurance of the Senate ballot papers 1 --- Page 4 --- Australian Electoral Commission | July 2022 detect if there were electronic data records held by the AEC that were not associated with a physical ballot paper or whether there were multiple electronic data records associated with a single physical ballot paper. The risk of the introduction of additional electronic records and data into AEC systems is the subject of 1.11 the requirements of section 273AA of the Act related to assurance of security of computer systems for scrutiny of votes in Senate election. This is to be reported separately by the Electoral Commissioner. In November 2021, the AEC received advice from the Australian Bureau of Statistics (ABS) to determine 1.12 the number of ballot papers to be selected for statistical assurance purposes.5 This advice identified the number of ballot papers nationally to be selected to provide statistical assurance and the breakdown of the number of ballot papers to be selected in each CSS state and territory location. The ballot paper assurance process used the sample sizes identified by the ABS. In all CSS state and territory locations, the assurance process oversampled ballot papers by a small number to allow for any contingencies such as where a batch was rescanned after testing. 1.13 The sample sizes are summarised in Table 1. Table 1: Ballot paper assurance process sample sizes CSS state and territory location ABS ballot paper sample size a Actual ballot paper sample size in 2022 NSW VIC QLD WA SA TAS ACT NT Total national sample size Note a: ABS advice to AEC on sampling methodology, November 2021 1,733 1,377 1,272 1,272 1,200 1,106 977 958 9,895 1,735 1,400 1,310 1,285 1,245 1,135 1,028 965 10,103 1.14 The ABS also identified that cluster sampling of ballot papers would assist with the practical implementation of the assurance process. The approach to cluster sampling would allow the AEC to test five ballot papers from a single batch of 50 ballot papers. This approach was used to reduce the number of batches of ballot papers to be subject to assurance testing. In each CSS state and territory location the sample was selected throughout the scanning process. 1.15 Sample selection was based on schedules of extraction times to select ballot paper transport containers (BPTCs) which had just been scanned. The sampling approach was to use the first batch in each of these selected BPTCs and to test five ballot papers in that batch. BPTCs were not tested until BPRS indicated that the status of the BPTC was ‘confirmed’, indicating that data had been transmitted and reconciled by the AEC. 1.16 The extraction schedules were developed by Axiom to align to the proposed Fujifilm scanning shift schedules provided to the AEC prior to the election. These sampling extraction schedules were updated during the course of scanning to reflect changes in the Fujifilm scanning shift schedules. Due to the late notification of some changes to scanning shifts (not scanning on some planned days, shorter shifts and extended shifts), the sampling extraction schedules were not fully aligned to Fujifilm scanning shifts and extractions could not always 5 Refer to Appendix A of this document for a full copy of the ABS advice. External statistical assurance of the Senate ballot papers 2 --- Page 5 --- Australian Electoral Commission | July 2022 be performed as planned. This deviation from the agreed methodology is considered to be low risk as assurance testers independently determined when to select the BPTCs for testing. Due to a number of unplanned staff absences in the Northern Territory CSS location, extractions were 1.17 not performed in line with the extraction schedules and as a result, there was a risk of significant under sampling in the Northern Territory. To overcome this risk and deviation from the agreed methodology, Axiom instructed the Northern Territory assurance staff to extract the sample by selecting two batches from each BPTC selected and testing five ballot papers in each batch. In effect, selecting 10 ballot papers from a single BPTC rather than five as specified in the approved methodology. This allowed for confidence that assurance testing was being performed on BPTCs that had been extracted immediately after scanning. This deviation from the agreed methodology is considered to be low risk because the independence of sample selection was maintained and the principles of the cluster sampling (five in a batch) was preserved. ASSURANCE TESTING 1.18 Assurance testing was undertaken using a two-stage testing approach. 1.19 The first stage tested the physical ballot paper to the scanned image transmitted to the AEC and included in BPRS. The first stage testing assessed the completeness and accuracy of the physical ballot paper to the scanned image. It was used to validate that all marks on the scanned image were an exact replica of the physical ballot paper. The second stage tested the physical ballot paper to the electronic data used in counting, contained in 1.20 BPRS. The second stage testing assessed whether: the electronic preference data file was an accurate reflection of the preferences recorded on the physical ballot paper; and interpretation of voter intent aligned with AEC business rules as stated in the Ballot Paper Formality Policy and Ballot Paper Formality Guidelines. Figure 1 outlines the two-stage testing approach. Figure 1: Two-stage ballot paper assurance testing approach 1.21 The ballot paper assurance testing was designed to detect the following: where the scanned image was an incomplete representation of the physical ballot paper (the marks on the scanned image were not an exact replica of the physical ballot paper); and where preferences recorded on the physical ballot paper did not match the electronic data in BPRS which was used in counting. External statistical assurance of the Senate ballot papers 3 --- Page 6 --- Australian Electoral Commission | July 2022 The assurance testing approach was designed to identify potential exceptions impacting on statistical 1.22 assurance results. The types of potential exceptions the assurance was designed to identify included6: where no scanned image that exactly replicates the physical ballot paper can be found; voter disenfranchised where a vote was recorded as informal when the voter expressed a valid preference(s); above the line voter preference sequence was broken too early within the first six preferences; above the line voter preference sequence was broken too early after the first six preferences; below the line voter preference sequence was broken too early within the first twelve preferences; below the line voter preference sequence was broken too early after the first twelve preferences; voter preference sequence was broken too late (additional preferences were recorded); and vote recorded as formal when it should have been informal. 1.23 Assurance testing was performed by Temporary Election Workforce (TEW) resources selected and engaged by the AEC to act as assurance testers. To maintain independence of the assurance activities performed, Axiom received representations from the AEC that no assurance testers had performed non- assurance activities in the relevant CSS state and territory location for the 2022 Senate election. Assurance testers were trained by Axiom on the assurance approach, detailed methodology, 1.24 operational approach and expectations prior to the election. For the Northern Territory, the assurance testers were also provided with supplementary training. The AEC recruited additional assurance testers after the election in the Northern Territory and Victoria. These additional assurance staff were also provided with training. Any questions or issues related to the application of the assurance methodology and results were 1.25 required to be referred to, and answered by, Axiom staff. 1.26 With one exception, Axiom staff performed their activities remotely. The assurance testers in the Victorian CSS location identified a number of complex potential exceptions which required an Axiom staff member to visit the CSS location to assist with resolution of the exceptions. QUALITY ASSURANCE 1.27 of re-performance of 30% of stage two testing by different assurance testers. Ballot paper assurance testing was subject to quality assurance. Quality assurance activities consisted 1.28 Quality assurance activities assessed whether: the electronic preference data file was an accurate reflection of the preferences recorded on the electronic scanned image of the ballot paper; and interpretation of voter intent aligned with AEC business rules as stated in the Ballot Paper Formality Policy and Ballot Paper Formality Guidelines. 6 For the 2022 Federal Election, voter instructions on the Senate ballot paper were to number a minimum of six preferences above the line or twelve preferences below the line, across all states and territories. These instructions were used as the cut off for the classification of exceptions. External statistical assurance of the Senate ballot papers 4 --- Page 7 --- Australian Electoral Commission | July 2022 1.29 In addition, all assurance test sheets were reviewed by Axiom staff and all potential exceptions were confirmed and collated by Axiom. The Axiom review of test sheets and the quality assurance processes did not indicate issues with the quality and consistency of testing performed by any of the assurance testing teams. AEC REVIEW OF POTENTIAL EXCEPTIONS 1.30 Stage Two potential exceptions frequently arise because of judgements, by Fujifilm and AEC staff, about the interpretation of preferences recorded on ballot papers. These judgements are intended to be determined through the application of the AEC’s Ballot Paper Formality Policy and Ballot Paper Formality Guidelines. 1.31 All potential exceptions from Stage Two testing were subject to review by three experienced AEC officers. The purpose of this AEC review was to confirm how AEC experienced officers would apply the AEC business rules for interpretation of voter intent. The AEC chose four experienced officers to perform the AEC review of potential exceptions. The AEC review of potential exceptions was performed by three of the four AEC officers. Three tranches of potential exceptions were sent to and reviewed by the AEC officers as follows: tranche one sent on 17 June 2022 to officers 1, 2 and 3; tranche two sent on 22 June 2022 to officers 1, 2 and 3; and tranche three sent on 28 June 2022 to officers 1, 2 and 4 (to accommodate leave taken by AEC officer 3). 1.32 For each potential exception identified in Stage Two testing by the ballot paper assurance process, the experienced AEC officers were asked to independently ‘blind test’ the relevant ballot paper and to assess how the voter intent should be interpreted and recorded in BPRS. Where: all three AEC experience officers considered that the application of the AEC business rules would result in an interpretation of the ballot paper that differed to that which had been recorded in BPRS, an exception was recorded in the final assurance results; and in all other cases, the potential assurance exception was not recorded in the final assurance results. 1.33 Results from the AEC review of potential exceptions were assessed and collated by Axiom staff. REPORT CREATION – STATISTICAL ANALYSIS 1.34 All exceptions identified from the assurance process (including AEC review of potential exceptions) were collated at the state and territory level and at the national level by Axiom staff. Axiom followed the ABS’ guidance for calculating, analysing and reporting the statistical conclusions that may be drawn from the 2022 Senate external statistical assurance activities.7 The assurance results and conclusions have been expressed as state and territory and national 1.35 exception rates. As the ballot paper assurance approach uses a different sampling rate for each state and territory location, the national exception rate was derived using the weighted number of exceptions in each state and territory location relative to the state and territory proportion of the national vote for 2022. 1.36 Stage One exceptions were not attributed to individual state or territory locations. One element of the ABS methodology was based on the AEC assertion that scanning processes were homogenous across Australia and the methodology provided for a single national sample for the assessment of Stage One exceptions. 7 Australian Bureau of Statistics, ABS advice to AEC on sampling methodology, November 2021, p. 4. External statistical assurance of the Senate ballot papers 5 --- Page 8 --- Australian Electoral Commission | July 2022 2. FINDINGS, STATISTICAL ANALYSIS AND STATEMENT OF ASSURANCE OUTCOMES An analysis of the findings of the Senate ballot paper sampling process has been provided below. 2.1 Exceptions impacting on the statistical assurance results were considered in each stage of assurance testing. For Stage One of assurance testing, which compared the physical ballot paper to the scanned image transmitted to the AEC and included in BPRS, a total of seven exceptions were identified. These exceptions were not attributed to individual state or territory locations when calculating the statistical exception rate in order to align the analysis to the agreed methodology based on advice from the Australian Bureau of Statistics. The seven Stage One exceptions are described in Table 2. Table 2: Stage One exceptions identified during assurance testing CSS state and territory location NSW NT VIC Description of the exception Two physical ballot papers in a sampled batch could not be matched against scanned images in that batch. The scanned images in the batch included duplicate images for two other ballot papers in the batch. The scanned image of one physical ballot paper did not reflect the preferences recorded on the physical ballot paper. The physical ballot paper had a light ‘smudge’ over one preference. This ‘smudge’ did not obscure the number entered into the box on the physical ballot paper. The scanned image enhanced the ‘smudge’ to extent that it was not possible to distinguish the preference number on the scanned image of the ballot paper and consequently it was not recorded in BPRS. One physical ballot paper in a sampled batch could not be matched against scanned images in that batch. The batch contained 51 physical ballot papers and only 50 scanned images were associated with that batch. Three physical ballot papers in a sampled batch could not be matched against scanned images in that batch. Despite efforts to locate the associated images, these could not be found. 2.2 For Stage Two assurance testing, which compared the physical ballot paper to the electronic data used in counting and contained in BPRS, a total of 33 exceptions were identified. These exceptions related to the accuracy of the electronic preference data file (used in counting) and the interpretation of voter intent. Determination of voter intent should be in line with the AEC business rules as stated in the Ballot Paper Formality Policy and Ballot Paper Formality Guidelines. 2.3 The 33 exceptions have been classified in accordance with the approved assurance methodology. The classification of exceptions has been outlined in Table 3. An additional classification of exceptions has been included in the table below to reflect the assurance findings. The categorisation of statistical exceptions now includes where an incorrect preference type (including related preferences) was admitted to the count. External statistical assurance of the Senate ballot papers 6 --- Page 9 --- Australian Electoral Commission | July 2022 Table 3: Categorisation of exceptions related to the interpretation of voter intent Categorisation of exceptions used in statistical analysis Voter disenfranchised where a vote is recorded as informal when the voter expressed a valid preference(s) Above the line voter preference sequence was broken too early within the first six preferences Above the line voter preference sequence was broken too early after the first six preferences Below the line voter preference sequence was broken too early within the first twelve preferences Below the line voter preference sequence was broken too early after the first twelve preferences Voter preference sequence was broken too late (additional preferences were recorded) Vote recorded as formal when it should have been informal Incorrect preference type accepted Total exceptions related to the interpretation of voter intent CSS state and territory Location NSW VIC QLD WA SA TAS ACT NT Total % of total 2 1 2 2 1 1 2 2 2 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 2 7 8 2 3 5 5 2 1 9 8 3 3 1 3 5 1 33 27% 24% 9% 9% 3% 9% 15% 3% 100% 2.4 Of the exceptions used in the statistical analysis, 27% related to where a voter was disenfranchised from the election. Another 24% related to where the ATL voter preference sequence was broken too early within the first six preferences. 15% of exceptions related to the counting of an informal vote (i.e., where vote was informal and should have been excluded from the count). 2.5 Stage Two exceptions frequently arose because of the inconsistent application of the AEC’s business rules, by either Fujifilm or AEC officers. Each assurance exception reflects an assessment of the marks on a ballot paper that has been identified as being inconsistent with the AEC Ballot Paper Formality Policy and/or the AEC Ballot Paper Formality Guidelines. There was no suggestion of any political or logical bias in the exceptions. The effect of including the exceptions in the count would have resulted in, some increase and /or some reduction to some candidates in the number of preferences counted. 2.6 Similarly, the exceptions identified did not suggest any systematic basis for including or excluding preferences or ballot papers. Many exceptions were the result of the need for interpretation where voters did not follow numbering instructions (including x’s and ticks and words rather than numerals) or where numbering sequences for preferences were not clearly written on the ballot paper. 2.7 The statistical analysis of the results of the assurance testing was carried out in line with the ABS methodology. The AEC required the presentation of the statistical results to be in the form of a sample mean with a 95% confidence interval with upper and lower bounds. The proportion of exceptions identified in the sample tested was extrapolated to the population of Senate ballot papers used in the AEC Tally Room. Table 4 outlines the statistical analysis and results. The results were analysed to provide 95% confidence over the range of possible exceptions in the populations (CSS state and territory location and national).8 This analysis indicates that there is a 95% likelihood that the true number of exceptions within the populations is within the stated ranges in the table. 8 The confidence intervals for 95% confidence are intended to provide assurance that, all things being equal, if an additional 100 samples from the population were tested, that for 95 of the 100 samples, the sample the mean would lie in the stated range. External statistical assurance of the Senate ballot papers 7 --- Page 10 --- Australian Electoral Commission | July 2022 Table 4: Statistical analysis and results Statistical sampling characteristics CSS state and territory location Actual number of ballot papers a Location as a % of national total Stage 1 exceptions c NSW 0.473% VIC 0.641% QLD 0.222% WA 0.303% 0.471% SA 0.510% TAS 0.264% ACT 0.173% NT National 0.069% Totals 0.443% Note a - Actual number of ballot papers was sourced from AEC BPRS on 30 June 2022 (Ballot papers on Tally Room) Note b - Confidence interval calculations from https://www.abs.gov.au/websitedbs/d3310114.nsf/home/sample+size+calculator Note c - National (Stage 1) exceptions are attributed to each CSS location in line with their proportion of the national vote 4,996,110 3,960,958 3,111,034 1,571,899 1,162,472 372,973 290,308 106,907 15,572,661 15,572,661 32.1% 25.4% 20.0% 10.1% 7.5% 2.4% 1.9% 0.7% 100.0% 23,619 25,378 6,905 4,759 5,474 1,901 766 185 10,790 68,988 7 7 33 Assurance sample size 1,735 1,400 1,310 1,285 1,245 1,135 1,028 965 10,103 10,103 Stage 2 exceptions 7 8 2 3 5 5 2 1 Exception rate including national Stage 1 exceptions c Extrapolated number of exceptions AEC confidence level Upper exception rate as a % Statistical sampling outcomes b Upper limit: highest number of ballot papers that are 95% likely to have an exception Lower exception rate as a % Lower limit: lowest number of ballot papers that are 95% likely to have an exception 95% 95% 95% 95% 95% 95% 95% 95% 95% 0.796% 1.059% 0.477% 0.604% 0.851% 0.924% 0.577% 0.434% 0.572% 39,769 41,947 14,840 9,494 9,893 3,446 1,675 464 89,076 0.150% 0.223% 0.000% 0.002% 0.091% 0.096% 0.000% 0.000% 0.314% 7494 8833 0 31 1058 358 0 0 48,898 The analysis of the Senate assurance results indicates that there is a 0.45% exception rate in the assurance sample. 9 The AEC can be 95% confident that, 2.8 nationally, the number of exceptions in 1,000 ballot papers is between 3.210 and 5.8. This can be analysed at a CSS state and territory location level. These have been listed below: NSW – The AEC can be 95% confident that there are between 1.5 and 8 exceptions in 1,000 ballot papers. VIC – The AEC can be 95% confident that there are between 2.3 and 11 exceptions in 1,000 ballot papers. QLD – The AEC can be 95% confident that there are between 0 and 4.8 exceptions in 1,000 ballot papers. WA – The AEC can be 95% confident that there are between 0 and 6.1 exceptions in 1,000 ballot papers. SA – The AEC can be 95% confident that there are between 0.9 and 8.6 exceptions in 1,000 ballot papers. TAS – The AEC can be 95% confident that there are between 0.9 and 9.3 exceptions in 1,000 ballot papers. ACT – The AEC can be 95% confident that there are between 0 and 5.8 exceptions in 1,000 ballot papers. NT – The AEC can be 95% confident that there are between 0 and 4.4 exceptions in 1,000 ballot papers. 9 This can also be expressed as follows, for 99.55% of ballot papers in the sample there was matching data in the AEC systems for the count, in addition the AEC can be 95% confident that the true population value is between 99.42% (99.55%-0.129%) and 99.68%. (99.55%+0.129%), or 99.55% +/- 0.129%. 10 All numbers have been rounded to the next whole decimal point. External statistical assurance of the Senate ballot papers 8 --- Page 11 --- Australian Electoral Commission | July 2022 APPENDICES APPENDIX A: MARKS ON BALLOT PAPERS NOT RECORDED ACCURATELY IN BPRS WITH NO IMPACT ON THE COUNT The Stage Two assurance testing compared the physical ballot paper to the electronic data used in counting, contained in BPRS. The second stage testing assessed whether: the electronic preference data file was an accurate reflection of the preferences recorded on the physical ballot paper; and the interpretation of voter intent aligned with AEC business rules as stated in the Ballot Paper Formality Policy and Ballot Paper Formality Guidelines. The Stage Two assurance testing identified six instances where BPRS did not accurately record preferences. These findings are described in Table 1 and Table 2 below. In five of these cases there was no impact on formality or the recording of valid preferences. Table 1: Incorrect recording of preferences in BPRS – with impact on the accuracy of the preferences counted CSS state and territory location SA Description of exception Screen shot of the relevant section of the scanned image of the ballot paper (ballot paper reference number) A physical ballot paper identified a third preference in column B and a second preference in column C. Data Entry 2 identified the third preference as column C. BPRS did not accurately record these preferences. BPRS recorded a second preference in column B and a third preference in column C. This instance has been included as an exception in the statistical results. (27966SA-SPEN032517060160301001) Table 2: Incorrect recording of preferences in BPRS – with no impact on formality CSS state and territory location ACT Description of exception Screen shot of the relevant section of the scanned image of the ballot paper (ballot paper reference number) BPRS did not record a first preference recorded Below the Line (BTL) on the ballot paper. This had no impact on formality as there was not a complete sequence of six preferences recorded BTL. Above the Line (ATL) preferences were accurately recorded in BPRS for this ballot paper. (27966ACTCANB010100360046501005) External statistical assurance of the Senate ballot papers 9 --- Page 12 --- Australian Electoral Commission | July 2022 CSS state and territory location SA NT NT SA Description of exception Screen shot of the relevant section of the scanned image of the ballot paper (ballot paper reference number) A ballot paper had six crosses recorded above the line. BPRS recorded seven crosses. BPRS recorded a cross in column A where there was no cross recorded on the ballot paper. A ballot paper included nine number one preferences. BPRS recorded eight number one preferences seven preference. BPRS inconsistently recorded the numbers. Audit logs in BPRS indicate this ballot paper was reviewed by AEC. and one number A ballot paper included a range of ticks and crosses. No ticks or crosses were recorded in BPRS. Audit logs in BPRS indicate this ballot paper was reviewed by AEC. A ballot paper included a range of ticks BTL. Not all of these ticks BTL were recorded in BPRS. The ticks outside preference boxes were not recorded in BPRS. (27966SA-BOOT018203960019201001) (27966NT-LING030600310015101001) (27966NT-SOLO030701640024301005) (27966SA-STUR019015400078701039) The AEC may wish to look at the processes and controls over data entry and review by Fujifilm and AEC to ensure more thorough consideration of whether there is a match between the ballot paper and the BPRS data entry. External statistical assurance of the Senate ballot papers 10 --- Page 13 --- Australian Electoral Commission | July 2022 APPENDIX B: ABS ADVICE TO AEC ON SAMPLING METHODOLOGY Attached is a copy of the ABS advice to AEC on sampling methodology. External statistical assurance of the Senate ballot papers 11 --- Page 14 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 12 --- Page 15 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 13 --- Page 16 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 14 --- Page 17 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 15 --- Page 18 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 16 --- Page 19 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 17 --- Page 20 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 18 --- Page 21 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 19 --- Page 22 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 20 --- Page 23 --- Australian Electoral Commission | July 2022 External statistical assurance of the Senate ballot papers 21
The provided document is Axiom Associates' "Detailed report and statement of outcomes (draft)" from July 2022, directly fulfilling the FOI request for their audit report on the Australian Electoral Commission's (AEC) 2022 Senate scanning and digitisation accuracy.
Summary of Document Contents:
* Purpose: The report details the external statistical assurance process for Senate ballot papers, mandated by the Electoral Legislation Amendment (Assurance of Senate Counting) Act 2021. Its goal is to assure that electronic data used in counting accurately reflects physical ballot papers.
* Methodology:
* A national sample of 10,103 ballot papers was selected across all states and territories, based on advice from the Australian Bureau of Statistics (ABS).
* Assurance testing involved a two-stage approach:
* Stage 1: Comparing physical ballot papers to their scanned images for completeness and accuracy.
* Stage 2: Comparing physical ballot papers to the electronic preference data in the AEC's Ballot Paper Reconciliation System (BPRS), assessing accuracy and consistency with AEC's voter intent interpretation rules.
* All potential Stage 2 exceptions underwent a "blind test" review by three experienced AEC officers to confirm discrepancies.
* Findings and Outcomes:
* Stage 1 Exceptions: Seven instances were identified nationally where scanned images did not exactly replicate physical ballot papers (e.g., missing images, obscured marks, duplicate images).
* Stage 2 Exceptions: Thirty-three discrepancies were found between physical ballots and the electronic data/interpretations in BPRS. These often arose from inconsistent application of AEC rules by data entry staff or unclear voter markings.
* Categorization of Stage 2 exceptions included voter disenfranchisement (27%), above-the-line preference sequences broken too early (24%), and informal votes recorded as formal (15%).
* The report notes no political or systematic bias in the exceptions.
* Statistical Analysis: The national sample revealed an exception rate of 0.45%. With 95% confidence, the estimated number of exceptions nationally is between 3.2 and 5.8 per 1,000 ballot papers. State-specific confidence intervals are also provided.
Relevance to FOI Request:
This document is the core output of the audit sought by the FOI request. It provides comprehensive details on the methodology, findings, and statistical outcomes of Axiom Associates' external assurance process, directly addressing the accuracy of the AEC's 2022 Senate ballot paper scanning and digitisation.
22FE Senate collation of Stage 2 assurance exceptions_DRAFT.pdf (pdf)
Download file--- Page 1 --- External statistical assurance of the Senate ballot papers Collation of Stage 2 statistical assurance exceptions Australian Electoral Commission July 2022 --- Page 2 --- Australian Electoral Commission | July 2022 Contents 1. STATISTICAL ASSURANCE EXCEPTIONS ................................................................................................ 1 2. VOTER DISENFRANCHISED .................................................................................................................. 2 3. ATL PREFERENCE SEQUENCE BROKEN TOO EARLY WITHIN THE FIRST SIX PREFERENCES ........................ 6 4. ATL PREFERENCE SEQUENCE BROKEN TOO EARLY AFTER THE FIRST SIX PREFERENCES .......................... 9 5. BTL PREFERENCE SEQUENCE BROKEN TOO EARLY WITHIN THE FIRST TWELVE PREFERENCES .............. 10 6. BTL PREFERENCE SEQUENCE BROKEN TOO EARLY AFTER THE FIRST TWELVE PREFERENCES ................ 12 7. VOTER PREFERENCE SEQUENCE BROKEN TOO LATE ........................................................................... 13 8. VOTE RECORDED AS FORMAL WHEN IT SHOULD HAVE BEEN INFORMAL ............................................ 14 9. INCORRECT PREFERENCE TYPE ACCEPTED ......................................................................................... 16 Statement of Responsibility © Axiom Associates (Aust) Pty Ltd 2022. All Rights Reserved. Axiom Associates (Aust) Pty Ltd documents and reports may be relied upon by the Australian Electoral Commission (AEC) for purposes only pursuant to the terms of our engagement outlined in the Services Agreement Contract Number: SRC001297 dated December 2021 and any subsequent amendments. We disclaim all responsibility to any other party for any loss or liability that the other party may suffer or incur arising from or relating to or in any way connected with the contents of our documents or reports, the provision of our documents or reports to the other party or the reliance upon our documents or reports by the other party. Liability limited by a scheme approved under Professional Standards Legislation. Document Control Date Provided Detail 5 July 2022 Provided to the Australian Electoral Commission Version Number 1.0 1.1 1.2 1.3 External statistical assurance of the Senate ballot papers i --- Page 3 --- Australian Electoral Commission | July 2022 1. STATISTICAL ASSURANCE EXCEPTIONS 1.1 The purpose of this document is to provide a collation of Stage Two statistical assurance exceptions as part of the 2022 external statistical assurance of Senate ballot papers. This document should be read in conjunction with the External statistical assurance of the Senate ballot papers: Detailed report and statement of outcomes, dated July 2022. STAGE 2 EXCEPTIONS 1.2 The Stage Two assurance testing compared the physical ballot paper to the electronic data used in counting, contained in the Australian Electoral Commission’s (AEC) Ballot Paper Reconciliation System (BPRS). The second stage testing assessed whether: ▪ ▪ the electronic preference data file was an accurate reflection of the preferences recorded on the physical ballot paper; and the interpretation of voter intent aligned with AEC business rules as stated in the Ballot Paper Formality Policy and Ballot Paper Formality Guidelines. 1.3 For Stage Two assurance testing a total of 33 exceptions were identified. These exceptions related to the accuracy of the electronic preference data file (used in counting) and the interpretation of voter intent. Determination of voter intent should be in line with the AEC business rules as stated in the Ballot Paper Formality Policy and Ballot Paper Formality Guidelines. 1.4 The 33 exceptions have been determined and classified in accordance with the approved assurance methodology. The classification of exceptions is outlined in Table 1. An additional classification of exceptions has been included in the table below to reflect the assurance findings. The categorisation of statistical exceptions now includes where an incorrect preference type (including related preferences) was admitted to the count. Table 1: Categorisation of exceptions related to the interpretation of voter intent 1.5 Of the exceptions used in the statistical analysis, 27% related to where a voter was disenfranchised from the election. Another 24% related to where the ATL voter preference sequence was broken too early within the first six preferences. 15% of exceptions related to the counting of an informal vote (i.e., where vote was informal and should have been excluded from the count). External statistical assurance of the Senate ballot papers 1 Categorisation of exceptions used in statistical analysisNSWVICQLDWASATASACTNTTotal% of totalVoterdisenfranchisedwhereavoteisrecordedasinformalwhenthevoterexpressed a valid preference(s)2211129 27%Abovethelinevoterpreferencesequencewasbrokentooearlywithinthefirst six preferences221128 24%Abovethelinevoterpreferencesequencewasbrokentooearlyafterthefirst six preferences1113 9%Belowthelinevoterpreferencesequencewasbrokentooearlywithinthefirst twelve preferences1113 9%Belowthelinevoterpreferencesequencewasbrokentooearlyafterthefirst twelve preferences11 3%Voterpreferencesequencewasbrokentoolate(additionalpreferenceswere recorded)1113 9%Vote recorded as formal when it should have been informal1225 15%Incorrect preference type accepted11 3%Total exceptions related to the interpretation of voter intent7 8 2 3 5 5 2 1 33 100%CSS state and territory Location --- Page 4 --- Australian Electoral Commission | July 2022 2. VOTER DISENFRANCHISED A voter is disenfranchised when a vote is recorded as informal when the voter expressed a valid 2.1 preference(s). NSW BPRS did not recognise a number 1 in column W Above the Line (ATL) and recorded the vote as informal. 2.2 The vote should have been formal ATL admitted to 6 preferences. Ballot paper: 27966NSWHUGH0124320000097_01_001 BPRS did not recognise a number 1 in column G ATL and recoded the vote as informal. The vote should 2.3 have been formal ATL admitted to 1 preference. Ballot paper: 27966NSWSHOR0148651700109_01_004 VIC 2.4 The voter recorded six preferences ATL on the ballot paper (columns: L, P, Q, V, W and X). They also marked all other boxes with an ‘x’. BPRS rejected the vote. The vote should have been formal ATL admitted to 6 preferences. Ballot paper: 27966VICFRAS0321538200305_01_004 External statistical assurance of the Senate ballot papers 2 --- Page 5 --- Australian Electoral Commission | July 2022 2.5 The voter recorded six preferences ATL on the ballot paper (columns: C, D, H, K, L, P and W). They also marked all other boxes with an ‘x’. BPRS rejected the vote. The vote should have been formal ATL admitted to 7 preferences. Ballot paper: 27966VICMONA0323566200401_01_002 QLD 2.6 The voter recorded twelve preferences Below The Line (BTL) on the ballot paper (columns: L, P, Q, V, W and X). BPRS did not record the first preference in column A. BPRS rejected the vote. The vote should have been formal BTL admitted to 12 preferences. Ballot paper: 27966QLDLILL0169227400110_01_001 WA 2.7 The voter recorded an unbroken sequence of 9 preferences BTL on the ballot paper (columns: B, D, L, O, R, T, U). BPRS recorded preference 9 in column D as a 4. BPRS rejected the vote. The vote should have been formal BTL admitted to 9 preferences. Ballot paper: 27966WA-PEAR0244135200105_01_001 External statistical assurance of the Senate ballot papers 3 --- Page 6 --- Australian Electoral Commission | July 2022 TAS 2.8 The voter recorded 11 preferences ATL on the ballot paper (columns: A, B, C, D, E, G, I, J, K, M and N). BPRS recorded the preference in column A as a 1 instead of an 11. BPRS rejected the vote. The vote should have been formal ATL admitted to 11 preferences. Ballot paper: 27966TASBRAD0193023900501_01_002 ACT 2.9 The voter recorded a first preference ATL in column D on the ballot paper. The voter crossed off all other boxes (except for column A where a 6 was recorded). These crossed out boxes were recorded as / in BPRS (signalling a first preference tick). BPRS rejected the vote. The vote should have been formal ATL admitted to 1 preference. Ballot paper: 27966ACTCANB0101002800103_01_005 External statistical assurance of the Senate ballot papers 4 --- Page 7 --- Australian Electoral Commission | July 2022 2.10 The voter recorded six preferences ATL (columns: A, B, C, D, H and K). The first preference in column D was recorded as a 7 in BPRS. BPRS rejected the vote. The vote should have been formal ATL admitted to 6 preferences. Ballot paper: 27966ACTBEAN0318035400236_01_001 External statistical assurance of the Senate ballot papers 5 --- Page 8 --- Australian Electoral Commission | July 2022 3. ATL PREFERENCE SEQUENCE BROKEN TOO EARLY WITHIN THE FIRST SIX PREFERENCES NSW BPRS did recognise a number 3 in column E ATL. BPRS recorded the vote as admitted to 2 preferences. 3.1 The vote should have been formal ATL admitted to 6 preferences. Ballot paper: 27966NSWKSMI0127364301401_01_001 BPRS recorded a 4 in column S ATL when it was a 7. BPRS recorded the vote as admitted to 3 3.2 preferences. The vote should have been formal ATL admitted to 7 preferences. Ballot paper: 27966NSWGREE0122305900015_01_005 VIC BPRS recorded an 8 in column H ATL instead of a 3. BPRS recorded the vote as admitted to 2 3.3 preferences. The vote should have been formal ATL admitted to 6 preferences. Ballot paper: 27966VICLTRO0223351600882_01_006 External statistical assurance of the Senate ballot papers 6 --- Page 9 --- Australian Electoral Commission | July 2022 BPRS did not recognise a number 3 in column I ATL. BPRS recorded the vote as admitted to 2 3.4 preferences. The vote should have been formal ATL admitted to 3 preferences. Ballot paper: 27966VICCASE0204080600154_01_004 WA BPRS recorded a 3 in column U ATL instead of a 5. BPRS recorded the vote as admitted to 2 preferences. 3.5 The vote should have been formal ATL admitted to 12 preferences. Ballot paper: 27966WA-SWAN0247169100018_01_003 SA 3.6 BPRS did not accurately record the preferences on the ballot paper. BPRS recorded a 2 in column B and 3 in column C. This did not match the ballot paper where a 3 was in column B and a 2 in column C. BPRS recorded the vote as admitted to 6 preferences. The vote should have been formal ATL admitted to 6 for different preferences. Ballot paper: 27966SA-SPEN0325170601603_01_001 External statistical assurance of the Senate ballot papers 7 --- Page 10 --- Australian Electoral Commission | July 2022 TAS BPRS recorded a 6 in column K ATL instead of a 9. BPRS recorded the vote as admitted to 5 preferences. 3.7 The vote should have been formal ATL admitted to 13 preferences. Ballot paper: 27966TASBRAD0193023500202_01_002 BPRS did not record a 6 in column C ATL. BPRS recorded the vote as admitted to 5 preferences. The 3.8 vote should have been formal ATL admitted to 12 preferences. Ballot paper: 27966TASLYON0196052400106_01_004 External statistical assurance of the Senate ballot papers 8 --- Page 11 --- Australian Electoral Commission | July 2022 4. ATL PREFERENCE SEQUENCE BROKEN TOO EARLY AFTER THE FIRST SIX PREFERENCES VIC BPRS recorded a 12 in column H ATL instead of a 17. BPRS recorded the vote as admitted to 11 4.1 preferences. The vote should have been formal ATL admitted to 22 preferences. Ballot paper: 27966VICNICH0324582101812_01_002 QLD BPRS recorded a 16 in column V ATL instead of a 10. BPRS recorded the vote as admitted to 9 4.2 preferences. The vote should have been formal ATL admitted to 12 preferences. Ballot paper: 27966QLDBLAI0304420800117_01_005 SA BPRS recorded a 19 in column Q ATL instead of a 17. BPRS recorded the vote as admitted to 16 4.3 preferences. The vote should have been formal ATL admitted to 19 preferences. Ballot paper: 27966SA-BARK0180027901502_01_003 External statistical assurance of the Senate ballot papers 9 --- Page 12 --- Australian Electoral Commission | July 2022 5. BTL PREFERENCE SEQUENCE BROKEN TOO EARLY WITHIN THE FIRST TWELVE PREFERENCES VIC BPRS recorded a 1 in column S BTL instead of an 11. This made the ballot paper BTL informal. BPRS then 5.1 admitted ATL preferences to 6. The vote should have been BTL formal admitted to 12 preferences. Ballot paper: 27966VICCHIS0205095700375_01_003 WA BPRS did not record an 8 in column P BTL. BPRS admitted BTL preferences to 7. The vote should have 5.2 been BTL formal admitted to 12 preferences. Ballot paper: 27966WA-COWA0237036600204_01_003 External statistical assurance of the Senate ballot papers 10 --- Page 13 --- Australian Electoral Commission | July 2022 NT BPRS did not record a 7 in column B BTL. BPRS admitted BTL preferences to 6. The vote should have 5.3 been BTL formal admitted to 10 preferences. Ballot paper: 27966NT-LING0306001800108_01_004 External statistical assurance of the Senate ballot papers 11 --- Page 14 --- Australian Electoral Commission | July 2022 6. BTL PREFERENCE SEQUENCE BROKEN TOO EARLY AFTER THE FIRST TWELVE PREFERENCES TAS BPRS recorded a 27 in column K BTL instead of a 17. BPRS admitted BTL preferences to 16. The vote 6.1 should have been BTL formal admitted to 39 preferences. Ballot paper: 27966TASLYON0196058500119_01_002 External statistical assurance of the Senate ballot papers 12 --- Page 15 --- Australian Electoral Commission | July 2022 7. VOTER PREFERENCE SEQUENCE BROKEN TOO LATE NSW BPRS recorded a 7 in column M ATL instead of a 2. BPRS admitted ATL preferences to 7. The vote should 7.1 have been formal admitted to 1 preference. Ballot paper: 27966NSWCOOK0112135000098_01_002 SA BPRS recorded a 4 in column Q and 5 in column R ATL which had been crossed out. BPRS admitted ATL 7.2 preferences to 10. The vote should have been formal admitted to 3 preferences. Ballot paper: 27966SA-BOOT0182040400292_01_003 TAS BPRS recorded an unusual mark as a 7 in column L. BPRS admitted ATL preferences to 7. The vote should 7.3 have been formal admitted to 6 preferences. Ballot paper: 27966TASBASS0192001400107_01_001 External statistical assurance of the Senate ballot papers 13 --- Page 16 --- Australian Electoral Commission | July 2022 8. VOTE RECORDED AS FORMAL WHEN IT SHOULD HAVE BEEN INFORMAL NSW BPRS recorded a 7 in column M ATL as a 1. BPRS admitted ATL preferences to 2. The vote should have 8.1 been informal and rejected from the count. Ballot paper: 27966NSWLIND0128379500216_01_005 VIC BPRS recorded an unusual mark in column A ATL as a 1. BPRS admitted ATL preferences to 1. The vote 8.2 should have been informal and rejected from the count. Ballot paper: 27966VICMELB0228417500315_01_005 BPRS recorded an unusual mark in column G ATL as a 1. BPRS admitted ATL preferences to 6. The vote 8.3 should have been informal and rejected from the count. Ballot paper: 27966VICSCUL0232447300204_01_003 External statistical assurance of the Senate ballot papers 14 --- Page 17 --- Australian Electoral Commission | July 2022 SA BPRS recorded an unusual mark in column B ATL as a 1. BPRS admitted ATL preferences to 1. The vote 8.4 should have been informal and rejected from the count. Ballot paper: 27966SA-GREY0183069601108_01_005 BPRS recorded an unusual mark in column A ATL as a 1. BPRS admitted ATL preferences to 1. The vote 8.5 should have been informal and rejected from the count. Ballot paper: 27966SA-KING0186097800275_01_004 External statistical assurance of the Senate ballot papers 15 --- Page 18 --- Australian Electoral Commission | July 2022 9. INCORRECT PREFERENCE TYPE ACCEPTED NSW 9.1 BPRS recorded unusual marks as BTL preferences in column E. BPRS admitted BTL preferences to 7. The vote should have been admitted ATL formal to 7 preferences (different set of preferences to those recorded BTL). Ballot paper: 27966NSWNENG0135486100247_01_003 External statistical assurance of the Senate ballot papers 16
This document, titled "Collation of Stage 2 statistical assurance exceptions," is part of Axiom Associates' external statistical assurance of the Australian Electoral Commission's (AEC) 2022 Senate ballot papers, directly relevant to the FOI request seeking the audit report on scanning and digitisation accuracy.
It serves as a detailed companion to the main audit report, specifically outlining the 33 Stage 2 exceptions identified. Stage 2 testing assessed the accuracy of the electronic preference data file in the AEC's Ballot Paper Reconciliation System (BPRS) against physical ballot papers and the alignment of voter intent interpretation with AEC business rules.
The document categorises and provides visual examples for each exception type, which include:
* Voter disenfranchisement (9 instances, 27%): Votes recorded as informal despite valid preferences.
* Above The Line (ATL) preference sequence broken too early (8 instances within first six preferences, 3 instances after six preferences).
* Below The Line (BTL) preference sequence broken too early (3 instances within first twelve preferences, 1 instance after twelve preferences).
* Voter preference sequence broken too late (3 instances): Additional preferences incorrectly recorded.
* Vote recorded as formal when it should have been informal (5 instances, 15%).
* Incorrect preference type accepted (1 instance).
The collation provides specific ballot paper IDs and visual evidence for each discrepancy, illustrating issues such as BPRS misinterpreting voter markings, failing to recognise preferences, or incorrectly sequencing preferences, aligning with the FOI request's overview that exceptions often arose from interpreting voter intent.