aec-membership-test-simulator
Request for Review of Decision to Deregister Flux
To: Mr Tom Rogers, Australian Electoral Commissioner
CC: The other members of the Commission: Justice Kenny, Chairperson, and Dr Gruen, the Australian Statistician.
This is a request for the Commission to review the decision, made on the 24th of March 2022, to deregister “VOTEFLUX.ORG | Upgrade Democracy!” (Flux) under s137(6) of the Electoral Act. It is made under s141(2) of the Electoral Act.
I am making this request because the AEC has been using and continues to use a faulty method to test party eligibility with regards to sufficient membership. Thus, I am dissatisfied with the decision to deregister Flux. Additionally, as an Australian elector, a fundamental oversight in the AEC’s policy could compromise the democracy that I am a constituent of, so I am affected, along with all other Australians. As part of this request, I provide statistical analysis as grounds that the AEC’s testing method is flawed.
My Statement to the Commission
I invite the Commission – Mr Rogers along with Justice Kenny and Dr Gruen – to consider the following hypothetical case regarding the AEC’s method for validating that a party meets the requirements of the Electoral Act. I assume that you all are familiar with the AEC’s testing methodology.
Consider a party with 9,000 members, and let’s say that half of those members (4,500) are validatable as electors, and a further 60% of those members (2,700) will respond “yes” to an AEC request for membership confirmation. It is natural for a party to have members that can not be validated against the electoral roll (which can happen for a variety of reasons; a member’s status of silent elector is one). Thus, parties go to some substantial effort to submit only those members that can be validated against the roll – for reasons that I hope are obvious to you who comprise the Commission.
That is, in our hypothetical case: 4,500 members are not validatable as electors, and a further 1,800 would deny membership if asked by the AEC. The remaining 2,700 are legitimate.
Justice Kenny, as I’m sure you’re aware, the Electoral Act (EA) specifies that an eligible party requires at least 1,500 members. In your legal opinion, would a party with 2,700 members satisfy that clause of the EA?
I hope that you agree that it would. I anticipate that you would also agree that, all else being equal, this hypothetical party appears eligible under the EA. At the very least, we do not have a reason to conclude that the party is ineligible, right?
Dr Gruen, I wonder if you are a man dedicated to facts and truth or falsehoods and political agendas. I’m sure that, as an expert statistician, you hold mathematical facts above unsubstantiated claims. I’m also sure that you appreciate that if a statistical test has a predetermined outcome, then that test is neither reliable nor suitable for any real-world purpose. If you had a blood sample processed by a doctor, would you accept the results if that doctor gave them to you before the blood sample was taken? No, of course not, that would be crazy.
So, Dr Gruen, please consider our hypothetical case. Given that this hypothetical party is eligible under the EA, what should we expect as the results of the AEC’s testing method as applied to this party? Given that this party cannot determine which of the 4,500 validatable members will respond “yes” or “no”, and that this party can submit no more than 1,650 members due to AEC policy alone: the best that can be done is selecting a subset of those members, essentially at random. There may be some small optimizations the party could make, but in principle the limiting factors are those that we have already discussed. Thus each member on the submitted list is expected to respond to the AEC with a membership denial with a probability of 0.4 (40%). Of course, we know this because 1,800 / (2,700 + 1,800) = 0.4, and because there is no reason that the ratio of denying members to confirming members would change substantially (outside the statistical variance of the selection of 1,650 members, which is not very substantial anyway).
Let us assume that this hypothetical party randomly selects 1,650 validatable members from its pool of 4,500. Dr Gruen, I hope that you find this a reasonable course of action for the party to take and have followed the logic thus far.
Dr Gruen, would you agree that the accuracy of a statistical method is roughly: how often it results in true positives and true negatives? That is: if a method results in false positives and false negatives some of the time, it cannot have 100% accuracy. If, for some cases, it only produces false positives and false negatives, then it has 0% accuracy for those cases. If there exists some case where the method will always result in false negatives, then we can conclude that such a method is, at least sometimes, inaccurate, yes?
I invite you, Dr Gruen, to please calculate the probability that this hypothetical party passes the AEC’s membership testing methodology – that the party passes the test that is endorsed by your bureau. You may assume that no members are filtered out, i.e., that during the AEC’s validation of the membership list no electors are excluded for being duplicates, or deceased, or unmatchable against the electoral roll, etc.
All the information required for such a calculation is specified above. I assume that such a calculation is trivial for a statistician such as yourself, with the resources available to you. Surely, you agree that it is fairly straight forward to calculate, yes?
Mr Rogers, while Dr Gruen is calculating that probability, let us discuss something that you wrote recently:
The AEC’s values of electoral integrity through agility, professionalism and quality underpin everything we do […]
I also believe that integrity is important. Integrity is necessary for a system (or a person) to remain robust. Integrity is a major difference between ‘stable and enduring’, and ‘compromised and corrupted’. When integrity fails, good systems become rotten. Do you not agree?
It is to your integrity, and the integrity of Justice Kenny and Dr Gruen, too, that I make this appeal. I am fully aware that you are free to decide whatever you wish – the Electoral Act is written such that you are practically unconstrained in this matter.
I ask you this: if the AEC and the ABS were wrong about the AEC’s testing methodology, how would you know? Surely you realize that it is possible that the AEC is mistaken somehow – that we are all fallible, and our ideas are fallible? Institutions like the AEC, and methods like the one the AEC use, are based on ideas that were thought-up by people, and are therefore fallible.
Justice Kenny and Dr Gruen, do you also realize this? That there are no infallible humans, no infallible institutions, and no infallible ideas?
Mr Rogers, if it is possible to be wrong, what does it mean to have integrity in the face of potentially being wrong, and thus potentially making progress? Should one take the path of honesty and truth-seeking, or should one take the path of enforcing authority through baseless claims, evasion, and dismissal? If you have some other path to take, is it not one of honesty and truth-seeking? Surely you would not advocate a path that is dishonest and avoids the truth? Perhaps Justice Kenny has some insight on this; I understand that judges often have some experience in these matters.
Now, let us return to the matter of the accuracy of the AEC’s testing methodology. Dr Gruen, are you done calculating those probabilities? Let us compare answers.
Here are my results with 95% confidence intervals: the probability that the hypothetical party passes the AEC’s testing method is 0.0020% ± 0.0012%. The mean membership denials via the AEC’s test is = 23.998 ± 0.010 (out of 60 successful contacts). The standard deviation of that distribution is = 3.765 ± 0.007. The SEM for these results is = 0.005.
Dr Gruen, would you consider a statistical method accurate if the probability of it producing a false negative is 99.9980% ± 0.0012%? Would you consider that, for all intents and purposes, such a statistical method is predetermined in its outcome?
Justice Kenny, do you consider it appropriate for an important legal institution to use a method that, in some conditions, fails more than 99% of the time? What would our society be like if the courts had this kind of failure rate for certain types of cases? How do we know that they don’t? How does the AEC know that its method doesn’t? Why are the answers to those questions different?
Mr Rogers, do you think that a test which, in some cases has a failure rate greater than 99%, should be used by a leader who values integrity and quality? Do you think that a test which, when applied to certain cases, succeeds only 0.002% of the time is “rational, fair and practical in all the circumstances”? [1]
Dr Gruen, do you think that a test which, for certain cases, is less than 1% accurate is “rational, fair and practical in all the circumstances”? After all, it is a fact that the AEC’s method performs this poorly some of the time. You have calculated so yourself. (You have done the calculations, haven’t you?)
Justice Kenny, do you think that, when a citizen appeals to a judicial process of their country’s legal institutions, they deserve the right to be taken seriously? That they deserve the right to present their case and, if it is supported by the facts, to have an injustice undone? Is that not one of the primary values of judicial processes and the courts? To safeguard citizenry from injustices? To undo decisions that would otherwise be mistakes? Is that not something that the integrity of our legal institutions depend on? Is the AEC not part of the bedrock of our legal system? (Its foundational role being that it provides the system by which parliamentarians, who alone can create and modify legislation, are elected.)
If a citizen petitions an institution via a judicial process, and proves their case with mathematics and evidence, what are they to do if that is not enough? If that institution is unconvinced by facts and evidence, how can a citizen have confidence in their legal system? How can they have confidence in that foundational component of our society?
Moreover, if judicial processes ignore facts and evidence, are those processes serving one of their major purposes – error correction? Is that not a core goal of judicial processes: righting wrongs, undoing mistakes, preventing mistakes, promoting justice, and so on? What is a citizen to do when a judicial process ignores facts and evidence?
Mr Rogers, I cannot force you to change your mind about anything, that is solely up to you.
I can present you with this case, though. Perhaps you (along with Justice Kenny and Dr Gruen) have guessed that the hypothetical party was not so hypothetical after all. The parameters of that case are spitting distance from those of Flux’s second membership test (conducted Feb/March 2022). There is at least one major difference – in the hypothetical case, the party selected members randomly; in Flux’s case, your delegate selected 1,650 members from the top of the list of 4,680 members that Flux submitted. That list was alphabetical, so the AEC only sampled from members whose first name started with one of A through G.
Dr Gruen, in your expert opinion and in light of the above, and given that Flux failed this test, which of these should we believe true?
- That Flux has fewer than 1,500 members; or
- That Flux has fewer than 1,500 members whose first name starts with one of A through G.
If a party has 1,000 (~1586 * 29/(17+29)) members whose first name starts with one of A through G, how many members whose name starts with one of H through Z do we expect that party to have? After all, what are the chances that a party only has members with first names that start with a letter between A and G inclusive.
The point of this argument is not that Flux has sufficient members to satisfy the electoral act. The point of this argument is that the AEC’s method is flawed. It is inaccurate, it is unreliable, it is unfair, and it is unsuitable – at least in this case. The facts prove this.
I am not asking that the Commission believe me, I am asking the Commission to believe facts and evidence.
Justice Kenny, I will ask you again, does a citizen deserve the right to present their case via a judicial process and, if it is supported by the facts, to have an injustice undone? I hope that your answer is the same in this case as it would have been for all other cases that you have presided over as a Judge of the Court.
The above is sufficient to conclude that Flux was wrongfully deregistered. It was wrongful because, regardless of whether Flux should be deregistered or not, the AEC’s method is not good enough to produce an accurate result in this case. From a truth-seeking point of view, the AEC’s test provides no meaningful information on whether Flux is eligible or not. Since we should not take my word for it, if our goal is to determine whether Flux is eligible with regards to sufficient membership, we are at square one. Scientifically, we can draw no conclusions based on the AEC’s testing. It is not my place to tell you what should be done instead of the current method, I can only demonstrate to you that the outcome is predetermined in Flux’s case, as the test results in a false negative with probability 0.999978 ± 0.000013. (These numbers differ slightly from those in the hypothetical case above because the calculations are based, specifically, on Flux’s second test.)
Mr Rogers, Justice Kenny, and Dr Gruen: perhaps this is not enough to convince that there is a reasonable chance of some problems with the AEC’s methodology. The only other thing I can do is present you with a more in-depth statistical analysis of the method – covering this problem and others, both historical and on-going. I suppose that it is most relevant for Dr Gruen, since he is the only one I can reasonably expect to have the knowledge of statistics necessary to judge the analysis. To my knowledge, this is the first third party review of the AEC’s methodology. If there has been a third party review that is not public, I think that it is reasonable for the Commission to provide it (without an FOI request).
After I sign off, you will find supporting graphs of the Probability Mass Functions comprising the statistical analysis from which my above results were drawn. Following that, the in-depth review of the method. Please consider these as part of this submission.
Mr Rogers, there is only one thing that I ask of you – besides that this matter is taken seriously. If the Commission resolves to affirm the decision to deregister Flux under s137(6) of the EA, please tell me: if this was not enough to convince you that there are problems with the testing method, what would have? Is there anything that would have convinced you?
If that case comes to bear, then I have a request for Dr Gruen, too: if you disagree with my statistical analysis, meet me on a level playing field by providing your calculations, as I am providing you mine [2]. Perhaps Justice Kenny can assist the Commission in determining if this is a fair request.
Mr Rogers, I leave it to you as the accountable authority of the Australian Electoral Commission, along with the other members of the Commission: Justice Kenny and Dr Gruen, to uphold “[the] AEC’s values of electoral integrity through agility, professionalism and quality […]”.
I await your review,
Max Kaye
Supporting Material: PMFs of Distributions in this Request for Review
Fig S.1: A hypothetical case (covered earlier in this submission) that shows the Probability Mass Function of the results of the AEC's testing method given: 4,500 members that an eligible party may select from; no members filtered out by the AEC; and a membership denial rate of 40%. An assumption of this statistical case is that the party is eligible under the EA. This therefore constitutes a case where the AEC's method is completely inaccurate.
Fig S.2: Modelling based on the AEC results from Flux's membership test in Feb/March 2022. Even if we assume that Flux is eligible, the AEC's method returns a false negative more than 99.99% of the time.
Fig S.3: Modelling of an example where the AEC's method works.
That is: it is accurate in this case.