“Ban the Scan” Campaign Underscores Need to Conduct Thorough Review of Facial Recognition Software

One area of artificial intelligence development that has been at the center of AI ethical and efficacy debates for years is facial recognition systems. Studies by the National Institute of Standards and Technology report significant improvements in the performance of facial recognition software between 2010 and 2018. The studies examined how successfully algorithms were able to match a person’s photo with a different one of the same person stored in a large database. In 2010, 5.0% of the algorithms failed the test; in 2018, just 0.2% of the algorithms failed.

However, NIST’s reports have also revealed concerns. The conclusions from one evaluation of 189 software algorithms from 99 developers raised several red flags:

  • “For one-to-one matching [i.e., confirming a photo matches a different photo of the same person in a database], the team saw higher rates of false positives for Asian and African American faces relative to images of Caucasians. The differentials often ranged from a factor of 10 to 100 times, depending on the individual algorithm.”
  •  “For one-to-many matching [i.e., determining whether the person in the photo has any match in a database], the team saw higher rates of false positives for African American females.”

The effects of these biases range from inconvenience (such as a false negative in a one-to-one search preventing someone from unlocking his or her phone on the first attempt) to inappropriate law enforcement attention (such as a false negative in a one-to-many search of an FBI database leading to the administrator of the system labeling the individual as warranting further scrutiny).

Concerns like these are behind Amnesty International’s recently announced “Ban the Scan” campaign in New York City. It warns that “[f]acial recognition technology can amplify racially discriminatory policing” and that “Black and minority communities are at risk of being misidentified and falsely arrested – in some instances, facial recognition has been 95% inaccurate.” The organization is asking New York residents to contact the New York Police Department and the New York City Council about banning facial recognition technology.

The NIST studies point out that the source of the software matters. Some developers have created better algorithms than others and that “those that are the most equitable also rank among the most accurate.” Any organization considering facial recognition software should carefully vet the systems. Does the provider guaranty a particular failure rate? Is that rate satisfactory? Can the provider produce assessment results, showing that the AI’s results have been reviewed for accuracy, bias, privacy, and other concerns? If relevant, does the provider have appropriate insurance? Does the software comply with the relevant statutes governing facial recognition software, like the Illinois Biometric Information Privacy Act, the California Consumer Privacy Act, and California Privacy Rights Act?

Facial recognition software can be useful, but also controversial and potentially deeply troublesome. Entities that are committed to adopting it can minimize potential legal, bias, and public relations problems by performing a thorough due diligence review of the systems under consideration.

John Weaver
John Weaver

As an emerging technologies lawyer, John advises a wide range of companies – from startups to international corporations – on regulatory and legal issues unique to those technologies, including consumer protection requirements governing artificial intelligence, regulations governing drones, state legislation affecting self-driving cars, and the impact of autonomous devices and programs on user and employment agreements.

Artificial Intelligence Regulation Under the California Privacy Rights Act

On November 3, 2020, while the rest of the country voted for president, California residents voted to adopt the most sweeping privacy law in the nation, the California Privacy Rights Act (“CPRA”). The new law is principally concerned with privacy rights and protections, expanding those created by the California Consumer Privacy Act (“CCPA”), the 2018 law that became effective this year. However, the CPRA also creates a new agency, the California Privacy Protection Agency (“CPPA”), and sometime in the middle of 2021, the California Attorney General’s office will assign its regulatory authority under the CCPA and CPRA to that agency. The CPRA instructs the CPPA to issue “regulations governing access and opt-out rights with respect to businesses’ use of automated decision-making technology, including profiling and requiring businesses’ response to access requests to include meaningful information about the logic involved in such decision-making processes, as well as a description of the likely outcome of the process with respect to the consumer” (emphasis added).

Continue reading