Meta, has released a new AI benchmark called FACET

News
August 31, 2023

Meta Releases New AI Benchmark to Evaluate Fairness of Computer Vision Models

Meta, the parent company of Facebook, has released a new AI benchmark called FACET to evaluate the fairness of computer vision models. FACET stands for “FAirness in Computer Vision EvaluaTion.”

FACET includes 32,000 images with 50,000 people labeled by human annotators and accounts for various demographic and physical attributes, allowing for deep evaluations of biases and classes (e.g., “basketball player,” “disc jockey,” and “doctor”). The images were sourced from a variety of sources, including public datasets and commercial photo providers.

Meta’s goal with FACET is to enable researchers and practitioners to benchmark fairness in their own models and monitor the impact of mitigations put in place to address fairness concerns. The benchmark aims to provide a deeper evaluation of biases against different classes, allowing for a more comprehensive understanding of disparities present in computer vision models.

While there have been previous benchmarks to probe for biases in computer vision algorithms, Meta claims that FACET is more thorough and can answer more nuanced questions regarding biases. It can evaluate whether models are better at classifying people based on certain attributes, such as gender presentation or hair type.

To create FACET, Meta had a team of annotators label each image for demographic attributes, physical attributes, and classes. The annotators were sourced from various geographic regions and were compensated with an hourly wage set per country. These annotations were then combined with labels from Meta’s “Segment Anything 1 Billion” dataset, which focuses on training AI models to identify and segment objects and animals within images.

Despite potential concerns about the origins of the data set, Meta believes that FACET can be used to probe various computer vision models for biases across different demographic attributes. As a test case, Meta applied FACET to its own DINOv2 computer vision algorithm and uncovered biases in gender presentations and stereotypical identifications.

It’s worth noting that Meta has faced criticism in the past for its responsible AI practices. However, Meta claims that FACET is a significant improvement over previous benchmarks and can be used to probe different demographic attributes in computer vision models.

Meta wrote in the blog post. “We plan to address these potential shortcomings in future work and believe that image-based curation could also help avoid the perpetuation of potential biases arising from the use of search engines or text supervision.”

“At this time we do not plan to have updates for this data set,” Meta writes in the whitepaper. “We will allow users to flag any images that may be objectionable content, and remove objectionable content if found.”

Meta has also released a web-based data set explorer tool that allows users to explore the FACET data set and evaluate their own computer vision models.

The release of FACET is a significant step forward in the effort to build fair and equitable AI systems. It provides researchers and developers with a valuable tool for understanding and addressing the biases that may exist in their models.

Potential Problems with FACET

While FACET is a valuable tool, it is important to be aware of some of its potential problems.

First, the benchmark is based on a limited dataset of images. This means that it may not be representative of the real world, and it may not be able to detect all biases in computer vision models.

Second, the benchmark is designed to be used with computer vision models that have been trained on a variety of different datasets. This means that it may not be appropriate for all models.

Finally, the benchmark is still under development, and it is possible that there are other problems that have not yet been identified.

Here are some additional thoughts on FACET:

It is encouraging that Meta is taking steps to address the issue of bias in AI. FACET is a significant contribution to this effort.

However, it is important to remember that no single benchmark can be used to fully evaluate the fairness of a computer vision model. It is important to use multiple benchmarks and to consider other factors, such as the model’s training data and its intended use.

It is also important to note that bias is a complex issue, and there is no easy solution. FACET is a valuable tool, but it is just one part of the solution. We need to continue to research and develop new ways to address bias in AI.

Conclusion

Despite its potential problems, FACET is a valuable tool for evaluating the fairness of computer vision models. It is important to be aware of its limitations, but it is a step in the right direction for building fair and equitable AI systems.

While benchmarks for probing biases in computer vision algorithms are not new, Meta claims that FACET is more thorough than previous benchmarks. It aims to answer questions like whether models classify people based on stereotypical gender attributes or whether biases are magnified based on physical attributes.

What do you think about Meta’s new AI benchmark, FACET? Do you think it is a valuable tool for evaluating the fairness of computer vision models? Share your thoughts below!