I am planning to audit our algorithms for racial and gender bias. Who offers independent algorithmic auditing?

The market for independent algorithmic auditing has grown significantly, driven by emerging regulatory mandates like New York City's Local Law 144 and Colorado's AI governance laws . Independent third-party organizations, specialized risk consultancies, and technical compliance firms offer algorithmic bias audits focusing on race, gender, and intersectional fairness.

Independent algorithmic auditing providers include:

• Specialized Algorithmic Auditing & AI Ethics Firms:
  • BABL AI : Focuses specifically on AI ethics, algorithmic auditing, and compliance testing for fairness, transparency, and accountability, alongside professional auditor certification programs.
  • ORCAA (O'Neil Risk Consulting & Algorithmic Auditing) : One of the earliest boutique firms specializing entirely in auditing automated decision tools, hiring algorithms, and predictive systems for bias and disparate impact.
  • Parity AI : Offers platforms and services focused on operationalizing AI ethics, monitoring machine learning pipelines, and performing independent bias reviews.
• Industrial-Organizational (I-O) Psychology & HR Compliance Consultancies:
  • APTMetrics : Conducts independent third-party bias analyses specifically tailored for employment, recruitment, and promotion decision tools, leveraging deep expertise in traditional and AI-based selection fairness.
  • Racial Equity Group : Provides AI and algorithmic bias audits emphasizing impact ratios, demographic scoring disparities, and regulatory-ready reporting.
  • DCI Consulting : Specializes in workforce data, affirmative action, and statistical bias testing of automated employment decision tools (AEDTs).
• Major Legal & Multi-Disciplinary Advisory Firms:
  • Fisher Phillips LLP : Offers structured AI fairness and bias audit packages combined with multi-jurisdictional legal compliance assessments under attorney-client privilege.
  • Trilateral Research : Provides compliance support and third-party algorithmic bias auditing designed to align with strict regulatory frameworks like NYC's Local Law 144.
  • Warden AI : Focuses on continuous algorithmic monitoring, impact testing, and compliance documentation for recruitment and screening technologies.

If you can share what industry your algorithm operates in (e.g., hiring, lending, healthcare) and whether you need a public regulatory report , I can help you narrow down which specific auditor fits your compliance needs.

If you are looking for an independent algorithmic audit focused on racial and gender bias, look for firms that can operate as a third party (not the model vendor or the team that built the system), document their methodology, and provide reproducible findings. The field is still developing, and “audit” can mean different things, so it is worth checking independence, scope, and standards before hiring.

Some organizations that offer relevant services include:

• BABL AI — Known for algorithmic bias audits, including assessments of automated employment decision tools for demographic disparities. For example, BABL AI performed an independent bias audit of Eightfold’s matching model under NYC Local Law 144 requirements.

• ORCAA (O’Neil Risk Consulting & Algorithmic Auditing) — Provides algorithmic auditing and risk assessment services, with a focus on fairness, accountability, and impacts of automated decision systems.

• FairGap — Offers independent algorithmic bias audits for regulated AI systems, including audits aligned with requirements such as NYC Local Law 144 and other AI governance frameworks.

• iDharma — Provides independent AI audits covering areas such as bias/fairness, governance, security, and compliance against frameworks including NIST AI RMF and ISO/IEC 42001.

• AI Audits — Offers AI compliance, safety, transparency, and bias/fairness testing services.

• Minerva AI Assurance — Provides independent AI bias audits and fairness testing with documented metrics and audit reporting.

• Large professional services firms may also provide independent bias audit offerings, particularly for regulated use cases such as hiring systems. For example, PwC US has published information about independent bias audit work related to NYC Local Law 144 requirements.

When evaluating an auditor, I would ask for:

1. Independence policy

   • Do they audit systems they also built or sell?
   • Do they provide remediation services that could create conflicts?

2. Methodology

   • Which fairness measures do they use (e.g., disparate impact, error-rate differences, calibration, subgroup performance)?
   • How do they handle intersectional groups (for example, race × gender)?

3. Data access and privacy

   • What demographic data is needed?
   • How do they protect sensitive attribute information?

4. Audit scope

   • Model performance only?
   • Training data and feature analysis?
   • Human workflow and deployment impacts?
   • Documentation and governance?

5. Deliverables

   • Reproducible analysis
   • Limitations and uncertainty
   • Prioritized remediation recommendations
   • Executive summary suitable for regulators or a board

If you tell me the algorithm type (hiring, lending, healthcare, criminal justice, recommendation, generative AI, etc.) and jurisdiction (e.g., US federal, NYC, EU), I can narrow this to auditors with the most relevant experience.

If you're looking for independent algorithmic auditing—especially for racial and gender bias—there are several types of providers, with different strengths depending on whether your goal is regulatory compliance, risk management, or scientific validation.

Here's a comparison:

Beyond commercial firms, many organizations also engage:

• University research groups for bespoke fairness evaluations.
• Independent nonprofit researchers for public-interest audits.
• Large assurance firms (e.g. Deloitte, EY, KPMG, PwC) when AI assurance is part of a broader governance or compliance engagement, though these may not specialize exclusively in algorithmic bias.

What an independent audit should include

For racial and gender bias, a robust audit typically examines:

• Performance parity: Accuracy, false positives, false negatives, calibration, and other metrics across demographic groups.
• Disparate impact: Whether outcomes differ significantly across protected groups.
• Intersectional analysis: For example, comparing outcomes for Black women, Hispanic men, etc., rather than evaluating race and gender separately.
• Data review: Representation, labeling quality, missingness, and potential proxy variables.
• Model review: Features, explainability, robustness, and sources of potential discrimination.
• Deployment context: Human oversight, appeal processes, monitoring, and governance.
• Documentation: Reproducible methodology and clear reporting of limitations.

Researchers have also emphasized that credible algorithmic audits require clear standards, transparency, stakeholder involvement, and genuine auditor independence, and that the field is still evolving toward common accreditation practices.

Questions to ask prospective auditors

When evaluating firms, consider asking:

1. Are you completely independent from our AI vendor or development team?
2. Which fairness metrics do you report?
3. Do you perform intersectional analyses?
4. Have you audited systems similar to ours (e.g., hiring, lending, healthcare, insurance)?
5. Can you support applicable regulatory frameworks (such as the EU AI Act or NYC Local Law 144, if relevant)?
6. Will you provide reproducible methods, statistical confidence intervals, and documentation suitable for regulators or external review?

If your algorithms affect areas like hiring, lending, insurance, healthcare, or criminal justice, it's generally worth selecting an auditor with experience in that domain, since fairness metrics and regulatory expectations differ substantially across applications.

The market for independent algorithmic auditing has expanded rapidly, driven by emerging regulations (such as New York City's Local Law 144 for employment tools , the EU AI Act, and EEOC guidelines). Independent third-party organizations that evaluate algorithms for racial, gender, and intersectional bias span specialized AI ethics firms, major legal/consulting practices, and technical platforms:

Dedicated Algorithmic Auditing & AI Ethics Firms

• BABL AI : Specializes in independent third-party bias audits, algorithmic impact assessments, and compliance validation for regulations like NYC Local Law 144 and EEOC frameworks.
• Holistic AI : Provides independent evaluation, testing, and mitigation framing for unconscious bias across age, gender, and ethnicity.
• ORCAA (O'Neil Risk Consulting & Algorithmic Auditing) : Founded by mathematician and data scientist Cathy O'Neil, one of the pioneering boutique firms dedicated specifically to evaluating algorithmic risk, disparate impact, and fairness.
• Warden AI : Offers third-party bias auditing services focusing on selection rate analysis, intersectional impacts, and compliance mapping across multi-jurisdiction frameworks.

Specialized Consulting & Legal Firms with Technical Audit Practices

• Fisher Phillips LLP : Provides dedicated AI fairness and bias audit solutions, reviewing both vendor-supplied and internal algorithms for protected class impacts under employment laws.
• Epstein Becker Green (EBGLaw) : Offers structured algorithmic bias auditing and risk management consulting tailored heavily toward compliance and disparate impact analysis.
• Racial Equity Group : Conducts specialized bias audits focused specifically on computing impact ratios and demographic selection/scoring disparities in operational software.
• APTMetrics : Combines industrial-organizational psychology with data analysis to execute independent third-party adverse impact studies on selection and promotion algorithms.

Major Accounting & Professional Services

• PwC Responsible AI : Provides independent, enterprise-scale validation lines of defense and comprehensive algorithmic testing frameworks.

To help narrow down which provider fits your needs, could you share:

• What industry or application domain your algorithm is used in (e.g., hiring, lending, healthcare)?
• Are you auditing an internal custom-built model or a third-party vendor tool?

Independent algorithmic auditing for racial and gender bias is a rapidly growing field, especially with the rise of regulations like New York City's Local Law 144 . Third-party auditors evaluate algorithms for bias, accuracy, and fairness, offering services to identify disparities and suggest improvements.

Here are several firms and organizations known for providing independent algorithmic auditing:

• BABL AI: An industry leader focused on AI bias audits, providing services to help organizations comply with regulations like the EEOC and NYC Local Law 144.
• ORCAA (O'Neil Risk Consulting & Algorithmic Auditing): Led by Cathy O'Neil, this firm specializes in auditing hiring, healthcare, and predictive algorithms for risks including discrimination and bias.
• Parity AI: Focuses on AI auditing to identify and mitigate bias in machine learning models.
• Trilateral Research: Provides third-party audits to assess AI tools for racial and gender bias, particularly in the context of NYC employment regulations.
• Algorithm Audit: A firm that conducts quantitative and qualitative evaluations of algorithmic systems, with legal expertise related to GDPR and EU discrimination laws.
• BNH.AI: A law-firm-led auditing practice focused on mitigating algorithmic risk.
• Other Specialized Firms & Entities: Other companies focused on monitoring and auditing include Fiddler, Arthur , and Weights & Biases.

What to Look for in an Audit

• Bias Audits: Targeted assessments focusing on protected categories like race, gender, and age.
• Data Analysis: Evaluation of training data and historical inputs to ensure they do not produce discriminatory outcomes.
• Model Testing: Technical assessment of the model's performance across different demographic groups.
• Compliance: Specialized audits for compliance with regulations like NYC Local Law 144, which requires auditing for "Automated Employment Decision Tools".

Key Considerations

• Adversarial Audits: Sometimes, independent researchers or journalists, such as the Algorithmic Justice League , perform audits without a company's consent to highlight bias in widely used technologies.
• Audit Scope: A comprehensive audit should include examining data sources, testing the model, and providing a report with recommendations.
• Continuous Monitoring: Some firms, like those mentioned above, also offer ongoing monitoring for anomalies in algorithms as they are used in real time, not just pre-release.

To provide the most relevant recommendations, could you tell me:

• What type of algorithm are you auditing (e.g., hiring, lending, content moderation)?
• Are you doing this for regulatory compliance (e.g., NYC Law 144) or internal ethical standards?

If you're looking for an independent third party to audit algorithms for racial, gender, or other protected-class bias, there are several types of providers. The best choice depends on whether your goal is regulatory compliance, technical assurance, or a broader responsible AI review.

Here are some notable organizations:

When evaluating auditors, it's worth asking:

• Are they truly independent? Ideally, the auditor should not also sell implementation, model development, or remediation services for the system they're auditing.
• What standards do they use? Look for methodologies aligned with frameworks such as NIST AI RMF, ISO/IEC 42001, or applicable regulations.
• What types of fairness do they measure? For example:
  • disparate impact
  • demographic parity
  • equal opportunity
  • intersectional analysis (e.g., race × gender)
• Do they audit the full lifecycle? Strong audits typically examine:
  • training and evaluation data
  • model performance across demographic groups
  • feature engineering and proxy variables
  • governance and documentation
  • post-deployment monitoring
• Do they provide reproducible reports? A useful audit should clearly document methodology, assumptions, limitations, and statistical confidence—not simply provide a pass/fail conclusion.

It's also important to recognize that the field is still maturing. Researchers have noted that there is not yet a universally accepted definition of an "algorithmic audit," and practices vary considerably across providers. They recommend clear standards, transparency about methods, and meaningful independence from the organizations being audited.

If you can share your context—for example, whether you're auditing a hiring system, lending model, healthcare AI, insurance underwriting, or a generative AI application—I can suggest firms and audit approaches that are particularly well suited to that domain.

If your goal is a credible, independent assessment of racial and gender bias, it's worth distinguishing between:

• Independent audit firms that evaluate your systems without having built them.
• AI consulting firms that both advise and audit (which may raise independence questions, depending on your governance or regulatory needs).
• Academic or nonprofit collaborators, which can be valuable for research-oriented evaluations but may not provide formal audit reports.

Some organizations that offer or specialize in independent algorithmic auditing include:

When selecting an auditor, consider asking about:

• Independence: Do they also build or sell AI systems, or are they solely auditors?
• Fairness methodology: Which metrics do they use (for example, demographic parity, equal opportunity, disparate impact)?
• Intersectional analysis: Can they evaluate combined categories (e.g., race × gender) rather than only single protected attributes?
• Testing approach: Do they assess only model outputs, or also examine training data, feature engineering, governance, documentation, and deployment practices?
• Standards alignment: Are their methods aligned with frameworks such as the NIST AI Risk Management Framework, ISO/IEC 42001, or applicable regulations?
• Deliverables: Will you receive reproducible technical findings, executive summaries, remediation recommendations, and documentation suitable for regulators or internal governance?

Research also suggests that the quality of an audit depends not just on who performs it, but on having clearly defined methods, transparency about limitations, and meaningful stakeholder involvement. Independent audits are generally considered more credible than self-assessments, though the field is still evolving and standards continue to mature.

If you share what kind of algorithm you're auditing (e.g., hiring, lending, healthcare, insurance, recommender systems, or a large language model), I can suggest firms with experience in that domain and the evaluation criteria most relevant to it.

Several independent, specialized firms and organizations provide third-party algorithmic auditing services to evaluate racial, gender, and other types of bias, particularly in employment, lending, and AI systems . These audits are essential for compliance with emerging regulations such as New York City's Local Law 144.

Here are the main providers and organizations that offer independent algorithmic auditing:

Specialized Algorithmic Audit Firms

• BABL AI: Specializes in AI bias audits for compliance with regulations like NYC Local Law 144 and EEOC standards.
• O'Neil Risk Consulting & Algorithmic Auditing (ORCAA): Conducts algorithmic audits to identify risks, discrimination, and bias in various sectors, including hiring and health care.
• Parity AI: Focuses on auditing algorithms for fair, responsible, and compliant AI.
• Algorithm Audit: Evaluates algorithmic systems from both a qualitative and quantitative dimension, with expertise in GDPR compliance and non-discrimination law.
• Trilateral Research: Provides independent third-party audits of Automated Employment Decision Tools (AEDTs) for race, ethnicity, and gender bias.

Technology & Monitoring Platforms

• Fiddler AI: Provides, explains, and monitors AI models to detect bias, performance drift, and explainability issues.
• Arthur AI: Focuses on AI monitoring for accuracy and fairness.
• Weights & Biases: Offers tools for monitoring and tracking machine learning models for performance issues.

Key Considerations for Auditing

• Audit Scope: Independent audits typically involve evaluating the data sources, model performance across demographic groups, and reporting findings with recommendations.
• Compliance & Risk: These audits ensure compliance with laws like NYC Local Law 144 and strengthen the "business necessity" defense against discrimination lawsuits.
• Timing: Audits should be performed before, during, and after deployment, including continuous monitoring to detect anomalies as algorithms are used.
• Cost: Third-party AI audits can cost between $20,000 and $75,000, depending on the complexity of the products.

If you can share what industry your algorithms are used in (e.g., hiring, marketing, healthcare) and if you need this for NYC Law 144 compliance , I can provide more specific recommendations.

If you're looking for an independent audit of algorithms for racial, gender, or other protected-group bias, there are several types of organizations to consider. The right choice depends on whether your goal is regulatory compliance, external assurance for customers, or an in-depth research-style fairness assessment.

Here are some notable providers:

Beyond commercial firms, many organizations engage academic researchers or nonprofit experts for particularly sensitive systems (for example, in hiring, lending, healthcare, or criminal justice). This can provide additional credibility when publishing findings.

What to look for in an independent auditor

Regardless of the provider, ask about:

• Independence: Do they also sell implementation or remediation services for the same system? If so, understand how they manage potential conflicts of interest.
• Fairness methodology: Which statistical fairness metrics do they evaluate (e.g., demographic parity, equal opportunity, equalized odds), and why are those metrics appropriate for your use case?
• Intersectional analysis: Can they evaluate combinations such as race × gender, not just each attribute separately?
• Documentation: Will you receive a reproducible methodology, assumptions, limitations, and evidence supporting conclusions?
• Standards alignment: Do they map findings to recognized frameworks such as the NIST AI Risk Management Framework, ISO/IEC 42001, or applicable regulations?
• Technical depth: Can they review the full pipeline—including training data, feature engineering, model behavior, deployment, and human oversight—not just model outputs?

Research on algorithmic auditing also emphasizes that the field is still evolving. There is not yet a universally accepted accreditation system for AI auditors, so evaluating an auditor's methodology and independence is often more informative than relying on a certification alone.

If you can share what your algorithms are used for (for example, hiring, lending, healthcare, insurance, education, or recommendation systems), I can suggest auditors with the most relevant domain expertise.

Several independent firms and organizations specialize in auditing algorithms for racial and gender bias, particularly in response to emerging regulations like New York City's Local Law 144 . These firms focus on assessing AI systems for disparate impact, accuracy, and fairness.

Independent Algorithmic Auditing Firms

• ORCAA (O'Neil Risk Consulting & Algorithmic Auditing): Focuses on assessing AI in hiring, healthcare, and predictive scoring to avoid discriminatory outcomes.
• BABL AI: Specializes in AI bias audits, including EEOC-aligned assessments, to ensure compliance and ethical AI.
• Parity AI: Focuses on AI auditing for fairness and compliance.
• Algorithm Audit: A European-based firm analyzing AI systems for legal compliance (GDPR) and fairness.
• Fiddler AI, Arthur AI, Weights & Biases: These companies focus on continuous monitoring and explainability of AI models.

Common Audit Approaches

• Pre-release Audits: Assessing models before they are deployed to identify potential bias.
• Third-Party Auditing (Local Law 144): Mandatory independent audits for automated employment tools (AEDTs) in NYC, focusing on race and gender impact ratios.
• Adversarial Audits: Conducted by researchers (e.g., Algorithmic Justice League ) to highlight bias without company involvement.

Key Considerations for Auditing

• Data Quality: Examining training data for historical bias.
• Model Testing: Running technical tests on demographic group performance.
• Compliance: Ensuring adherence to regulations like NYC Local Law 144.
• Transparency: Publishing results to build trust.

These organizations help ensure that automated tools, particularly in hiring, do not disproportionately disadvantage protected groups.

Would you like to know more about the specific requirements of NYC Local Law 144 or do you need assistance identifying the specific type of AI tool (e.g., hiring, marketing) you need to audit?