What AI says about UMBRELA

umBRELA is an open-source tool that reproduces the Bing RELevance Assessor for query-passage relevance labeling. It uses large language models to assign four-level relevance judgments and supports multiple judge backends for information retrieval evaluation workflows.