Meridian Prime_AI Safety Fund: AI Agent Evaluation RFP, Due: Jan 31, 2025

[opencode4good]

Overview:
AI Safety Fund

Our Mission

The AI Safety Fund (the AISF) is on a mission to accelerate and expand the field of AI safety research to promote the responsible development and deployment of frontier models, minimize risks, and enable independent, standardized evaluations of capabilities and safety.

Born from a collaborative vision of leading frontier AI developers and philanthropic partners, the fund will award research grants to independent researchers addressing critical safety risks associated with developing frontier models and preventing or mitigating malicious use of AI systems. AISF is a multi-million-dollar research initiative independently administered by Meridian.

AISF-AI-Agent-Evals-RFP-1.pdf

AI Agent Evaluation and Synthetic Content RFP

The AI Safety Fund (AISF) seeks to support technical research for AI agent identity verification systems and AI agent safety evaluations. This funding aims to promote the safe and responsible development of AI agents while establishing robust frameworks for agent authentication and verification. The following is a list of examples of the kinds of research we might like to support. We welcome proposals on these topics and other relevant topics under this domain.

Agent Governance Infrastructure

As AI agents become increasingly integrated into critical systems, it is essential to ensure their authenticity, integrity, and trustworthiness. A robust governance infrastructure may combine cryptographic identity frameworks with behavioral analysis to safeguard against spoofing, manipulation, and unauthorized modifications.

Research Objectives: Development and evaluation of:

Cryptographic protocols for establishing and managing persistent AI agent identities
Techniques for detecting unauthorized modifications to agent identity systems
Methods for detecting anomalous agent behavior that may indicate compromise
Oversight and Accountability in Deployment

As AI agents are deployed across organizations, maintaining verifiable chains of custody and tracking authorized modifications becomes critical. Enabling efficient and effective oversight of agents in deployment within an organization is essential in order to facilitate human detection and triage of issues when problems arise.

Research Objectives: Development of:

Provenance tracking systems for AI agent deployment
Methods for verifying legitimate AI agent actions, up to and including modifications to AI agent management systems
Techniques for detecting unauthorized tampering in deployment pipelines
Emergent Behavior Analysis

Increasingly complex AI agents may exhibit unexpected emergent behaviors that require systematic evaluation approaches.

Research Objectives:

Methods for detecting and analyzing emergent anomalous agent behaviors
Frameworks for evaluating behavioral stability over time
Techniques for testing agent responses to novel situations
Scalable Oversight Mechanisms

Developing robust and scalable oversight mechanisms is essential to ensure accountability as AI systems become increasingly autonomous.

Research Objectives:

Methods for enabling AI systems to provide scalable and reliable oversight
Techniques for enhancing human capabilities to maintain effective oversight of AI systems
Techniques for detecting when agents operate outside safe parameters
Synthetic Media Authentication

As AI-generated content proliferates, ensuring authenticity, traceability, and accountability requires a multifaceted approach. Addressing these challenges requires a holistic perspective that integrates multiple strategies for content credentialing, such as behavioral fingerprinting at the agent level and watermarking, metadata embedding, and content fingerprinting at the content level.

Research Objectives:

Development of evaluation frameworks and benchmarks to test the security, robustness, privacy, and fairness of digital content transparency techniques
Assess the resilience of authenticating content credentials including watermarking, signed metadata, and fingerprinting (and their intersection) against adversarial removal, tampering, and forgery across multiple modalities (images, audio, video, text).
Evaluation Criteria

Impact
Research proposals will be assessed based on their potential to
improve the security of protocols for managing interactions with AI agents and
evaluate AI agents’ safety and security.
This includes the practical applicability of the expected results and their potential for implementation in real-world settings.
Feasibility
The proposed project should include a clear timeline with well-defined milestones.
The proposal should address potential challenges and include strategies for addressing them.
Relevance
The proposed research must directly apply to frontier AI models.
The proposal should cover existing research and how it relates to their project.
Peer Review
The proposal must include a robust plan for engaging with the broader research community and receiving feedback.
The proposal should demonstrate how peer feedback will be incorporated into the research process and how the broader scientific community will validate findings.
Technical Qualifications
The evaluation will consider the team's AI safety and cybersecurity track record.
Proposals should include the applicants’ academic degrees, previous publications, projects, and contributions to the field.
We’re open to applicants without such track records if the project is particularly well-scoped and promising.
Especially in this case, having named advisors on the project with relevant subject matter expertise and research experience can be helpful.
Ethics
Proposals must outline specific safety protocols that address both immediate research risks and potential downstream implications of the findings.
This should detail how sensitive data and results will be handled, secured, and accessed throughout the project lifecycle.
The proposal must also include a clear protocol for identifying and managing security-sensitive findings, particularly any unexpected discoveries that may emerge during the research process.
Additionally, proposals should demonstrate an ethical approach to all research methodologies, avoiding any practices that may inadvertently mislead or compromise collaborators without their informed consent.
Equity
Proposals should describe how the project will advance equity and diversity in the research community, particularly regarding underserved populations.
Accessibility
The research product should prioritize open accessibility through open-source licensing, promoting transparency and broad utility.
However, if unrestricted access poses a risk of harm or compromises privacy, proposals should provide a justification for limited access.
Evaluators will assess the proposal’s approach to balancing accessibility with safety and security considerations.
Funding

The AISF plans to fund research projects by academic labs, non-profits, independent researchers, and for-profit mission-driven entities across these topics. Proposed budgets should not exceed $500k.

You can learn more about this opportunity by visiting the funder's website.

Eligibility:
The AISF makes grants for independent researchers affiliated with academic institutions, research institutions, NGOs, and social enterprises across the globe that aim to promote the safe and responsible development of frontier models by testing, evaluating, and/or addressing safety and security risks.
The AISF seeks to fund research that accelerates the identification of threats posed by the development and use of AI to prevent widespread harm.
Eligible Proposals
Technical research projects focused on evaluating AI agents and improving agent infrastructure, as described above.
Projects must focus on frontier AI models and their deployed versions.
The research duration must be one year or less, and the budget must not exceed $500k.
Eligibility with the AISF’s Conflict of Interest Policy.

Ineligibility:
The U.S. Department of the Treasury’s Office of Foreign Assets Control (OFAC) imposes restrictions on services and transactions with individuals or entities located in countries subject to comprehensive U.S. sanctions.
As a result, due to applicable US sanctions, the AISF is unable to award grants to the following countries:
Cuba
Iran
North Korea
Syria
Russia
Regions of Ukraine: Crimea, Donetsk, Luhansk
Belarus
Please note: The list of sanctioned countries and activities may change at any time without prior notice, in accordance with OFAC regulations.