Skip to main content
If you already have an agent, you can integrate it with Eval Protocol by using the GithubActionRolloutProcessor. GithubActionRolloutProcessor delegates rollout execution to GitHub Actions workflows that you control. It’s useful for implementing rollouts with your existing agent codebase by wrapping it in a GitHub Actions workflow.

​
Setup

For the GitHub token, create a Personal Access Token (classic) with permissions for repo and workflow. 
GITHUB_TOKEN="ghp_..."

​
GitHub Actions Contract

We expect the GitHub Actions workflow to accept the following inputs:
  • completion_params: JSON completion parameters (model, temperature, etc.)
  • metadata: JSON string containing rollout execution metadata
  • model_base_url: Base URL for the model API (e.g., “https://tracing.fireworks.ai”)
  • run-name: You must add run-name: rollout:${{ fromJSON(inputs.metadata).rollout_id }} to your workflow. The GitHub API doesn’t return the run ID when dispatching, so this allows us to find and monitor the correct workflow run.
name: Eval Protocol Rollout
run-name: rollout:${{ fromJSON(inputs.metadata).rollout_id }}

on:
  workflow_dispatch:
    inputs:
      completion_params:
        description: 'JSON completion params'
        required: true
        type: string
      metadata:
        description: 'JSON serialized metadata object'
        required: true
        type: string
      model_base_url:
        description: 'Base URL for the model API'
        required: true
        type: string

​
Metadata Piggyback

To push trajectories back to eval-protocol, you must include the following metadata along with your traces so that they can correlated with the EvaluationRows.
  • invocation_id
  • experiment_id
  • rollout_id
  • run_id
  • row_id

​
Example

See the following example implementation: 
pytest tests/github_actions/quickstart.py -vs
⌘I