Skip to main content
If you already have an agent, you can integrate it with Eval Protocol by using the GithubActionRolloutProcessor. GithubActionRolloutProcessor delegates rollout execution to GitHub Actions workflows that you control. It’s useful for implementing rollouts with your existing agent codebase by wrapping it in a GitHub Actions workflow.

​
High Level Flow

Github Action Rollout Processor flow
  1. /init triggers one rollout: Eval Protocol dispatches a GitHub Actions workflow with completion_params, metadata (incl. rollout_id), and model_base_url.
  2. Polling to check rollout status: The processor finds the rollout:<rollout_id> run and polls GitHub Actions until it completes.
  3. Send chat completions and store as trace: The workflow executes your agent and sends completions/logs to Fireworks with the rollout’s correlation tags.
  4. Once rollout finished, pull full trace and evaluate: Eval Protocol fetches the Fireworks trace by rollout_id and scores the result.
Everything inside the dotted box is handled by Eval Protocol — you only need to implement the GitHub Actions workflow, more on this below.

​
Setup

For the GitHub token, create a Personal Access Token (classic) with permissions for repo and workflow. 
GITHUB_TOKEN="ghp_..."

​
GitHub Actions Contract

We expect the GitHub Actions workflow to accept the following inputs:
​
completion_params
string
required
JSON completion parameters (model, temperature, etc.)
​
metadata
string
required
JSON string containing rollout execution metadata
​
model_base_url
string
required
Base URL for the model API (e.g., “https://tracing.fireworks.ai”)
​
api_key
string
required
API key for the model API
You must add run-name: rollout:${{ fromJSON(inputs.metadata).rollout_id }} to your workflow. The GitHub API doesn’t return the run ID when dispatching, so this allows us to find and monitor the correct workflow run.
name: Eval Protocol Rollout
run-name: rollout:${{ fromJSON(inputs.metadata).rollout_id }}

on:
  workflow_dispatch:
    inputs:
      completion_params:
        description: 'JSON completion params'
        required: true
        type: string
      metadata:
        description: 'JSON serialized metadata object'
        required: true
        type: string
      model_base_url:
        description: 'Base URL for the model API'
        required: true
        type: string
      api_key:
        description: 'API key for the model API'
        required: true
        type: string

​
Metadata Correlation

When making model calls in your GitHub Actions workflow, include the following metadata in your traces and logs so that eval-protocol can correlate them with the corresponding EvaluationRows during result collection. GithubActionRolloutProcessor automatically generates this and sends it to the server, so you don’t need to worry about wrangling metadata.
  • invocation_id
  • experiment_id
  • rollout_id
  • run_id
  • row_id

​
Example

See the following repo for a simple end to end example: