Skip to content

Port SWEfficiency benchmark to Harbor #725

Description

@neubig

Context

benchmarks/swefficiency does not appear to have a corresponding Harbor registry dataset/adapter yet. It has resource-sensitive Docker behavior and performance-oriented verification, so it should be ported deliberately with parity checks.

Proposed scope

  • Create a Harbor adapter/dataset for SWEfficiency tasks.
  • Preserve CPU/memory/resource assumptions and performance scoring semantics.
  • Validate Oracle/reference behavior where available.
  • Compare current OpenHands harness results against Harbor runs.

Acceptance criteria

  • SWEfficiency tasks can be run through Harbor with appropriate resource controls.
  • Performance/verifier parity is documented.
  • The OpenHands wrapper can delegate SWEfficiency execution to Harbor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions