Skip to content

Feature: Accept image list as input to get_frame_cov (bypass MP4 decode) #19

@EYH0602

Description

@EYH0602

Motivation

In game-fuzz, we record gameplay as MP4 at a fixed FPS, then pass the file to
get_frame_cov() which decodes it back into frames for perceptual hashing. This is
wasteful in two ways:

  1. Encode-then-decode roundtrip — we capture screen frames, encode them into MP4,
    then gamecov decodes them right back into frames.
  2. No control over which frames are hashed — the caller has no way to pass
    semantically meaningful frames (e.g., captured at action boundaries or during key
    game state transitions). All frames at the recording FPS are processed.

Games are continuous simulations — unlike browser/desktop automation, the game world
evolves between player inputs. An ideal sampling strategy would capture frames both at
action boundaries and periodically during idle/wait periods. This requires the ability
to pass pre-captured images directly.

Proposed API

Add a function that accepts a sequence of images instead of an MP4 path:

from gamecov import get_frame_cov_from_images, FrameCoverage
from PIL import Image
from returns.result import Result

def get_frame_cov_from_images(
    images: list[Image.Image],  # or list[np.ndarray]
) -> Result[FrameCoverage, str]:
    """Compute frame coverage from a list of pre-captured images."""
    ...

This would complement the existing get_frame_cov(path: str) which remains useful when
an MP4 is already available.

Use Case

# In the fuzzer's action execution loop:
screenshots: list[Image] = []
for action in seed.actions:
    execute(action)
    screenshots.append(capture_screen(region))

# Compute frame coverage directly from screenshots
frame_cov = get_frame_cov_from_images(screenshots)

Benefits

  • Eliminates MP4 encode/decode overhead for coverage computation
  • Enables action-boundary and event-driven frame sampling strategies
  • Maintains backward compatibility (existing MP4 API unchanged)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions