Reusable Bias Evaluation Framework for LMs and VLMs

This technical research framework evaluates whether language and vision-language models introduce systematic framing differences across social, political, or cultural groups.

The framework supports both text-only matched prompts and image-instruction VLM tasks. It separates case-specific resources from shared code for data loading, prompt construction, model execution, metric scoring, group disparity analysis, bootstrap summaries, annotation sheet creation, and agreement analysis.

My contribution was to design the reusable framework, implement the shared evaluation modules, structure three case studies, and define a validation boundary: automatic metrics are screening signals, while strong bias claims require human annotation and agreement analysis.

• Built a configuration-driven framework for geographic, gender-occupation, and VLM political/moral bias case studies.
• Implemented matched-prompt generation, model inference wrappers, automatic framing metrics, and group disparity summaries.
• Extended the framework from text-only variables to image-instruction VLM description tasks.
• Added human annotation sheet generation and Cohen's kappa support for validation.
• Positioned the project as a reusable research tool and case-study platform rather than a finished benchmark claim.

Image Review and Validation Workflow

The VLM case study uses reviewed image sets, metadata, captioning instructions, automatic framing metrics, and human validation sheets.