AI Engineering · Applied ML

Ghana Image Captioning

MSc research: adapting a general image captioner to Ghanaian visual culture, with an honest account of what did and did not work.

Status: Completed
Year: 2026
Role: Researcher (MSc)
Client: Independent project

Overview

Built an openly licensed image dataset with provenance logging, annotation and rating tools, then ran a controlled set of experiments on Kaggle T4 GPUs: pretrained BLIP, fine-tuned BLIP, and several tag-conditioned variants using a CLIP-based tag predictor, over five paired seeds.

Python
Jupyter
PyTorch
Hugging Face Transformers
BLIP
CLIP
Kaggle (T4 GPU)

The challenge

General captioning models describe Ghanaian everyday scenes poorly and miss local names. The question was whether a small, locally curated dataset can adapt a model like BLIP, and whether conditioning it on predicted Ghana-specific tags helps further.

Look-alike dishes (kenkey, banku, fufu) were the main confusion
Tag predictor F1 was only about 0.5

System requirements

Dataset pipeline: open-licence image collection, review, caption and tag annotation tools, frozen splits with checksums
Seven experiment variants evaluated over five paired seeds
Blind human rating tool and error analysis
Local demo comparing the pretrained and fine-tuned captioners

Solution architecture

Content needed: architecture, drawn only from documented components

Implementation

Content needed: real screenshots (desktop and mobile). Generated mockups are never shown as screenshots.

Technical decisions

Freeze the validation and test sets with checksums before running experiments
Use paired seeds so differences between variants are comparable
Report negative results instead of only the best number

Results and validation

Fine-tuning raised CIDEr from 0.304 (pretrained) to 0.531 on a 58-image test set in a single run (five-seed mean 0.509). Tag conditioning did not help: every variant scored below plain fine-tuning on paired CIDEr in most seeds, although some raised recall of Ghanaian terms. In a blind human rating by one rater, the pretrained model scored highest (3.81 versus 3.43 for fine-tuned), so the automatic gains did not translate into human preference.

Evidence

Results are generated from the project's own run outputs (docs/FINAL_RESULTS.md)
Limitations recorded in the repo: AI-written references, 58 test images, one human rater

Lessons learned

Content needed: what you would do differently

Content checklist (remove once filled)

Decide whether to make the repository public. 84 images carry non-commercial or no-derivatives licences and must stay out of any public release
Confirm the university allows publishing these results
Add the CIDEr and human-rating charts as images