Comparing Cognitive Load in Visual Quality Assessment of Dot Images
​

We created a dataset of dot images with a 640 × 480 pixels resolution. Each image had a large number of non-overlapping dots. We compared a reference image containing 300 dots to 20 test images with a greater number of dots. The number of dots in the test images ranged from 302 to 340, with a step size of 2.
​
We conducted a crowdsourcing study with 254 participants using a within-subject design. We presented paired comparisons in both AFC (two-alternative forced choice) and RFC (relaxed forced choice with "not sure" response option) formats. Each participant was asked to respond to 40 pair comparisons, with and without the "not sure" response option. We also asked participants to complete the NASA-TLX questionnaire to evaluate their cognitive load for each testing condition.

Cite us

CogDotVQA  dataset is freely available to the research community. If you use our database in your research, please cite the following reference:

@inproceedings{jenadeleh2023relaxed,
  title={Relaxed forced choice improves performance of visual quality assessment methods},
  author={Jenadeleh, Mohsen and Zagermann, Johannes and Reiterer, Harald and Reips, Ulf-Dietrich and Hamzaoui, Raouf and Saupe, Dietmar},
  booktitle={2023 15th International Conference on Quality of Multimedia Experience (QoMEX)},
  pages={1--6},
  year={2023},
  organization={IEEE}
}
​

Downloads

Images

346.3 MB download

Subjective data

23.3 kB download

Subjective Data

• The subjective data file includes two folders: 
  
  • NASA-TLX_questionnaire_answers: This folder includes 235 JSON files, each containing the responses from a worker to the NASA-TLX questionnaire for cognitive load assessment in both test conditions. It also includes 19 rejected responses. 
    
  • Workers_responses_to_PCs: This folder includes 235 JSON files of the approved responses from 235 workers to the paired comparisons in the two test conditions. It also includes 19 rejected response.