When AI thinks it understands pictures and text: what modern multimodal models can (and cannot) do

For today’s models this is a piece of cake. Their accuracy is close to one hundred percent. So it would seem that the problem of “understanding images and text” is practically solved.

But what if we do not give them three options, but ten, a hundred, or a thousand? And what if there are sentences among them with a very similar meaning, like “A woman runs along a road”, “A person stands on a bridge” or “A man runs uphill”?

Figure 1: The opening example. With three very different sentences the choice is easy. Among thousands of similar ones, some of them correct, some debatable and some wrong, it is not. Photo: Jules Verne Times Two, Wikimedia Commons, CC BY-SA 4.0.

Suddenly it turns out that testing what AI models can do is a lot more complicated than it looks at first sight.

And exactly this gap, between “understanding images and text is solved” and “in reality, multimodal models are still far from really understanding and connecting images and text”, is what my PhD research was about.

What is this project about?

I have been working on this topic for five years now, as a PhD student at KInIT (our PhD programme runs together with the Faculty of Information Technology of the Brno University of Technology). The official name of my dissertation is “Understanding and grounding of multimodal image-language models”. Simply put: does a model that “sees and reads” really connect the two? And how can we even find out? Partially, my research was supported by PwC Slovakia and this blog is a short summary of what I achieved.

What are these models that “see and read”?

In the last few years, so-called vision-language models appeared, models that can work with images and text at the same time. CLIP, BLIP-2 or the newer multimodal versions of large language models like GPT-4, GPT-5, Gemini or Qwen belong here.

Under the hood they do it in two ways. The first kind, for example CLIP, turns the image and the sentence into number representations, into vectors in the same “space of meaning”. If the sentence is a good description of the image, the two vectors are close to each other. If not, they are far apart. The second kind, the generative models like GPT-4 or Qwen, cuts the image into small pieces, turns them into “visual words” and feeds them into a language model together with the text. The model then answers in words: it describes the image, or it tells you whether a sentence fits it. (With a bit of extra training, a generative model can also be turned into the first kind, which we did in some of our experiments.) In both cases we can ask the same question: here is an image and here are a few sentences, which one fits best?

Figure 2: Two ways of connecting an image with text. Embedding models (left) place the image and the sentences in one shared space, where the distance says how well the sentence fits. Generative models (right) turn the image into “visual words”, read them together with the text, and answer in words.

Thanks to this, multimodal models can for example:

On paper they look very convincing. On the standard benchmarks (test datasets) they get high numbers and good ratings. And this is exactly where my research started: do these models really understand what they see, or are they just playing a clever statistical game?

The problem: we test them too easily

When I started my PhD, most tests of model understanding worked like this:

We give the model an image and two sentences:

The model has to pick the correct description.

Figure 3: An image from the SVO-Probes dataset. In the classic test, the model only has to prefer the correct caption “A person walking on a trail” over a single altered one, “A person runs on the trail”. (Source: our SOFSEM 2025 paper)

It is a bit like testing a history student with the question:

Was the Second World War in 1945 or 1545?

If they answer correctly, it does not mean they understand history. Only that they know which option is obvious nonsense. Besides, why is it 1545 in the question and not 1845, or even 1946? It is quite possible that with the options 1945 and 1545 the student picks the right year, with 1945 and 1845 they already hesitate, and with 1945 and 1946 they easily pick the wrong one.

Many AI benchmarks work the same way. They do not test the models on all the wrong options, and probably not even on the ones that are the biggest challenge for them. The model learns to tell the extremes apart, but we do not know what it would do in the real world, where the options are much more similar.

From two options to the whole database

So in my research I changed the perspective.

Instead of “for this image, pick the correct text out of these two options” I started asking “out of the whole database of thousands of sentences, which 10 are, according to you, the most fitting for this image?”

This task is called retrieval.

It is a more realistic scenario; this is exactly how image search engines or multimodal databases work.

And this is where interesting things started to happen.

When a high score lies

On the classic two-option tests, the models we studied picked the correct sentence in eight or nine cases out of ten. That looked great.

But then we gave them the real task: for a given image, find the matching sentence among thousands of candidates. Now we count a success only when the very first sentence the model returns is the one that was written for that image. And suddenly the models succeeded in only about one case out of ten.

Most of the time the model did find “something similar”, but it got the action or the relation between the objects wrong.

What surprised us even more: the hardest wrong answers were not the ones somebody wrote by hand for the benchmark, but the ones the models themselves pulled out of the database.

Figure 4: The same query, three candidates. (a) The correct image; (b) the hand-made wrong image from the benchmark, in which the object was changed; (c) a wrong image that the models found themselves in their top 10: the road and the motorcycle are there, but the action is different. Five of the six models we tested ranked (c) above the correct image. (Source: our work under review)

Typical mistakes:

It turned out that:

In other words, the model knows there is a “person” there, but it is not so sure what they are doing. They go for activity that is most probable for given subject and surroundings (one never swims on the road, almost never sleeps in the park and just sometimes dances on the beach).

For real applications this is quite a fundamental problem.

Another catch: not all “mistakes” are really mistakes

With retrieval, one more interesting effect appeared.

Check the image at the beginning of this blog with the sentence: “A girl runs across a bridge.”

The model returns: “A woman runs along a road.”

Is it wrong? Or actually… quite reasonable?

The classic metrics mark it as an error, because the sentence is not exactly the “golden” one.

But semantically it is completely fine.

So we introduced:

The result? The real quality was often much better than the hard numbers suggested. Under the human labels, the first result was correct for about half to nine out of ten depending on the model.

This shows that evaluating AI is sometimes harder than training it.

A model as a judge

Why bother with a model as a judge? Because human annotation is expensive, and we wanted to be able to evaluate new models on new datasets without it.

One example from our own work: to check a single dataset, we took 100 queries and the ten results each model returned for them. That gave us about 3,000 image-caption pairs. Each pair was judged by three people, so roughly 9,000 decisions in total. Even at ten seconds per pair, that is more than a full working day for every annotator. And the work is as dull as it sounds: look at a picture, read a sentence, click yes or no. Then again. And again. Thousands of times. For the next model or the next dataset, you start from zero.

So we let a model help. Once we have the ten results retrieved for a query, we turn them into ten pairs: the query with the first result, the query with the second result, and so on down to the tenth. Each pair then becomes a simple question: “Does this image match this caption? If not, what does not fit: the subject, the verb or the object?”

Judging one pair at a time is a much easier task than searching through thousands of candidates, and as we already know, the strongest models today do it very well. So we handed the pairs to the best model available and let it decide.

To be sure, we compared its verdicts with human annotators on the same pairs. The agreement was high. Not quite as high as people have with each other, but close.

This opens the way to cheaper and scalable evaluation of large datasets.

Who bit whom?

In my newest work I went one step further. The hardest test of whether a model understands an action is to swap the participants: “A dog bites a hand” becomes “A hand bites a dog”. Today’s big models pass this test in about nine cases out of ten. But when we ask the same model to show us “the one who is biting” in the picture, without naming it, it finds both participants only in roughly a third to a half of the cases. So we once again show that passing the image-caption matching test does not prove that the model knows understanding text and image.

Figure 5: A model prefers the caption “A dog bites a hand” over “A hand bites the dog”. But asked to point at “the one who is biting”, it boxes the person (orange-agent?), and asked for “the item that is being bitten”, it boxes the dog’s muzzle. The blue boxes are the correct answers. (Source: our work under review)

What follows from this?

The main finding of my research is simple, but a bit unpleasant:

A high score on benchmarks does not yet mean that the model really understands.

The models:

That is why we propose beside other options to evaluate models “after retrieval”: look at their top results and analyse the mistakes.

It is a bit like watching how a person behaves in the real world instead of giving them a multiple choice test.

And where is it useful?

These findings are not just an academic game.

They concern:

If a model misunderstands “who does what to whom”, it can lead to wrong conclusions, for example when analysing news or disinformation.

So we do not only need bigger models, but also better tested ones.

What came out of my research

In concrete terms, the research produced three published papers, four papers currently under review, and a publicly available dataset and code.

Figure 6: Two images from the CV-Probes dataset. “A woman sits in the street with a cup in her hand” fits both pictures, but “A woman begs for money in the street” fits only the left one. Telling them apart takes social knowledge, not just what is visible. (Source: our CogSci 2025 paper)

A personal note from the PhD

One thing I realised during my PhD is that in AI it is very easy to chase numbers.

Increase the accuracy by one percent. Add a bigger model. More data.

But sometimes it is more important to stop and ask:

“What do these numbers actually mean? Are we really measuring what we think we are measuring?”

A big part of my research was not about making the model better, but about making the evaluation more honest.

And paradoxically, that is often what shows us most clearly what the models still cannot do.

We would like to thank the PwC Endowment Fund at the Pontis Foundation for funding this research!