Finding What Works in GeoAI
Knowing what combination of geoAI tooling works best for your unique problem is not easy. Individual geospatial models may have been benchmarked for generic tasks. But you need to understand how likely they will work for your specific domain, on your imagery and embedded in your analysis pipeline. At LGND, we often get asked "how well can we find X" and run evaluations to find the best workflow for each question. Those evaluations have evolved into a knowledge base: for every possible concept, what setup of imagery source, embedding model, preprocessing choices and retrieval method has performed well and which ones failed and why?
The infrastructure
We evaluate the technology underlying our products on real-world questions, using the workflows our customers might actually use. This often means neat academic benchmarks don't apply. We also are quick to swap new models or imagery sources into our workflows as they become available. Together, this shapes how we approach evals:
1. We evaluate the end-to-end workflow
We score each ingredient on its own: 0-shot retrieval to seed a workflow (Can we retrieve a couple of correct locations to begin with?) all the way to a complete fine-tuned classification over an area (can we find all of a concept?). These are distinct challenges. Scoring each stage shows us where a workflow breaks down, and at what cost.
Consider this example: The goal is to detect all utility-scale solar farms in Massachusetts. A text query, "solar farm", returns 40 images; a reviewer marks each image as true or false; the system learns from the answers, searches the whole state again and brings 40 more. After ten such steps we can evaluate, at every step, how many of the known sites the search found, how many the reviewer confirmed by hand, and how noisy the results were, against the baseline of a reviewer picking images at random.
The first 40 text results are all solar farms but only 3% of the known sites: a good start, not an inventory. By the tenth step the search finds 61% of them, three times what the reviewer confirmed by hand, with 99 of every 100 results correct.
We test different pipelines and tools to optimize the search and to answer the question: which setup requires the least amount of review effort for the most complete retrieval?
2. Multiple tool setups
The number of testable combinations of tools, models and imagery is endless. The size of the target concept determines the required imagery resolution. However, it is not always predictable at what size a concept stops being visible to an embedding model. Different embedding models will differ in their performance across concepts.
We compared Clay and FarSLIP embedding models in how well they retrieve 0-shot across 52 OpenStreetMap concepts, given the same imagery. For each we graded the top 100 results. While the models have similar performance on average, they differ on how well they perform concept by concept.
Clay leads on linear features (railway lines, airports, runways), FarSLIP on compact facilities (dams, substations, orchards). The knowledge base records which model to use for which concept.
3. Qualitative error analysis
To find the geoAI workflows that give the best results, we first need to understand where they fail. Embedding models fail in surprising ways. So we do not only assess retrieval quality in numbers, we also dive into qualitative error analysis.
In the solar farm detection example, we noted that an automated search found every large farm and misses many small ones, because when a small site fell on the tile boundary any individual image only has a tiny portion depicting the concept 'solar farm'. Other missed detection were simply different in how they looked, like rooftop solar and car-park arrays: a definition question.
An automated failure-mode report over missed sites: the embeddings are clustered, and a VLM categorized each cluster into a failure mode.
Four kinds of miss: rooftop and car-park arrays (39%), ground-mount arrays in woodland where the panels are plainly visible (39%), farmland sites not yet built when the photo was taken (19%), and tiles cut at the edge of the imagery (2%).
4. Multi-source agreement
The concepts that matter to our users often have no clean reference data. Therefore, we adopt the stance that every reference lies. We built a multi-judge framework, benchmark datasets, a VLM judge and human review scoring the same retrievals, which lets us prioritize human review where reference data is least available or consistent.
OpenStreetMap (OSM) is an established reference dataset, but it also has gaps. We combine testing against OSM tags with a VLM judge. Vision models are strong on some concepts but can hallucinates on others or get tricked by close confounds. Where both sources disagree we invest in human review. Often times we find that our model retrieves locations that were missed on OpenStreetMap.
5. Reference-free metrics from embedding geometry
Even without any reference labels, the embedding gemoetry itself holds information that can be exploited, to prioritize review. We showed one such signal before, ELLE which captures the visual complexity of an embedded scene. Another helpful signal is how tightly the retrieved locations cluster: the 'spread'.
Spread quantifies how dispersed a seed pool is in embedding space. If the seed pool is too homogeneous there is a risk that the model has collapsed the concept onto a narrow set of exemplars as illustrated below and is not able to capture its true diversity.
Both text seeds are correct (98 and 96 of top 100) and both are narrow: a single turbine on dry rangeland with a service track. In reality, turbines exist in diverse environments: in ploughed fields, cropland and irrigated land.
Spread can tell us how good a set of embeddings is as a starting point. The broader the starting set, the more nuance of a concept the classifier can retrieve, which in turn determines the ceiling that an iterative fine-tuning workflow (like the solar farm example above) can reach. We compare embedding models on the spread of their 0-shot retrieval compared to a random set of known true exemplars.
For each concept and model, the seed is the top 100 retrieval results after 0-shot text search. Its spread is divided by the spread of 100 random real examples of the concept from OpenStreetMap, so 1 means the seed is as varied as the concept itself. Below
Why this matters
Every run we score adds to a record of which model, imagery and method has worked for which question, and every insight is kept for future reference.
Today that record guides us. Next it guides the agent. We are linking it into LGND Geo, so that when an agent picks a retrieval method it sees what has worked before on questions like yours, and where a better option exists. Stay tuned.