Resources

Sanity Checks for Geospatial Embeddings

Dingqi

6 min. read

When we train a geospatial AI model, the final output is just a huge table of numbers called embeddings. The problem is that a good model and a broken model often produce outputs that look almost identical. There are no obvious error messages, the files have the expected size, and everything appears to have worked. Yet weeks later, the model may start producing poor search or classification results, and by then it’s difficult to trace the problem back to its source.

To solve this, we treat every embedding run the way software engineers treat a new software build: before we ship it, it has to pass a series of automated tests. Every time a new batch of embeddings is produced, we run twelve automated checks. Rather than presenting them as twelve separate metrics, we organize them into four simple questions.

First, did the pipeline work correctly?

These checks catch mechanical failures before anyone looks at quality.

  • Finite values. Every vector must be free of NaN, infinite values and all-zero rows. A single bad vector fails the check.
  • Duplicate embeddings. Two different chips should never get near-identical vectors. If they do, the pipeline has probably mixed up chip IDs or reused a cached result. Any pair at or above a cosine similarity of 0.99999 fails the check.
  • Agreement with the baseline. We compare each chip's vector with its vector in the trusted baseline run. The average cosine similarity must be at least 0.999, so a run that silently changes its outputs is caught.

Figure 1. Pipeline-integrity results for one run. All three checks pass. There are no NaN, infinite or all-zero vectors. The near-duplicate score of 0.977 stays under the 0.99999 cutoff, and row-wise cosine similarity to the baseline embed

Second, is the embedding space healthy?

A model can technically produce embeddings while still collapsing into a much smaller representation than intended. These tests measure whether the model is actually using its full representational capacity. 

  • Participation ratio. The model's effective number of dimensions, as a share of the total. It must stay at or above 0.03.
  • Dead dimensions. The share of dimensions whose values barely change across chips. More than 10% fails.
  • Redundancy. The average correlation between dimensions. If dimensions mostly move together, the space carries less information than its size suggests. Above 0.5 fails.
  • Uniformity. How evenly vectors spread over the space. Lower is more even, and anything above −1 fails.

The first four checks look at the same property from different angles, so they tend to fail together. They also explain failures further down: a collapsed space can't carry land cover or terrain.

  • Norm outliers. This last check works differently. It flags individual chips whose vector length is a statistical outlier, rather than judging the shape of the whole space. More than 2% of chips fails.

Figure 2. Embedding-space health across five checks. The curve plots cumulative variance against the number of dimensions. 123 of 1,024 dimensions hold 90% of the variance, so the space is spread across many directions rather than collapse

Third, do the embeddings capture meaningful information about the real world?

A healthy space still has to encode things people care about. We evaluate whether similar land-cover types cluster together and other measurable variables, always with training and test data drawn from separate spatial blocks.

  • Semantic separability. Do chips with the same land-cover class sit near each other? We report three measures: silhouette score, k-nearest-neighbor accuracy and k-means agreement with the labels. Only kNN accuracy decides pass or fail, and it must reach 0.55. Silhouette is reported but doesn't gate, because mixed and transitional land cover keeps it low even for good embeddings.
  • Physical interpretability. A simple linear model trained on the embeddings predicts nine variables, including elevation, temperature, precipitation, soil properties, human modification and population. Elevation must reach an R² of 0.3; the other eight are reported for diagnosis.
  • Reconstruction-loss proxy. A probe on the embeddings estimates how well the original imagery could be reconstructed from them. The estimate must stay at or below 2.6.

We use simple linear models on purpose. If a simple model can read a variable out of the embeddings, the information is there in an easy-to-use form. A more powerful model could learn the task itself and hide a weak embedding.

Figure 3. k-nearest-neighbor confusion matrix for 11 land-cover classes. Rows are the true class, and columns show what a tile’s neighbors voted for. Built, bare and tree separate cleanly, while grass is often confused with crop, bare and

Figure 4. Out-of-fold linear-probe R² for nine targets. Human modification (0.82) and temperature (0.76) are the easiest to read from the embeddings. Elevation (0.36) clears the 0.3 gate, while population (0.10) falls well below it.

Finally, does the model generalize?

Instead of testing only on familiar regions, we hold out an entire geographic region and measure how much performance drops. This tells us how well the model is likely to perform in places it has never seen before.

  • Transfer robustness. We split the golden set into nine regions. For each one, we train the land-cover classifier on the other eight and test it on the held-out region. We then compare average held-out accuracy with accuracy from a random split. Held-out accuracy must be at least 70% of the random-split accuracy.

This ratio is our best estimate of how the model will do somewhere new. The per-region results also show which parts of the world it handles worst.

Figure 5. Land-cover accuracy with each of nine regions held out, against the random-split baseline of 0.61. Overall, held-out accuracy is 98% of the baseline, well above the 70% threshold. Two regions transfer worst: n_e (0.45) and m_w (0

Why the order matters

The order of these tests is intentional. If the embedding space has already collapsed, then poor semantic performance is only a symptom of a deeper problem. By checking pipeline integrity first, embedding quality second, semantic meaning third, and generalization last, we can quickly identify where a failure originates.

From research model to release

The final result is more than a simple pass or fail. Each group of tests points to a different stage of the system—whether the issue lies in data processing, model training, or deployment. That makes debugging much faster and gives us confidence that every model we release has been thoroughly validated.

In short, these sanity checks transform a research model into a production-ready system. Instead of hoping the embeddings are correct, we can demonstrate that they are numerically sound, capture meaningful geographic information, and continue to work even in parts of the world the model has never seen before.