Evaluation of Agentic Systems
for Geospatial Analysis

Developing a Benchmark
for vector-based GIS tasks

Daniel Allmer

What are agents?

Large Language Models (LLMs) encode knowledge and can reason - but can't act in the world.
Tool use turns an LLM into an agent.
- Tools include writing and executing Python code.
- A harness runs tools and steers the model - via system prompt and tool design.
- Popular examples: Microsoft Copilot, Codex, Claude Code, OpenClaw.
The agent runs in a loop until the task is finished.
Traditional desktop software gets augmented or replaced with agents.

How well do they work?

New models drop constantly - no single number captures what each is good at.
Models compete on knowledge, intelligence, price and speed.
Benchmarks measure specific capabilities: coding (SWE-bench), reasoning (ARC-AGI), and many more.
For geospatial analysis: no established benchmark yet.

Towards a geospatial benchmark

I set out to design such a benchmark, and answer these questions:

How can one construct a representative suite of Vector-GIS tasks?
How can geospatial output be graded objectively?
Can a benchmark be used to drive measurable improvements on agents?

Coverage

Regions	cape-town 5 vienna 5 cairo 4 paris 4 tokyo 4 bangkok 3 lagos 3 new-york 3 london 2 antarctica 1 fiji 1 svalbard 1
Data sources	bundled 31 osm-overpass-current 3 overture-current 2 geofabrik-pbf 1 osm-overpass-historical 1
Formats (in)	geojson 14 gpkg 7 geoparquet 6 csv-wkt 2 flatgeobuf 2 shapefile 2 csv 1 kml 1
Formats (out)	geojson 12 geoparquet 8 gpkg 7 csv 5 json 4 parquet 2
CRS variants	wgs84 29 conformal 23 equal-area 2 polar 2 antimeridian 1 web-mercator 1
Geometry types	polygon 24 point 18 multipolygon 13 linestring 9 multilinestring 2 multipoint 1
Data scale	small 18 medium 11 large 1
Quality issues	encoding 3 inconsistent-values 3 multi-single-coercion 3 null-attributes 3 type-coercion 2 invalid-geometries 2 slivers 2 duplicates 1 empty-geom 1 mixed-crses 1 mixed-geom 1 shp-truncation 1 unsnapped-vertices 1 wrong-ring-order 1 zero-area 1
Geometric ops	bounding-box 6 collect 5 buffer 4 intersection 4 dissolve 3 difference 2 sym-difference 2 union 2 centroid 1 clip 1 convex-hull 1 explode 1 point-on-surface 1 simplify 1
Spatial-analysis ops	aggregation 4 join-within 3 closest-facility 1 distance-matrix 1 hotspot 1 isochrone 1 knn 1 nn 1 network-distance-matrix 1 pip-count 1 shortest-path 1 join-contains 1 join-crosses 1 join-intersects 1 join-overlaps 1 join-touches 1 within-distance 1
Overture themes	base.land_use 4 places.place 4 buildings.building 3 divisions.division_area 3 base.water 2 addresses.address 1 base.bathymetry 1 base.land 1 base.land_cover 1 divisions.division 1 transportation.connector 1 transportation.segment 1
OSM tag families	amenity 5 boundary-admin 5 highway 5 building 3 railway 3 landuse 1 leisure 1 natural 1 place 1 public-transport-routes 1 shop 1 waterway 1

Benchmark organization

code	category	what we test
dd	Data Discovery & Fetching	Overture · Overpass · Geofabrik · attic queries
fio	Format I/O & Conversion	GeoJSON · GPKG · Shapefile · Parquet · CSV/WKT · KML
crs	Coordinate System Transforms	WGS84 · UTM · equal-area · national grids · antimeridian
geo	Geometric Operations	Buffer · centroid · hull · simplify · dissolve · explode
spa	Spatial Analysis	Joins · nearest-neighbour · PIP · hot-spot · routing
dc	Data Cleaning	Invalid geometries · slivers · duplicates · encoding

Difficulties

L1 single operation 18 tasks

buildings.shp → centroid → centroids.geojson

L2 compose 2-3 operations 12 tasks

shops + anchors → 5-NN → distance matrix → normalise names → market.json

L3 multi-step pipeline 6 tasks

incidents.csv → Overpass fetch → routing graph → shortest path →

↪ distance matrix → isochrones → routing.gpkg

Example Task: fio-l3-vienna-geofabrik-highways

Building a traffic-noise model of Vienna's Gürtel corridor. Pull every highway from the current Geofabrik Austria PBF that intersects a 500 m buffer around the Gürtel — the ways whose name ends in “Gürtel”.

Plus every public-transport route relation (bus, tram, subway, train…) crossing that buffer, each as one feature. Both into vienna_network.gpkg — highways LineString, routes MultiLineString, in Austria's standard projected CRS.

On it.

Step by step

Download Austria PBF - 765 MB!
Find Gürtel ways (name ends “Gürtel”)
Buffer 500 m in a metric CRS
Filter highways intersecting buffer
Assemble public-transport relations
Write 2-layer GPKG in EPSG:31287

Models under Test

Gemma 4

lightweight baseline

Vendor: Google · Apache 2.0
Parameters: 26 B total / 4 B active
Architecture: Mixture-of-Experts
Context: 256 K tokens
Reasoning: configurable thinking mode
Deployment: consumer GPUs (quantized)

DeepSeek V4 Flash

efficient mid-size

Vendor: DeepSeek · MIT
Parameters: 284 B total / 13 B active
Architecture: Mixture-of-Experts
Context: 1 M tokens
Reasoning: adjustable effort (high–max)
Deployment: 2× H200 (FP4/FP8)

Gemma 4

Plan — parse the Geofabrik PBF with osmium.

The right tool for the .pbf export.

PBF stalls — 765 MB parse hits the timeout (2 min).

Never gets through the file.

Overpass fallback — switches to the API; Error codes 406/504 prevent access.

Abandons the requested source.

Fabricate — hand-writes fake “Gürtel” ways + U6/60A routes.

Made-up coordinates — a “synthetic dataset.”

Game the test — right layers, columns & “Gürtel” names clear the gate.

Admits to fabrication, but still fails the test.

duration27 min steps28 cost$0.05

50partial score

DeepSeek V4 Flash

Plan — drafts the full osmium pipeline before touching the data.

Even infers the CRS (EPSG:31287) from “Austria's standard.”

PBF stalls — same issue, the 765 MB parse times out too.

But raises its own run timeout to 10 min.

Intelligent filtering — find Gürtel streets first, then only search near them.

Skips the millions of ways elsewhere in Austria.

Graz shows up — “…Gürtel” also names Graz streets; model notices.

Buffer had spanned both cities. Model excludes everything outside Vienna.

Verify — reads the output back before stopping.

Counts within ±15 % of reference → 100.

duration55 min steps33 cost$0.05

100 perfect score

Grading

Hard gate format_schema_valid

gpkg present, both layers readable
required columns (untruncated)
usable projected CRS

15 subchecks

highway_count±15 % · ×3

pt_route_count±15 % · ×3

highway_coords_rangein EPSG:31287 env · ×3

pt_route_projectedmetric coords · ×2

hw_geometry_typeLineString

pt_geometry_typeMultiLineString

pt_multilinestring≥ 90 % multi

minimum_feature_countshw ≥ 100, pt ≥ 5

diacritics_preserved“ürtel” survives

hw_highway_attr_populated≥ 70 % non-empty

pt_route_attr_populated≥ 70 % non-empty

hw_type_diversityprimary/secondary/…

pt_route_type_diversitybus + tram

crs_is_canonicalEPSG:31287

crs_in_meaningful_set∈ Austria CRS set

Per-task scores

Gemma 4 26B

	dd	fio	crs	geo	spa	dc
L1	100	99	100	100	74	87
	100	98	100	95	100	83
	100	96	100	90	100	93
L2	93	98	62	93	90	93
L2	42	76	94	40	55	95
L3	23	0	36	26	59	9

DeepSeek V4 Flash

	dd	fio	crs	geo	spa	dc
L1	100	99	60	100	100	84
	85	100	100	95	100	100
	100	99	100	100	91	100
L2	98	94	100	84	100	95
L2	56	98	96	63	85	100
L3	100	74	73	74	87	39

Mean score over 5 runs per task · basic prompt

Agentic failure modes

Coordinates from memory

LLMs don't recall coordinates with high precision — must always ground in real data

Wrong coordinate systems

sometimes picks a conformal CRS for area calculations, or adjacent UTM zones

Operational robustness

agent struggles in real-world environments with large queries, timeouts and infrastructure errors

Schema violations

sometimes uses different file or column names, or other datatypes than requested

Degenerate repetition

small models get stuck in loops in longer workflows

Fabrication under pressure

models hallucinate or use fake data when they stop making progress on a task

Two system prompts

You are a GIS analyst. Solve the task by writing and executing Python scripts in your working directory. The instruction tells you what file to produce and where.

Your working directory is /work. All file tool paths must be absolute (e.g. /work/solve.py).

Running Python code

Write your script to a file (e.g. /work/solve.py) and run it with python solve.py. The common GIS libraries (geopandas, shapely, pyproj, duckdb, …) are already installed. If you need an extra package, install it with pip install --user <name>.

You are a GIS analyst agent. Solve geospatial analysis tasks by writing Python scripts and running them with python.

Working environment

Your working directory is /work. All file tool paths must be absolute (e.g. /work/solve.py).
Your working directory contains any input files the task uploaded. Read them by name, exactly as the instruction refers to them.
Outputs MUST land in the working directory under the filename the instruction specifies.

Tools

Read — inspect a file's content. Use this before Edit on any file you didn't author this session.
Write — create a new file or fully overwrite one. This is how you author your solve.py.
Edit — make a targeted substitution in a file you've already Read; for larger rewrites prefer Write.
Bash — run shell commands: execution (python solve.py), inspection (ls, wc -l), one-off CLI tools.

Running scripts

Write a regular Python script and run it with python solve.py. The common GIS libraries listed below are already installed in the system interpreter — no venv, no PEP 723 metadata, no uv involved. If you need an extra package, install it with pip install --user <name>.

Common libraries (pre-installed)

geopandas, shapely	Vector geometry I/O and ops
pyproj	CRS transforms
pyogrio, fiona	Fast vector read/write
pandas, pyarrow	Tabular + Parquet
duckdb	SQL over geo files; load the spatial extension to read GeoParquet/Shapefile/GPKG
osmium	Read OSM PBF files (Geofabrik extracts etc.)
osmnx	OSM network analysis; Overpass queries
overturemaps	Overture Maps download (Python + CLI); wraps the Overture GeoParquet bucket so you don't assemble S3 paths by hand.

External data sources

When fetching from an external API (Overpass, Overture, Geofabrik, etc.):

Retry at most 3 times with brief backoff.
If still failing, try one alternative endpoint if known — e.g. Overpass mirrors overpass.kumi.systems and lz4.overpass-api.de.
If all attempts fail, STOP and report the upstream error.

Before ending your turn

Re-read the user's most recent prompt.
For every output filename, column name, and CRS the prompt names: verify the file on disk matches character-for-character.
ls the working directory to confirm each named output file exists.
Sanity-check coordinate magnitudes: degrees are ~−180…180; metric CRSs produce 10⁵–10⁶ for regional data. A mismatch means the CRS is wrong.

If any item fails, fix it before stopping.

The 2 × 2 matrix: each model run under both prompts

Prompt effect per task

Gemma 4 26B

	dd	fio	crs	geo	spa	dc
L1	0	+1	0	0	+26	0
	0	−2	0	−7	0	−16
	0	+1	0	0	0	+7
L2	−2	−3	0	+2	−6	+1
L2	+31	+18	−5	+40	+18	−10
L3	−7	+10	−14	+15	−25	−1

DeepSeek V4 Flash

	dd	fio	crs	geo	spa	dc
L1	0	+1	0	0	0	+7
	+7	0	0	0	0	0
	0	0	0	0	0	0
L2	+2	−3	−8	+8	0	−12
L2	+34	−2	+2	0	−8	0
L3	−12	+10	−11	−4	−34	−14

Δ mean task score, basic → gis_detailed · blue = improved, red = regressed

Does the prompt help?

Gemma 4 26B

100

basic77.8

detailed79.8

DeepSeek V4 Flash

100

basic89.7

detailed88.6

Mean score over 5 runs · error bars = std-dev of the per-run mean

Each gap is within one standard deviation — the prompt effect is not statistically significant.
Welch's t-test: Gemma p = 0.18, DeepSeek p = 0.57 (both > 0.05)

Related Work

Benchmark	Released	Tasks	Geo	Agent loop	Multi-turn	Runs code	Live data	Graded on
SWE-bench	2024	2,294	✗	✗	✗	✓	✗	test execution
BFCL	2025	5,551	✗	◑	✓	◑	✗	tool-call match
MapEval	2025	700	✓	◑	✗	✗	✗	multiple-choice answer
GeoAnalystBench	2025	50	✓	✗	✗	✗	✗	workflow + code similarity
GeoBenchX	2025	202	✓	✓	✗	✗	✗	tool-call seq (judge)
Cloud-Based Geo Bench	2025	45	✓	◑	✗	✓	✗	numeric answer
Bench4GeoCode	2026	100	✓	✗	✗	✓	✗	code output + judge
GeoAgentBench	2026	36	✓	✓	✗	✓	✓	file output (geometry)

Evaluation of Agentic Systems for Geospatial Analysis

Daniel Allmer · daniel.allmer@axtesys.at

Geoinformation Group, TU Graz

Supervisor: Ivan Majić

Summary

RQ1

How can one construct a representative suite of Vector-GIS tasks?

The taxonomy combination approach is feasible; task complexity is highly relevant.

RQ2

How can geospatial output be graded objectively?

Grading on geometry (IoU) works, but other facets (CRS, attribute selection, data wrangling) are equally important.

RQ3

Can a benchmark be used to drive measurable improvements on agents?

Improving agent performance from prompting is not straightforward. Model capability seems the deciding factor.

Agentic GIS is feasible, but requires a strong harness to avoid infrastructure pitfalls.

Code & tasks

github.com/pothos-dev/
geoagentbench

harness, 36 tasks, graders

Live eval UI

geoagentbench.pothos.dev

browse runs & results

Questions?

Benchmark limitations

Limited scale 36 tasks

focused on atomic operations and few-step chains

Slow evaluation up to 4h / run

code execution in the agent loop makes a full run expensive to repeat

LLM-authored tasks + review

model-generated then human-reviewed; not drawn from real analyst workloads

Reference correctness synthetic data

references are model-generated, then checked by self-consistency and grader self-tests

Single-turn only 1 prompt

no ambiguity or clarification dialog by design, tasks must be fully described in one prompt

Live-data drift Overture / OSM

references are regenerated with drift-tolerant checks, but live data might change over time

	dd	fio	crs	geo	spa	dc
L1	100	99	100	100	74	87
	100	98	100	95	100	83
	100	96	100	90	100	93
L2	93	98	62	93	90	93
L2	42	76	94	40	55	95
L3	23	0	36	26	59	9

	dd	fio	crs	geo	spa	dc
L1	100	99	60	100	100	84
	85	100	100	95	100	100
	100	99	100	100	91	100
L2	98	94	100	84	100	95
L2	56	98	96	63	85	100
L3	100	74	73	74	87	39

	dd	fio	crs	geo	spa	dc
L1	100	99	100	100	74	87
	100	98	100	95	100	83
	100	96	100	90	100	93
L2	93	98	62	93	90	93
L2	42	76	94	40	55	95
L3	23	0	36	26	59	9

	dd	fio	crs	geo	spa	dc
L1	100	99	60	100	100	84
	85	100	100	95	100	100
	100	99	100	100	91	100
L2	98	94	100	84	100	95
L2	56	98	96	63	85	100
L3	100	74	73	74	87	39

Evaluation of Agentic Systemsfor Geospatial Analysis

What are agents?

How well do they work?

Towards a geospatial benchmark

Coverage

Benchmark organization

Categories

Difficulties

Example Task: fio-l3-vienna-geofabrik-highways

Step by step

Models under Test

Gemma 4

DeepSeek V4 Flash

Grading

Per-task scores

Agentic failure modes

Two system prompts

Running Python code

Working environment

Tools

Running scripts

Common libraries (pre-installed)

External data sources

Before ending your turn

Prompt effect per task

Does the prompt help?

Related Work

Summary

Benchmark limitations

Evaluation of Agentic Systems
for Geospatial Analysis

	dd	fio	crs	geo	spa	dc
L1	100	99	100	100	74	87
	100	98	100	95	100	83
	100	96	100	90	100	93
L2	93	98	62	93	90	93
L2	42	76	94	40	55	95
L3	23	0	36	26	59	9

	dd	fio	crs	geo	spa	dc
L1	100	99	60	100	100	84
	85	100	100	95	100	100
	100	99	100	100	91	100
L2	98	94	100	84	100	95
L2	56	98	96	63	85	100
L3	100	74	73	74	87	39