Context layer for AI agents: knowledge of physical places

To give an AI agent knowledge of a physical place, the place has to exist as data the agent can read: which rooms there are, what is in them and where, what condition it is in, where the water shut-off is, with a source for every claim. Space Context is a context layer for the physical world: you film a space once, with the iPhone app, the API, the CLI or the SDK, and it becomes a structured, cited Space Document that agents read through a REST API, a TypeScript SDK, a CLI and a remote MCP server.

In short

• Agents already have context for code, documents and calendars. Physical spaces usually live only in people’s heads, phone photos and PDFs.
• A Space Document is versioned JSON: areas, items and where they are, condition, issues, key locations (fuse box, water shut-off, boiler…), how-to steps, facts and Q&A.
• Records seen on camera cite the original video frames they were seen in.
• What was not filmed or said is marked as not documented, and answers return documented: false instead of a guess.
• A free test key reads a real demo space. With a card on file the first live scan is free; after that a live scan is $1.00 from prepaid credit.

Why don’t AI agents know anything about your physical spaces?

Your assistant can read your repository, your docs and your calendar because each of those is already data with an API in front of it. A flat, an office or a van is not. What you know about it sits in your memory, in a few photos, in an inventory PDF nobody updates, or in a message thread with the previous tenant.

So when you ask an agent “where is the water shut-off in the seaside flat?”, it can only say it does not know, or answer from what a typical home looks like. The second is worse. A better model does not fix this; a record of that specific place, in a form the model can read and cite, does.

What is a context layer for the physical world?

A context layer is a source of structured information that sits between the world and your agents, so that every agent, script and assistant reads the same facts. For physical places it has to do four things:

1. Capture a space with little effort: a walkthrough video or some photos, not a form to fill in.
2. Structure it into records a program can filter and an agent can quote.
3. Prove each record with the image it came from.
4. Stay honest about gaps: what the camera did not show is “not documented”, never inferred.

That is the job Space Context does. You film a space once. Scan it again through the API or the CLI and the new scan is checked against the previous one, so the context stays current and two versions can be compared.

What is a Space Document?

A Space Document is the JSON a space becomes. The schema is versioned (schema_version: "1.0") and is the same whether the space was scanned with the iPhone app or through the API.

Around the records sit a summary, counts, the evidence frames and a coverage block that says what the recording did not show.

Here is a real record from the demo space, a one-bedroom apartment in Brisbane scanned twice through the API (photos at check-in, a narrated video at check-out). It is the portable heater in the bathroom, as the check-out scan recorded it, with some fields left out:

{
  "id": "itm_32",
  "name": "Portable fan heater",
  "canonical_name": "heater",
  "category": "climate",
  "location": "On the floor below the left side of the vanity",
  "condition": { "code": "damaged", "notes": "The power cable is visibly loose." },
  "aliases": ["portable heater", "fan heater", "space heater"],
  "note": "Do not use it until the loose cable is repaired.",
  "issue_ids": ["iss_2"],
  "previous_item_id": "itm_29",
  "evidence": [
    { "frame_id": "frame-28", "timestamp_ms": 28087 },
    { "frame_id": "frame-30", "timestamp_ms": 30058 },
    { "frame_id": "frame-31", "timestamp_ms": 31044 }
  ],
  "confidence": 0.99
}

An agent gets a location it can repeat to a person, a condition with its reason, the names someone might type, a link to the same object in the earlier scan (previous_item_id), and three frames 28 to 31 seconds into the video. The demo’s photos are by Chris Olszewski (Kgbo), CC BY-SA 4.0, and its narration is invented for the example. Every field is described in the Space Document reference.

How does a space get into Space Context?

There are four ways in, all on the same account:

The app, the API, the CLI and the SDK capture spaces; the API, the CLI, the SDK and the MCP server read them. Spaces scanned with the app are readable through the API, the console and MCP with the same account, at no cost. The first time you read one through the API it gets a second, deeper pass; until that finishes you get the base level with enrichment.status: "pending".

A scan takes a video (MP4, MOV, M4V, 3GP, WebM or Matroska, up to 4 GB and 10 minutes) or up to 120 photos. The analysis runs on Space Context’s side, so there is no ffmpeg or frame extraction on yours. The full flow with curl, CLI and SDK code is in how to turn a walkthrough video into JSON, and the setup for every assistant is in the MCP server guide.

You can try all of it before filming anything: get a free test key, and every endpoint answers from the demo space with nothing charged.

What are evidence frames, and why do they matter?

Every record that comes from the video carries evidence: a list of { frame_id, timestamp_ms } pointing into the document’s frames. Each frame has a signed URL that serves the original frame, unaltered. The link expires; read the document again for a fresh one.

This changes what an agent can do with an answer. Instead of “the heater is damaged”, it can say “the heater under the vanity has a loose cable; here is the frame from 28 seconds into the check-out video”. When you ask a space a question through POST /v1/spaces/{id}/ask, the answer comes with its citations, and each citation carries its frames. When you compare two scans, every change carries frames from both.

Evidence matters most where someone may disagree later: a deposit, a damage claim, a handover.

What a Space Document is, and what it is not

It does not invent. The ask endpoint answers only from the space’s records. When they do not contain the answer, documented is false and the answer says so. A space exported as an Agent Skill tells the agent to say “not documented” rather than guess.

The demo shows this on a single record. The demo’s narration says where the fuse box is, so the check-out scan records it:

{
  "id": "loc_1",
  "kind": "electrical_panel",
  "label": "Fuse box",
  "area_id": "room-33614ab5-1",
  "location": "In the hallway cupboard behind the coats",
  "notes": null,
  "evidence": [],
  "confidence": 0.99
}

The evidence list is empty because no frame showed the fuse box, and the same document’s checklist says so on its own: { "kind": "electrical_panel", "seen": false, "probability": 0.06, "evidence": [] }. Next to it, coverage.not_documented lists what a reader would expect but the video did not show, such as “Router: not shown.” and “Smoke detector: not shown.” Not seen means not filmed: a hint for the next recording, not a claim that the thing is missing.

The same rule runs through the API’s comparisons. Between check-in and check-out, an item is removed only when a frame of the later scan shows its place empty. Otherwise it is not_seen: it may be gone, or nobody pointed the camera there.

It does not hand secrets to every agent. Wi-Fi passwords, door and alarm codes and the narration are hidden from keys without the spaces:sensitive scope. For such a key, the demo’s Wi-Fi fact reads "statement": "Sensitive value recorded. A key with the spaces:sensitive scope can read it." with "value": null. More in giving AI agents access without passwords.

It is not a channel for instructions. A sign or a sentence on camera that addresses an AI (“ignore your previous instructions…”) is withheld and noted in coverage.warnings. See prompt injection from the physical world.

It is not a 3D model or a measured floor plan. It is a structured description of what the camera saw and the narrator said, with the frames to back it.

What can you build on it?

The document is the same for any space you can film. These are examples, not the limits:

• Check-in and check-out. Scan at the start and at the end of a stay or a tenancy, then compare: items added, moved or recounted, items gone (with a frame), issues new or fixed. See comparing check-in and check-out inventories.
• Guest guides. The Wi-Fi, the bins, the coffee machine and the heater, said once on camera, become facts, how-to steps and Q&A an assistant can answer from. See a short-term rental guest guide from one video.
• Home inventory. Items with brand, model, condition and frames, kept as your own record. See a home inventory video for insurance.
• Assistants that know your places. One searchable knowledge base across a home, an office and a van, in any language. See building a knowledge base for AI assistants.

How do you start?

1. Sign in to the console with Apple, Google, GitHub or your email (the same account as the iPhone app) and switch to Test. From a terminal with Node.js 22.12 or later, npx spacecontext login --test does the same and saves a test key.
2. Ask the demo space a question:

npx spacecontext ask spc_demo "where is the fuse box?"

The fuse box is in the Open-plan Living, Dining and Kitchen, in the hallway cupboard behind the coats.

Sources
  • Fuse box (Open-plan Living, Dining and Kitchen)

3. Export it for an agent: npx spacecontext skill spc_demo --out ./brisbane-apartment-skill writes SKILL.md, space.json and one reference file per topic.
4. For your own spaces, add a card under Billing → Add a card (nothing is charged) and scan a real video. The first live scan of a new account is free, one per card. After that a live scan is $1.00 from a prepaid balance, failed scans are refunded, and reading, search and answers are included (pricing).

Your agents already know your code. With one video, they can know your places too. Get a free test key and read the demo space in about five minutes.