# The deterministic, multimodal, self-evolving memory engine for autonomous AI agents

> Fractera Memory Starter

An autonomous, self-hosted long-term memory engine and the cognitive core for AI agents. Built to work as the architect's personal command centre through Telegram and a unified REST API, it closes the gap between a volatile context window and real cognitive continuity.

The engine ingests raw, unstructured real-world input — text, images, voice notes, video, whole PDF documents, Markdown and HTML pages, source code, precise spatial-temporal coordinates and dates — and turns it into an indexed knowledge graph and structured relational stores, without unnecessary model calls and without per-request token costs.

- Zero per-request fees
- Zero vendor lock-in
- Full privacy on your server

- [Read the API](https://memory.aifa.dev/en/settings?section=api)
- [Open the passport](https://memory.aifa.dev/en/passport)

## The architect's operating system

Standard RAG pipelines and vector stores make agents lose critical context at every session reset, burn compute re-reading long logs, and never synthesise personal experience over time.

Fractera Memory works as a black box engine: in go multimodal input and runtime context parameters, out come structured objects, synthesised data, verified conclusions or actionable reports. One architecture unifies four storage layers under a deterministic multi-level router.

## How a request travels

One entry point, one router, two very different costs behind it.

1. Incoming stream — text, geolocation, voice, images, video, PDF, Markdown, HTML, code, dates
2. Deterministic multi-level router
3. Levels 1–3 · direct database and graph traversal — Zero tokens, no model, sub-10 ms
4. Levels 4–5 · vector search and deep reasoning — A model turn: hypothesis chains and reports

## End-to-end schema adaptability

No manual migrations, no static schema design. The engine adapts its schema on the fly — adding columns, and generating fully typed relational SQL tables whenever new structured entities and relationships appear.

## Cost-first architecture: the cost ladder

Every request is resolved with the minimum compute that can answer it. A query escalates only when the cheaper, deterministic tiers fail to produce a complete answer.

| Level | Retrieval mechanism | Cost and purpose | Opened by |
|---|---|---|---|
| Level 1 | Direct SQL / key-value query, no model | $0 / 0 tokens. Sub-10 ms latency. Exact factual properties. | Engine router |
| Level 2 | Single-pass model call without conversation history | Minimal. Direct execution and simple parsing. | Engine router |
| Level 3 | Knowledge graph traversal plus a context session | Low. Context retrieved without generating model tokens. | Engine router |
| Level 4 | Semantic vector store retrieval | Higher. Fuzzy semantic search across historical context. | Caller — depth: deep |
| Level 5 | Bounded recursive deep reasoning, up to 10 minutes | Maximum. Multi-hypothesis research and unstated facts. | Caller — depth: extreme |

*«What is my passport number?» resolves instantly at level 1 for zero tokens. «Which of my contacts could have known this person?» escalates through levels 3–5 and comes back as a probabilistic reasoning chain.*

## Spatial-temporal context

Time and coordinates are first-class indexes here, not flat text tags.

- **The strict boundary rule.** An empty spatial-temporal scope means «location and time unknown» — never «everywhere and always».
- **Context isolation.** «Which taxi service do I usually use here?» asked in Madrid returns Madrid knowledge, and never collides with or overwrites the same question answered in London.
- **Radius search.** A built-in spatial index over lat, lon and radius_m answers proximity queries: notes, expenses and records near this point.

## Knowledge becomes an object, not a paragraph

Asked to summarise complex data — last month's spending, a project's state — memory does not hand back a wall of text. It builds the thing you asked for:

1. Instantiates a structured entity: a typed table with the columns the answer needs.
2. Compiles, sorts and formats a clean Markdown artifact with its own Object ID.
3. Returns a short executive summary next to the artifact, so the answer reads well and the detail stays referenceable.

## The memoization loop

Nothing expensive is paid for twice. Every high-cost chain is folded back down into the cheaper tiers.

1. An expensive computation or research loop runs at level 4 or 5
2. An artifact is created with its ID, alongside a concise conclusion
3. The conclusion is indexed into the vector store, the knowledge graph and the tables
4. Repeat questions are answered in 0.2 s at levels 1–3, for zero tokens

## A self-evolving skill core with shadow A/B testing

When the engine detects repeated misses or a sub-optimal path, it writes a candidate skill and runs it as a challenger in the shadow — on real production traffic, while people keep being answered by the verified champion.

- **No self-evaluation.** The model is forbidden from scoring its own work. Verdicts come from outside — explicit architect feedback and strict compute-cost ratios.
- **Deterministic promotion.** A challenger is promoted to champion only when it wins on external quality metrics with no regression in speed or cost.
- **Versioning and safe rollback.** Every modification is a commit. One click reverts the instructions to the baseline version through Git, with no data loss.

## Four unified storage tiers

Four layers, one contract. The caller never learns which of them answered.

- **Relational store.** Tabular structures, typed facts, exact entity properties.
- **Vector store.** High-dimensional semantic embeddings for fuzzy similarity search.
- **Knowledge graph.** Directional links between entities, people and events.
- **Object store.** Whole files on your server — images, audio, video, PDF, Markdown, HTML, code — each next to its full description.

## Native multimodality

Not a preprocessor bolted on the side. The pipeline lives inside the engine.

- **Audio.** Speech-to-text by OpenAI whisper-1, with a timestamp on every segment.
- **Images.** Read by a vision model: every element, its position, colours and all visible text.
- **Video.** The sound track is transcribed and six frames are read — one timeline, frames between the lines.
- **PDF.** The document is read whole: its structure and its content, tables row by row.
- **Markdown and HTML.** Markdown is kept as the document it renders into; HTML as a page and as its source.
- **Source code.** Source code is described — purpose, structure, exports — with the source verbatim, and never run.

## Testing and verification in the built-in playground

The engine ships with an interactive bench. It is not a demo page: it is where an integration is proven before it is written.

- Execute direct API requests against the memory core with no front-end abstraction in the way.
- Inspect raw JSON payloads, execution timings and exact model token usage.
- Verify the request body before committing a line of client code.

`/{lang}/settings?section=memory-test`

## How it compares

Two comparisons: one against the categories of memory tooling, one against a ready-made assistant of a different philosophy.

### Against the categories

| Capability | Fractera Memory | Standard RAG frameworks | MemGPT / Letta | Mem0 / Zep |
|---|---|---|---|---|
| Storage architecture | Hybrid: graph + vector + relational + object store | Vector DB only | Relational / text files | Vector plus a basic graph |
| Zero-token reads | Yes — deterministic paths at levels 1–3 | No | No | Partial |
| Native multimodality | Built in: audio, video, images, PDF, Markdown, HTML, source code | Requires external parsers | Requires external parsers | Text focused |
| Spatial proximity indexing | Native lat/lon radius search | Text matching only | Function calling only | Basic metadata |
| Skill evolution | Champion / challenger A/B testing | None | Manual prompt edits | None |
| Self-hosted / open source | 100% on-premise, single node | Varies | Yes | Freemium / cloud |

### Against a ready-made assistant

| Capability | Fractera Memory | IVA Agent (smixs/iva-agent) |
|---|---|---|
| System classification | An autonomous memory engine behind an API, for any front-end | An end-to-end Telegram assistant tied to an Obsidian vault |
| Architecture | A decoupled microservice; the Telegram bot is an optional client | A monolith: Telegram, userbot and vault manager in one codebase |
| Cost optimisation | A five-tier deterministic router; instant zero-token reads | Every operation leans on model passes, BM25 and vector lookups |
| Data processing | Dynamic SQL tables, structured artifacts, a knowledge graph | Markdown cards written to a folder for Obsidian to sync |
| Integrations | Many front-ends at once over one REST API | Bound to one Telegram account and an Obsidian setup |

## API quickstart

One REST API, one key. Every example below runs against a live instance as it stands.

### Store a voice note with spatial coordinates

```bash
curl -X POST https://memory.your-domain.com/v1/remember \
  -H "Content-Type: application/json" -H "x-memory-key: YOUR_MEMORY_KEY" \
  -d '{
    "who": "roman",
    "text": "Office lease note",
    "media": [{ "url": "https://.../note.oga" }],
    "scope": [{ "at": "2026-09-11", "lat": 40.4168, "lon": -3.7038, "radius_m": 500 }]
  }'
```

### Recall everything within a radius

```bash
curl -X POST https://memory.your-domain.com/v1/recall \
  -H "Content-Type: application/json" -H "x-memory-key: YOUR_MEMORY_KEY" \
  -d '{
    "who": "roman",
    "text": "What notes or files did I save within 500 meters of here?",
    "scope": [{ "lat": 40.4168, "lon": -3.7038, "radius_m": 500 }]
  }'
```

### Deep reasoning with the chain returned

```bash
curl -X POST https://memory.your-domain.com/v1/recall \
  -H "Content-Type: application/json" -H "x-memory-key: YOUR_MEMORY_KEY" \
  -d '{
    "who": "roman",
    "text": "Summarize all my taxi expenses from last month into a table",
    "depth": "deep",
    "want_chain": true
  }'
```

## Installation

There is exactly one thing to know about installing this.

One run of the Fractera installer robot on your own server brings up every microservice of the platform, memory included — nginx, certificates and the access key are arranged for you. There is nothing to assemble by hand.

## Design principles



- **Complete data ownership.** All data, graphs and media stay strictly on your machine. No telemetry, no hidden cloud dependency.
- **Fact attribution.** What a person stated is logged as fact (said); what the engine inferred is flagged as hypothesis (guess) and stored only with its evidence (basis).
- **Headless engine architecture.** Connect the official Telegram starter, or attach your own web chat, mobile app and automation pipelines over HTTP. The bundled console is a microservice of its own, and it is optional.

## Questions and answers

Short answers to what people ask before they integrate.

### Does every request cost tokens?

No. The engine answers levels 1 to 3 without a model at all: a direct lookup, a graph traversal, a conclusion already folded back into the stores. A model turn is spent only when the cheap deterministic paths return nothing, and the answer reports depth_used so you can see what you paid for.

### Can it answer questions about a place by coordinates, not by a word?

Yes. A scope entry carries lat, lon and an optional radius_m, and the coordinates are spatially indexed. You can ask what you know within 500 metres of a point, and knowledge recorded in Madrid never merges with knowledge recorded in London.

### What can I send besides text?

Voice notes, images, video, PDF, Markdown, HTML and source code (TypeScript, Python, SQL and more). Every file gets a full description detailed enough for another AI to reconstruct it, and a summary of about 50 words: speech is transcribed by OpenAI whisper-1 with timestamps, a video is split into its sound track and frames on one timeline, and pictures, documents, pages and code are read by Claude — code is never executed. The original stays in the built-in object store next to its full description and is referenced from answers by id.

### What schema do I have to design first?

None. You send a sentence. The engine adds columns as new kinds of fact appear and generates typed relational tables when a kind grows into an entity. There are no migrations to write.

### What happens after an expensive research run?

It folds the result back. The artifact goes to the object store, its summary into text, into the vector store and into the knowledge graph, and the relation tables are updated. The same question is then answered from the cheap levels, in fractions of a second.

### How does it improve itself without breaking what works?

It writes a second version of the skill and runs it as a challenger in the shadow, on real traffic, while people keep being answered by the champion. Promotion needs an external verdict and no regression in cost: the engine is never allowed to grade its own work.

### What can I connect to it?

Any HTTP client: a Telegram bot, a web chat, a mobile app, a scheduled job. The engine also ships with its own console, already connected, and that console is optional: nothing in the API path depends on it.

### Where does my data live?

On your server, in your database, in your object store, behind a key you can revoke in one click. There is no metered API in the middle and no telemetry leaving the machine.

## The Fractera project on GitHub

Fractera Memory is one microservice of the Fractera platform, the engineering infrastructure for autonomous agents. The whole project, this engine included, is open source.

- https://github.com/Fractera/Agentic-Engineering-Infrastructure

---

Source page: https://memory.aifa.dev/en