> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olira.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Backfilling historical data

> Load large volumes of past data in one controlled job with historical ingestion: JSONL upload, review, confirmation, replay, and view backfill.

Use **historical ingestion** when you need to load **large volumes** of past data in one controlled job (months or years of records) instead of streaming everything through the live logging API. Your data should already match Olira's expected **log types and payloads** (same shapes as live `log` / `log_batch`); this path does not transform arbitrary external formats.

## How it differs from live logging

Use the same log **payloads** as **`log`** / **`log_batch`**; historical ingestion runs them through a **job** instead of streaming each batch to the live endpoint.

| Aspect                 | Live logging                        | Historical ingestion                                                    |
| ---------------------- | ----------------------------------- | ----------------------------------------------------------------------- |
| **Purpose**            | Ongoing, real-time or small batches | One-time or bulk backfill                                               |
| **HTTP endpoint**      | `POST /v1/logs/batch`               | `POST /v1/ingestion/jobs`                                               |
| **How you call it**    | SDK (`log`, `log_batch`) or REST    | SDK, CLI, or REST                                                       |
| **When state updates** | As each batch is processed          | After bulk insert, then a **replay** step applies logs to patient state |
| **Tracking**           | Fire-and-forget                     | **Job record** with stage, progress, and error summary                  |

## What you need

1. **Credentials**: an API key with the **`sdk:historical-ingest`** scope. Create the key in the Console or with `olira keys create --scopes sdk:historical-ingest` (see the [CLI key commands](/cli/keys) and [Authentication](/authentication)).
2. **Input**: a **JSONL file** of **patient and/or log** records (one JSON object per line). Every `patient_id` in the log rows must refer to a patient that **already exists** in your organization or is created by a patient row earlier in the same file (see below). Log `patient_id` can be an Olira id or an external identifier you map during the job.
3. **Optional `idempotency_key` on the job**: a stable string you choose (e.g. `epic-export-2026-q1`). It prevents accidentally starting **duplicate jobs** for the same intended upload. Optional **per-log** `idempotency_key` fields in the file prevent duplicate rows if you retry after a partial failure.

## Before you start a job

Work through these steps in order:

<Steps>
  <Step title="Create patients in Olira">
    Provision patients that map to your internal patient models so identifiers
    exist before you load historical logs. See
    [Logging from your codebase](/send-data/logging-from-codebase) for batch
    patient creation.
  </Step>

  <Step title="Convert your data">
    Convert your data to Olira's expected log schemas and write them to a JSONL
    file suitable for ingestion. *Coming soon:* MCP tools and resources for
    coding agents to make this refactor and validation loop easier.
  </Step>

  <Step title="Start an ingestion job">
    Upload that file via the Python SDK or CLI (see below).
  </Step>
</Steps>

## Starting a job

During **job creation** (SDK or CLI), you might get errors, for example **patient IDs that do not exist** in your organization, or **log payloads that fail schema validation**. Treat those responses as signals to fix data or patient provisioning before retrying.

### Python SDK file upload

Point at a local JSONL path; the SDK can stream it to storage and open the job in one call:

```python theme={null}
import olira, time

job = olira.create_ingestion_job(
    file="historical_logs.jsonl",
    idempotency_key="initial-onboarding-2026",  # optional but recommended
    require_confirmation=True,                  # default: pause for review before replay
)
```

### CLI (data teams, large files)

For teams that prefer the terminal, **`olira ingest upload`** streams the file, creates the job, and can poll through confirmation:

```bash theme={null}
olira login
olira ingest upload historical_logs.jsonl \
  --idempotency-key "initial-onboarding-2026"
# Optional: --no-confirm to skip the review pause (same as require_confirmation=false)
```

Use an API key with **`sdk:historical-ingest`** for both SDK and CLI ingestion commands.

## Pipeline phases and optional confirmation

By default, **`require_confirmation=True`** splits work into two phases:

**Phase 1 (automatic):** validate → insert historical logs in bulk (**no** full patient-state update yet). The job then moves to **`AWAITING_CONFIRMATION`** so you can review counts and errors before the expensive replay.

**Phase 2 (after you confirm):** replay logs into the graph in timestamp order, then run **view backfill** so rendered summaries catch up.

If you set **`require_confirmation=False`**, the job runs straight through without pausing.

## Review before replay

While the job is **`AWAITING_CONFIRMATION`**, use the **Console** to inspect the job and the logs that were uploaded, or **query the job with the Python SDK** for the same QA pass. After you **confirm**, replay runs and **patient state starts updating** from the ingested history.

During review, or when **confirming** the ingestion job, you can optionally choose which **`view_types`** should be **backfilled** as part of this run; you can also defer view backfill and run it later if you prefer.

* **Logs processed**: how many rows succeeded in the bulk load.
* **`logs_failed`** and **`error_summary`**: per-line or per-batch validation issues; decide whether to fix the file and start a new job, or proceed if errors are acceptable.
* **Confirm** to start replay and view backfill, or **cancel** if something looks wrong.

<CodeGroup>
  ```python Python SDK theme={null}
  job = olira.get_ingestion_job(job_id=job.job_id)
  # Inspect job.logs_processed, job.logs_failed, job.error_summary (and related counters)

  olira.confirm_ingestion_job(job_id=job.job_id)   # proceed to replay + view backfill
  # or
  olira.cancel_ingestion_job(job_id=job.job_id)   # stop; retention of created patients vs rolled-back data depends on job options (e.g. rollback_on_cancel)
  ```

  ```bash CLI theme={null}
  olira ingest list
  olira ingest status <job-id>
  olira ingest confirm <job-id>
  olira ingest cancel <job-id>
  ```
</CodeGroup>

While paused, ingestion status and counts may also appear in the **Console** for your organization. A job left in **`AWAITING_CONFIRMATION`** beyond the platform timeout (typically **7 days**) may be **auto-cancelled**, so confirm or cancel promptly.

**Idempotency behavior (high level):** submitting the same **job** `idempotency_key` while another job is **running** should be rejected as a duplicate. If a previous job **completed**, reusing the same key is rejected. If it **failed**, a new job may be allowed; per-log keys still deduplicate rows already stored.

## Track progress

During **QUEUED**, **VALIDATING**, **INSERTING\_…**, **REPLAYING**, and **BACKFILLING**, poll the job:

```python theme={null}
while job.status not in ("completed", "failed", "awaiting_confirmation"):
    time.sleep(5)
    job = olira.get_ingestion_job(job_id=job.job_id)
    print(job.stage, job.progress_pct)
```

Use **`stage`** and **`progress_pct`** (and CLI status output) for dashboards and runbooks. When the historical job reaches **completed**, patient state reflects replayed events; **view** regeneration may continue under a linked **view backfill** job; poll the same job resource if your API surfaces nested backfill progress.

## File format (JSONL)

Each line is one JSON object with a **`type`** of `"patient"` or `"log"` and a **`data`** object. Log rows carry **`event_type`** (a value from the [log types catalog](/reference/log-types/symptom-reports)), `patient_id`, `timestamp`, the type-specific `payload`, and optional `idempotency_key` and `trace`. Patient rows carry the patient-creation fields, including `external_identifiers` used to resolve `patient_id` references in later log rows. Declare patients before the logs that reference them; sort logs chronologically per patient if you want deterministic replay order (the pipeline also sorts by timestamp before replay).

```json theme={null}
{"type": "patient", "data": {"first_name": "Ada", "last_name": "Example", "date_of_birth": "1975-06-15T00:00:00Z", "timezone": "America/New_York", "external_identifiers": [{"system": "emr", "value": "MRN-104"}]}}
{"type": "log", "data": {"event_type": "symptom_report", "patient_id": "MRN-104", "timestamp": "2025-11-02T09:00:00Z", "payload": {"symptom": "fatigue", "score": 6}}}
```

Validate the file locally with [`olira validate`](/cli/validate) before uploading.

## Auditing and retention

Uploaded files are stored for **audit** (lifecycle policies such as transition to cold storage and long retention may apply). Plan uploads accordingly.