Skip to main content
Use historical ingestion when you need to load large volumes of past data in one controlled job (months or years of records) instead of streaming everything through the live logging API. Your data should already match Olira’s expected log types and payloads (same shapes as live log / log_batch); this path does not transform arbitrary external formats.

​
How it differs from live logging

Use the same log payloads as log / log_batch; historical ingestion runs them through a job instead of streaming each batch to the live endpoint.
AspectLive loggingHistorical ingestion
PurposeOngoing, real-time or small batchesOne-time or bulk backfill
HTTP endpointPOST /v1/logs/batchPOST /v1/ingestion/jobs
How you call itSDK (log, log_batch) or RESTSDK, CLI, or REST
When state updatesAs each batch is processedAfter bulk insert, then a replay step applies logs to patient state
TrackingFire-and-forgetJob record with stage, progress, and error summary

​
What you need

  1. Credentials: an API key with the sdk:historical-ingest scope. Create the key in the Console or with olira keys create --scopes sdk:historical-ingest (see the CLI key commands and Authentication).
  2. Input: a JSONL file of patient and/or log records (one JSON object per line). Every patient_id in the log rows must refer to a patient that already exists in your organization or is created by a patient row earlier in the same file (see below). Log patient_id can be an Olira id or an external identifier you map during the job.
  3. Optional idempotency_key on the job: a stable string you choose (e.g. epic-export-2026-q1). It prevents accidentally starting duplicate jobs for the same intended upload. Optional per-log idempotency_key fields in the file prevent duplicate rows if you retry after a partial failure.

​
Before you start a job

Work through these steps in order:
1

Create patients in Olira

Provision patients that map to your internal patient models so identifiers exist before you load historical logs. See Logging from your codebase for batch patient creation.
2

Convert your data

Convert your data to Olira’s expected log schemas and write them to a JSONL file suitable for ingestion. Coming soon: MCP tools and resources for coding agents to make this refactor and validation loop easier.
3

Start an ingestion job

Upload that file via the Python SDK or CLI (see below).

​
Starting a job

During job creation (SDK or CLI), you might get errors, for example patient IDs that do not exist in your organization, or log payloads that fail schema validation. Treat those responses as signals to fix data or patient provisioning before retrying.

​
Python SDK file upload

Point at a local JSONL path; the SDK can stream it to storage and open the job in one call: 
import olira, time

job = olira.create_ingestion_job(
    file="historical_logs.jsonl",
    idempotency_key="initial-onboarding-2026",  # optional but recommended
    require_confirmation=True,                  # default: pause for review before replay
)

​
CLI (data teams, large files)

For teams that prefer the terminal, olira ingest upload streams the file, creates the job, and can poll through confirmation: 
olira login
olira ingest upload historical_logs.jsonl \
  --idempotency-key "initial-onboarding-2026"
# Optional: --no-confirm to skip the review pause (same as require_confirmation=false)
Use an API key with sdk:historical-ingest for both SDK and CLI ingestion commands.

​
Pipeline phases and optional confirmation

By default, require_confirmation=True splits work into two phases: Phase 1 (automatic): validate → insert historical logs in bulk (no full patient-state update yet). The job then moves to AWAITING_CONFIRMATION so you can review counts and errors before the expensive replay. Phase 2 (after you confirm): replay logs into the graph in timestamp order, then run view backfill so rendered summaries catch up. If you set require_confirmation=False, the job runs straight through without pausing.

​
Review before replay

While the job is AWAITING_CONFIRMATION, use the Console to inspect the job and the logs that were uploaded, or query the job with the Python SDK for the same QA pass. After you confirm, replay runs and patient state starts updating from the ingested history. During review, or when confirming the ingestion job, you can optionally choose which view_types should be backfilled as part of this run; you can also defer view backfill and run it later if you prefer.
  • Logs processed: how many rows succeeded in the bulk load.
  • logs_failed and error_summary: per-line or per-batch validation issues; decide whether to fix the file and start a new job, or proceed if errors are acceptable.
  • Confirm to start replay and view backfill, or cancel if something looks wrong.
job = olira.get_ingestion_job(job_id=job.job_id)
# Inspect job.logs_processed, job.logs_failed, job.error_summary (and related counters)

olira.confirm_ingestion_job(job_id=job.job_id)   # proceed to replay + view backfill
# or
olira.cancel_ingestion_job(job_id=job.job_id)   # stop; retention of created patients vs rolled-back data depends on job options (e.g. rollback_on_cancel)
While paused, ingestion status and counts may also appear in the Console for your organization. A job left in AWAITING_CONFIRMATION beyond the platform timeout (typically 7 days) may be auto-cancelled, so confirm or cancel promptly. Idempotency behavior (high level): submitting the same job idempotency_key while another job is running should be rejected as a duplicate. If a previous job completed, reusing the same key is rejected. If it failed, a new job may be allowed; per-log keys still deduplicate rows already stored.

​
Track progress

During QUEUED, VALIDATING, INSERTING_…, REPLAYING, and BACKFILLING, poll the job: 
while job.status not in ("completed", "failed", "awaiting_confirmation"):
    time.sleep(5)
    job = olira.get_ingestion_job(job_id=job.job_id)
    print(job.stage, job.progress_pct)
Use stage and progress_pct (and CLI status output) for dashboards and runbooks. When the historical job reaches completed, patient state reflects replayed events; view regeneration may continue under a linked view backfill job; poll the same job resource if your API surfaces nested backfill progress.

​
File format (JSONL)

Each line is one JSON object with a type of "patient" or "log" and a data object. Log rows carry event_type (a value from the log types catalog), patient_id, timestamp, the type-specific payload, and optional idempotency_key and trace. Patient rows carry the patient-creation fields, including external_identifiers used to resolve patient_id references in later log rows. Declare patients before the logs that reference them; sort logs chronologically per patient if you want deterministic replay order (the pipeline also sorts by timestamp before replay). 
{"type": "patient", "data": {"first_name": "Ada", "last_name": "Example", "date_of_birth": "1975-06-15T00:00:00Z", "timezone": "America/New_York", "external_identifiers": [{"system": "emr", "value": "MRN-104"}]}}
{"type": "log", "data": {"event_type": "symptom_report", "patient_id": "MRN-104", "timestamp": "2025-11-02T09:00:00Z", "payload": {"symptom": "fatigue", "score": 6}}}
Validate the file locally with olira validate before uploading.

​
Auditing and retention

Uploaded files are stored for audit (lifecycle policies such as transition to cold storage and long retention may apply). Plan uploads accordingly.