Case study · Personal open-source project · MIT licence

Precertly

An open-source AI copilot that checks prior-authorization criteria from official Medicare coverage policies against a patient chart in Fast Healthcare Interoperability Resources (FHIR) R4 format, cites a verified quote for every verdict, and hands the result to a human reviewer. It never denies a request; a human decides.

Public site: A public read-only site of recorded runs, always available; live environment on request.

Personal open-source project. All data is synthetic.

Scope
4 procedures, 76 criteria, 93 verified policy quotes
Backend
Python 3.12, FastAPI, PostgreSQL with pgvector
Model
Amazon Nova 2 Lite on Amazon Bedrock
Cloud
Amazon Web Services (AWS), six Cloud Development Kit (CDK) stacks
Facts as of
2026-10-28

Problem

In a 2024 American Medical Association (AMA) survey, physicians reported about 39 prior-authorization requests per week, taking about 13 hours a week, and more than 25% reported a serious adverse event for a patient.

The Centers for Medicare & Medicaid Services (CMS) Interoperability and Prior Authorization final rule, CMS-0057-F, requires payers to provide prior-authorization application programming interfaces (APIs) by January 1, 2027, using the Health Level Seven (HL7) Da Vinci Prior Authorization Support (PAS) implementation guide.

What it does

  • Encodes coverage policy. Four procedures from the governing CMS coverage text: facet joint interventions, continuous positive airway pressure (CPAP) for sleep apnea, spinal cord stimulators and bariatric surgery. 76 criteria with all-of / any-of variant logic, and 93 verbatim policy quotes verified against the CMS Coverage API.
  • Reads the chart. Ingests a synthetic FHIR R4 patient chart, generated with Synthea, into chunks that keep resource ids and offsets, stored in PostgreSQL with pgvector. Retrieval is deterministic hybrid search: full-text and vector search merged by reciprocal rank fusion, with Amazon Titan Text Embeddings V2 (1024 dimensions).
  • Judges each criterion. Amazon Nova 2 Lite on Amazon Bedrock, with forced structured output. Outcomes are met, not met, insufficient and not applicable. Code verifies every quote and combines the all-of / any-of logic.
  • Lets rules settle what coded data can settle. A deterministic rules layer decides 12 criteria. It marks “met” only when fully determined, never marks “not met” from missing data, and the model cannot override it. Rule-decided verdicts show their computed facts (statement, rule, rules version and the resource ids read) instead of a quote.
  • Drafts standard requests. Draft HL7 Da Vinci PAS 2.2.1 request bundles for the facet cases, validated with the official HL7 FHIR Validator in continuous integration (CI) with 0 errors. They are never submitted and are not ready to send.

How it works

Judgment pipeline
Synthetic FHIR chartFHIR R4, generated with Synthea
Chunking and hybrid retrievalPostgreSQL + pgvector; full-text and vector search, reciprocal rank fusion
Rules layercoded data; the model cannot override it
Model judgmentnotes; Amazon Nova 2 Lite, forced structured output
Quote verificationcode checks every quote and combines all-of / any-of logic
Human reviewermakes the decision
Coded data goes to deterministic rules; free-text notes go to the model. Every verdict is checked by code before a person sees it.

A modular monolith, not microservices. One Python 3.12 / FastAPI service and one container image, split into packages: ingest, retrieval, judgment, PAS export and live runs. The same image runs in three roles: the API (zero tasks unless started), the live API, and a one-off tools task for migrations and corpus loading. Not built: a queue, multi-tenancy, a load test.

Recorded mode. A deterministic export (byte-identical between runs, with no database or model call) feeds a static React 19 / TypeScript (strict) / Vite / Tailwind app: cases, criteria, verdicts and cited quotes highlighted in the patient chart, a badge on each chart chunk saying whether the model was shown it, a four-step guided tour, and landing tiles that show the quality gates honestly.

Cloud. Six AWS CDK (Python) stacks in us-east-2 (the CloudFront certificate is in us-east-1): base (budget alarms, container registry), app (virtual private cloud (VPC) with no network address translation (NAT) gateway, Amazon Elastic Container Service (ECS) cluster, Fargate service at zero tasks), data (Aurora Serverless v2 PostgreSQL 16 with pgvector, pausing after 300 seconds), web (private Amazon Simple Storage Service (S3) bucket, CloudFront with origin access control, custom domain, AWS Certificate Manager (ACM) certificate), auth (Cognito user pool, admin-created users only, time-based one-time password (TOTP) multi-factor required, no SMS text-message codes) and live (API Gateway HTTP API with a JSON Web Token (JWT) authorizer, throttling and cross-origin resource sharing (CORS) limited to the site origin, VPC link, Cloud Map, Fargate). Annual spend hard limit: $60.

Live mode. Sign-in uses the Cognito hosted sign-in page with authorization code and Proof Key for Code Exchange (PKCE, S256), no software development kit (SDK), and the token kept in memory only. One judgment of a curated synthetic case was run from the site through the API gateway on 2026-10-24 (16 criteria, about 77 seconds). Every live result carries the notice “Single unregistered run, not part of the eval, no accuracy claim. Results are not stored.”

Public, read-only path
Browser
CloudFrontcustom domain, ACM certificate, origin access control
Private S3 bucketstatic React app and recorded-run export
Live path invite-only, started on request
Cognito sign-inhosted sign-in, PKCE (S256), TOTP required
API GatewayHTTP API, JWT authorizer, throttling
VPC link → Cloud Mapprivate service discovery
Fargatezero tasks unless started
Aurora Serverless v2PostgreSQL 16 + pgvector, scales to zero
Amazon BedrockNova 2 Lite, Titan Text Embeddings V2
The static site is separate from the live path. With the environment stopped, the live page says so and links to the recorded runs.

Results

Experiment 1.1, scored on the 11 facet cases. The eval set has 29 hand-labeled cases (326 labeled criteria) across the 4 procedures, built by script from a synthetic patient so every expected answer is known by construction, with a labeling guideline based on the CMS Comprehensive Error Rate Testing (CERT) programme. The other 18 cases are labeled but not yet scored.

Precertly evaluation results, experiment 1.1
MeasureResultTargetStatus
Criterion accuracy96.2%
177 of 184 criteria
None setReported
Cases correct overall7 of 11None setReported
Citation precision74.4%
229 of 308
Above 90%Not met
False approvals1 of 9
negative labels; a retrieval miss on case facet-11
Below 2%Not met
Run-to-run noise3 of 184
verdicts differ between the two recorded runs; the largest difference measured anywhere is 14 of 184
None setReported
Rules layerFixed 4 stable errors and removed an unsafe pain-scale conversion; partial-path invalidations went from 7 of 20 to 0 of 20None setImproved

One annotator; scored on the 11 facet cases only; not clinically validated.

Experiment 2 (line-level evidence handles) failed its pre-registered criteria and was reverted; the failure is documented.

Engineering decisions

  1. Safety

    Taking arithmetic away from the model

    A pain-scale conversion done by the model produced a false approval. Deterministic code now decides what coded data can settle, and the model cannot override it.

  2. Evaluation

    Reverting a failed experiment

    Line-level evidence handles missed their written, pre-registered criteria, so the change was reverted and the negative result documented rather than shipped.

  3. Regression

    A fix that broke something

    A partial fix left computed citations unrecognized by the verifier. It was fixed with a computed-fact handle and per-branch requirements, then measured: invalidations went from 7 of 20 to 0 of 20.

  4. Cost

    Architecture chosen by the budget

    An Application Load Balancer plus Amazon Relational Database Service design came to about $43 a month (list-price arithmetic). API Gateway, Cloud Map and Aurora Serverless v2 scaled to zero came to about $1 to $2 idle. An always-on load balancer alone would have cost about $197 a year against a $60 limit.

  5. Latency

    Designing for a slow request

    A live run takes about a minute, longer than the API gateway's 30-second limit. The API returns a run id and the page polls (start-then-poll), with one run at a time (a Postgres advisory lock) and caps of 10 runs a day overall and 3 per user. Results are held in memory only.

  6. Verification

    Checking a cost instead of assuming it

    The default Cost Explorer view netted credits and showed about $0. A rerun excluding credits showed $0.67 over three days, reported with date, Coordinated Universal Time (UTC) and scope. Measured idle cost (Cost Explorer, owner-run 2026-10-27, UTC days, gross before credits, one idle day): about $0.014 a day, roughly $0.9 a month including a Route 53 hosted zone.

Quality and security

About 83pull requests, merged one at a time after green continuous integration
2,062Python tests (44 skipped), plus database tests on a pgvector service container
261infrastructure tests, plus CDK synthesis and a container build check
About 590web tests; layout checked with a Playwright script at 320, 360, 390, 768 and 1280 px (manual, not in CI)
  • ruff, pyright, pip-audit and CodeQL in CI, with a protected main branch.
  • Threat model against the Open Worldwide Application Security Project (OWASP) Top 10 for large language model applications (2025) and the Application Security Verification Standard (ASVS) 5.0, in docs/QUALITY.md.
  • Secret scan of the full git history (gitleaks 8.18.4 plus pattern searches, 2026-10-26): no credential found.
  • Static hosting with a strict content security policy (CSP) with no inline code, HTTP Strict Transport Security (HSTS) without includeSubDomains, and hashed assets cached as immutable.
  • A read-only smoke check of 37 assertions against the published site passed (owner-run, latest on 2026-10-28).
  • Deployed from the command line on my own machine through a script with diff review, because the account's organization policy blocks Identity and Access Management (IAM) identity providers.

Stack

  • Python
  • FastAPI
  • Pydantic
  • SQLAlchemy
  • Alembic
  • PostgreSQL
  • pgvector
  • Docker
  • Amazon Bedrock (Nova 2 Lite, Titan Text Embeddings V2)
  • boto3
  • uv
  • GitHub Actions
  • AWS CDK
  • ECS Fargate
  • ECR
  • Aurora Serverless v2
  • API Gateway
  • Cognito
  • Cloud Map
  • Secrets Manager
  • CloudWatch
  • S3
  • CloudFront
  • ACM
  • FHIR R4
  • HL7 Da Vinci PAS
  • React
  • TypeScript
  • Vite
  • Tailwind CSS
  • Vitest
  • Playwright

Built with AI assistance

Built with AI coding assistance (Claude Code). I set the architecture, invariants and cost limits, reviewed the pull-request diffs, ran every deployment myself and verified deployed state from command output.

Limits

  • Synthetic data only.
  • 18 of 29 cases are labeled but not yet scored.
  • Live mode is invite-only and started on request.