Problem
In a 2024 American Medical Association (AMA) survey, physicians reported about 39 prior-authorization requests per week, taking about 13 hours a week, and more than 25% reported a serious adverse event for a patient.
The Centers for Medicare & Medicaid Services (CMS) Interoperability and Prior Authorization final rule, CMS-0057-F, requires payers to provide prior-authorization application programming interfaces (APIs) by January 1, 2027, using the Health Level Seven (HL7) Da Vinci Prior Authorization Support (PAS) implementation guide.
What it does
- Encodes coverage policy. Four procedures from the governing CMS coverage text: facet joint interventions, continuous positive airway pressure (CPAP) for sleep apnea, spinal cord stimulators and bariatric surgery. 76 criteria with all-of / any-of variant logic, and 93 verbatim policy quotes verified against the CMS Coverage API.
- Reads the chart. Ingests a synthetic FHIR R4 patient chart, generated with Synthea, into chunks that keep resource ids and offsets, stored in PostgreSQL with pgvector. Retrieval is deterministic hybrid search: full-text and vector search merged by reciprocal rank fusion, with Amazon Titan Text Embeddings V2 (1024 dimensions).
- Judges each criterion. Amazon Nova 2 Lite on Amazon Bedrock, with forced structured output. Outcomes are met, not met, insufficient and not applicable. Code verifies every quote and combines the all-of / any-of logic.
- Lets rules settle what coded data can settle. A deterministic rules layer decides 12 criteria. It marks “met” only when fully determined, never marks “not met” from missing data, and the model cannot override it. Rule-decided verdicts show their computed facts (statement, rule, rules version and the resource ids read) instead of a quote.
- Drafts standard requests. Draft HL7 Da Vinci PAS 2.2.1 request bundles for the facet cases, validated with the official HL7 FHIR Validator in continuous integration (CI) with 0 errors. They are never submitted and are not ready to send.
How it works
A modular monolith, not microservices. One Python 3.12 / FastAPI service and one container image, split into packages: ingest, retrieval, judgment, PAS export and live runs. The same image runs in three roles: the API (zero tasks unless started), the live API, and a one-off tools task for migrations and corpus loading. Not built: a queue, multi-tenancy, a load test.
Recorded mode. A deterministic export (byte-identical between runs, with no database or model call) feeds a static React 19 / TypeScript (strict) / Vite / Tailwind app: cases, criteria, verdicts and cited quotes highlighted in the patient chart, a badge on each chart chunk saying whether the model was shown it, a four-step guided tour, and landing tiles that show the quality gates honestly.
Cloud. Six AWS CDK (Python) stacks in us-east-2 (the CloudFront certificate is in us-east-1): base (budget alarms, container registry), app (virtual private cloud (VPC) with no network address translation (NAT) gateway, Amazon Elastic Container Service (ECS) cluster, Fargate service at zero tasks), data (Aurora Serverless v2 PostgreSQL 16 with pgvector, pausing after 300 seconds), web (private Amazon Simple Storage Service (S3) bucket, CloudFront with origin access control, custom domain, AWS Certificate Manager (ACM) certificate), auth (Cognito user pool, admin-created users only, time-based one-time password (TOTP) multi-factor required, no SMS text-message codes) and live (API Gateway HTTP API with a JSON Web Token (JWT) authorizer, throttling and cross-origin resource sharing (CORS) limited to the site origin, VPC link, Cloud Map, Fargate). Annual spend hard limit: $60.
Live mode. Sign-in uses the Cognito hosted sign-in page with authorization code and Proof Key for Code Exchange (PKCE, S256), no software development kit (SDK), and the token kept in memory only. One judgment of a curated synthetic case was run from the site through the API gateway on 2026-10-24 (16 criteria, about 77 seconds). Every live result carries the notice “Single unregistered run, not part of the eval, no accuracy claim. Results are not stored.”
Results
Experiment 1.1, scored on the 11 facet cases. The eval set has 29 hand-labeled cases (326 labeled criteria) across the 4 procedures, built by script from a synthetic patient so every expected answer is known by construction, with a labeling guideline based on the CMS Comprehensive Error Rate Testing (CERT) programme. The other 18 cases are labeled but not yet scored.
| Measure | Result | Target | Status |
|---|---|---|---|
| Criterion accuracy | 96.2% 177 of 184 criteria | None set | Reported |
| Cases correct overall | 7 of 11 | None set | Reported |
| Citation precision | 74.4% 229 of 308 | Above 90% | Not met |
| False approvals | 1 of 9 negative labels; a retrieval miss on case facet-11 | Below 2% | Not met |
| Run-to-run noise | 3 of 184 verdicts differ between the two recorded runs; the largest difference measured anywhere is 14 of 184 | None set | Reported |
| Rules layer | Fixed 4 stable errors and removed an unsafe pain-scale conversion; partial-path invalidations went from 7 of 20 to 0 of 20 | None set | Improved |
One annotator; scored on the 11 facet cases only; not clinically validated.
Experiment 2 (line-level evidence handles) failed its pre-registered criteria and was reverted; the failure is documented.
Engineering decisions
- Safety
Taking arithmetic away from the model
A pain-scale conversion done by the model produced a false approval. Deterministic code now decides what coded data can settle, and the model cannot override it.
- Evaluation
Reverting a failed experiment
Line-level evidence handles missed their written, pre-registered criteria, so the change was reverted and the negative result documented rather than shipped.
- Regression
A fix that broke something
A partial fix left computed citations unrecognized by the verifier. It was fixed with a computed-fact handle and per-branch requirements, then measured: invalidations went from 7 of 20 to 0 of 20.
- Cost
Architecture chosen by the budget
An Application Load Balancer plus Amazon Relational Database Service design came to about $43 a month (list-price arithmetic). API Gateway, Cloud Map and Aurora Serverless v2 scaled to zero came to about $1 to $2 idle. An always-on load balancer alone would have cost about $197 a year against a $60 limit.
- Latency
Designing for a slow request
A live run takes about a minute, longer than the API gateway's 30-second limit. The API returns a run id and the page polls (start-then-poll), with one run at a time (a Postgres advisory lock) and caps of 10 runs a day overall and 3 per user. Results are held in memory only.
- Verification
Checking a cost instead of assuming it
The default Cost Explorer view netted credits and showed about $0. A rerun excluding credits showed $0.67 over three days, reported with date, Coordinated Universal Time (UTC) and scope. Measured idle cost (Cost Explorer, owner-run 2026-10-27, UTC days, gross before credits, one idle day): about $0.014 a day, roughly $0.9 a month including a Route 53 hosted zone.
Quality and security
- ruff, pyright, pip-audit and CodeQL in CI, with a protected main branch.
- Threat model against the Open Worldwide Application Security Project (OWASP) Top 10 for large language model applications (2025) and the Application Security Verification Standard (ASVS) 5.0, in docs/QUALITY.md.
- Secret scan of the full git history (gitleaks 8.18.4 plus pattern searches, 2026-10-26): no credential found.
- Static hosting with a strict content security policy (CSP) with no inline code, HTTP Strict Transport Security (HSTS) without includeSubDomains, and hashed assets cached as immutable.
- A read-only smoke check of 37 assertions against the published site passed (owner-run, latest on 2026-10-28).
- Deployed from the command line on my own machine through a script with diff review, because the account's organization policy blocks Identity and Access Management (IAM) identity providers.
Stack
- Python
- FastAPI
- Pydantic
- SQLAlchemy
- Alembic
- PostgreSQL
- pgvector
- Docker
- Amazon Bedrock (Nova 2 Lite, Titan Text Embeddings V2)
- boto3
- uv
- GitHub Actions
- AWS CDK
- ECS Fargate
- ECR
- Aurora Serverless v2
- API Gateway
- Cognito
- Cloud Map
- Secrets Manager
- CloudWatch
- S3
- CloudFront
- ACM
- FHIR R4
- HL7 Da Vinci PAS
- React
- TypeScript
- Vite
- Tailwind CSS
- Vitest
- Playwright
Built with AI assistance
Built with AI coding assistance (Claude Code). I set the architecture, invariants and cost limits, reviewed the pull-request diffs, ran every deployment myself and verified deployed state from command output.
Limits
- Synthetic data only.
- 18 of 29 cases are labeled but not yet scored.
- Live mode is invite-only and started on request.