We Built an AI Medical Analyst in a Weekend at the Caltech Longevity Hackathon | HackerNoon

Why longevity?

• Longevity care is longitudinal and data-heavy. Patients accumulate lab panels, imaging, and clinical notes over years.
• Clinicians and individuals need fast, explainable triage: What’s abnormal? What changed? What should I read next?
• A generalizable pipeline that works “in a weekend” helps teams experiment faster and validate real-world impact.

What we shipped at the hackathon

• A Next.js/React UI with a dead-simple uploader and a results table.
• Client-side text extraction for PDFs and images to support mixed inputs quickly.
• A structured LLaMA prompt that returns summary, keywords, categories, abnormal flags, suggested filename, and PubMed titles.
• Supabase Storage for raw files; Postgres table documents for structured metadata.
• A Supabase Edge Function to process stored PDFs server-side (useful for background jobs and batch workflows).

Architecture

• Upload → Extract text → LLM analysis → Persist → Render
• Two processing paths:
• Client-led: immediate feedback, great for demos and small files.
• Server-led (Edge Function): scalable, secure, and good for background/batch processing.

Key moving parts:

• Frontend: Next.js + Tailwind
• OCR/Parsing: pdf-parse, tesseract.js
• AI: LLaMA chat completions API with a rigid, parse-friendly prompt
• Backend: Supabase (Storage for blobs, Postgres for metadata)
• Serverless: Supabase Edge Function for server-side PDF processing

Product walk-through

Home page

            <h1 className="text-3xl font-bold leading-tight text-gray-900">
              Medical Document Analysis
            </h1>
          </div>
        </header>
        <main>
          <div className="max-w-7xl mx-auto sm:px-6 lg:px-8">
            <MedicalDocUploader />

• Clean landing with a single CTA: upload a document.

Upload → extract → analyze → persist (client path)

• Extracts text conditionally based on file type.
• Calls LLaMA with a structured prompt to ensure predictable parsing.
• Uploads original file to Supabase Storage and inserts metadata into documents.

  const handleFileUpload = useCallback(async (event: React.ChangeEvent<HTMLInputElement>) => {
    try {
      setIsUploading(true);
      setError(null);

      const file = event.target.files?.[0];
      if (!file) return;

      // Extract text based on file type
      const text = file.type === 'application/pdf' 
        ? await extractTextFromPDF(file)
        : await extractTextFromImage(file);

      // Analyze the text with Llama
      const analysis = await analyzeWithLlama(text, file.name);

      // Upload to Supabase Storage
      const { data: uploadData, error: uploadError } = await supabase.storage
        .from('medical-documents')
        .upload(analysis.renamed_file, file);

      if (uploadError) throw uploadError;

      // Store metadata in Supabase
      const { data: metaData, error: metaError } = await supabase
        .from('documents')
        .insert({
          filename: file.name,
          renamed_file: analysis.renamed_file,
          file_url: uploadData.path,
          summary: analysis.summary,
          keywords: analysis.keywords,
          categories: analysis.categories,
          word_count: countWords(text),
          report_type: detectReportType(text),
          threshold_flags: analysis.threshold_flags,
          pubmed_refs: analysis.pubmed_refs,
          ai_notes: analysis.ai_notes,
          status: 'processed',
          version: 1
        })
        .select()
        .single();

      if (metaError) throw metaError;

      setDocuments(prev => [...prev, metaData]);
    } catch (err) {
      setError(err instanceof Error ? err.message : 'An error occurred');
    } finally {
      setIsUploading(false);
    }
  }, []);

The prompt that makes it reliable

• The LLM is only as useful as its prompt structure. We force a schema, so parsing is straightforward and less brittle than free-form responses.

  const analyzeWithLlama = async (text: string, originalFilename: string) => {
    const prompt = `Analyze this medical document and provide a detailed analysis in the following format:

1. Summary: Provide a clear, plain-English summary
2. Keywords: Extract key medical terms and their values (if any)
3. Categories: Classify into these categories: ${VALID_CATEGORIES.join(", ")}
4. Filename: Suggest a clear, descriptive filename
5. Threshold Flags: Identify any abnormal values and mark as "high", "low", or "normal"
6. PubMed References: Suggest relevant PubMed articles (just article titles)
7. Additional Notes: Any important medical guidance or observations

Document text:
${text}

Please format your response exactly as follows:
Summary: [summary]
Keywords: [key:value pairs]
Categories: [categories]
Filename: [filename]
Flags: [abnormal values]
References: [article titles]
Notes: [additional guidance]`;

Server-side processing (Edge Function)

• Useful for background jobs, webhook-driven processing, or scaling beyond client limits.
• Downloads file from Supabase Storage, extracts text, calls the same LLaMA prompt, inserts documents row.

    // Prepare LLaMA prompt
    const prompt = `Analyze this medical document and provide a detailed analysis in the following format:

1. Summary: Provide a clear, plain-English summary
2. Keywords: Extract key medical terms and their values (if any)
3. Categories: Classify into these categories: ${VALID_CATEGORIES.join(", ")}
4. Filename: Suggest a clear, descriptive filename
5. Threshold Flags: Identify any abnormal values and mark as "high", "low", or "normal"
6. PubMed References: Suggest relevant PubMed articles (just article titles)
7. Additional Notes: Any important medical guidance or observations

Document text:
${text}

Please format your response exactly as follows:
Summary: [summary]
Keywords: [key:value pairs]
Categories: [categories]
Filename: [filename]
Flags: [abnormal values]
References: [article titles]
Notes: [additional guidance]`

    // Insert into Supabase
    const { data: insertData, error: insertError } = await supabase
      .from('documents')
      .insert(documentData)
      .select()
      .single()

Implementation details

Database schema (Supabase Postgres)

Use JSONB where the structure can vary or expand over time.

-- documents table
create table if not exists public.documents (
  id bigint generated always as identity primary key,
  created_at timestamp with time zone default now() not null,
  user_id uuid null,
  filename text not null,
  renamed_file text not null,
  file_url text not null,
  summary text not null,
  keywords jsonb not null default '{}'::jsonb,
  categories text[] not null default '{}',
  word_count integer not null,
  report_type text not null,
  threshold_flags jsonb not null default '{}'::jsonb,
  pubmed_refs jsonb not null default '{}'::jsonb,
  ai_notes text not null default '',
  status text not null check (status in ('uploaded','processed','failed')),
  user_notes text null,
  version integer not null default 1
);

-- Optional: RLS policies for multi-tenant setups
alter table public.documents enable row level security;

-- Example policies (tune for your auth model)
create policy "Allow read to authenticated users"
  on public.documents for select
  to authenticated
  using (true);

create policy "Insert own documents"
  on public.documents for insert
  to authenticated
  with check (auth.uid() = user_id);

create policy "Update own documents"
  on public.documents for update
  to authenticated
  using (auth.uid() = user_id);

Storage bucket

• Create medical-documents bucket.
• Lock down access if you’re storing PHI; consider signed URLs and RLS on the storage.objects table.

Environment configuration

• Never hardcode secrets client-side. Use environment variables and server-side access.
• For local dev, rely on .env.local and do not commit it.

Example variables to configure:

• SUPABASE_URL
• SUPABASEANONKEY (client reads ok if your RLS is correct)
• SUPABASESERVICEROLE_KEY (server-only; never expose to browser)
• LLAMAAPIKEY (server-only)

For the Edge Function, set:

• SUPABASE_URL
• SUPABASE_ANON_KEY (or service role if needed)
• LLAMA_API_KEY

Deploying the Edge Function

• Install the Supabase CLI
• Link your project
• Deploy function

supabase functions deploy process-medical-pdf
supabase functions list
supabase functions serve process-medical-pdf --no-verify-jwt

Wire the function behind an HTTP trigger or call it from your app to process files already stored in the bucket.

UX considerations

• Drag-and-drop uploader with clear accept types.
• Progress and error state visibility.
• Terse, readable summaries with expandable details.
• Badges for categories and flags for abnormalities.

Reliability strategies

• Structured prompt → predictable parsing.
• Keep LLM temperature moderate (0.3–0.7) to reduce variance.
• Validate parsed JSON fields; default to safe fallbacks.
• Track version and status to support re-processing and migrations.

Security and compliance

• Treat all uploads as potentially sensitive (PHI).
• Don’t expose secrets in the browser. Move LLaMA calls server-side if needed.
• Consider de-identification or redaction at upload.
• Encrypt at rest (Supabase handles storage encryption), and use HTTPS for all calls.
• RLS across documents and signed URLs for downloads.

Performance and cost

• OCR (tesseract.js) can be CPU-heavy; pre-processing images helps (deskew, denoise, contrast).
• Use server-side processing for large PDFs or batch jobs.
• Cache repeated LLM calls when re-processing the same file or version.

What we’d build next

• Normalized lab values with medical ontologies (e.g., LOINC) and unit conversions.
• Trend analysis across time and change detection.
• Confidence scoring and a reviewer checklist for clinical safety.
• Human-in-the-loop editing with audit trails.
• Export to FHIR-compatible bundles.

Demo script (5 minutes)

• Upload a lab report PDF.
• Show immediate “Processing document…” state.
• Reveal results: summary, categories, abnormal flags, and PubMed suggestions.
• Click through the stored file link (signed URL if private).
• Open Supabase Studio to show the corresponding documents row.

Credits

• Built at the Caltech Longevity Hackathon by our team in a sprint focused on turning complex medical paperwork into fast, explainable insights.