Skip to content
hp.

run log, 2026Hugh PeganSan Francisco

Data that runs itself.

Senior Data Engineer at IRU, running the data platform end to end: Kafka, Snowflake, dbt, Airflow, Terraform. Building agent-first since 2026.

Every AI system is only as good as the data underneath it. I build the platforms, standards, and guardrails that make that data trustworthy, cheap, and boringly reliable.

fig. 1 — the plumbingnot to scale

Pipeline sketch: Kafka and S3 sources flow into Snowflake and dbt, orchestrated by Airflow and Terraform, serving 13+ teams.01 · sourcekafka → s302 · transformsnowflake · dbt04 · serve13+ teams03 · airflow · terraform~2.7B msgs / day~3.9T rows · ~470 TB

boring is a feature.

Now

tail -f status.log
  • [ok]Running IRU's data platform
  • [run]Building agent-first
  • [loc]Based in SF
  • [open]Open to interesting conversations

By the numbers

§02stage: transform

Specs to production, at platform scale.

Streaming ingest, warehouse modeling, access control, and cost. Each project started as a written spec and ended in production, with a rollback plan.

§03stage: orchestrate

Agent-first, human-gated.

I build with a personal multi-agent harness. Cheap models search and read, mid-tier models build, top-tier models review adversarially, and I make the calls. The result is more output without lowering the bar: every change still ships with a spec, a test, and a rollback.
fig. 2 — model routingharness.route()
  1. 1search · readcheap models
  2. 2buildmid-tier models
  3. 3reviewtop-tier, adversarial
  4. 4decidemehuman gate
  • every change ships with:
  • spec
  • test
  • rollback
  • 120+

    AI-assisted PRs shipped

  • ~6×

    merged-code throughput vs the year before

  • 19 skills · 5 model-routed agents

    19 skills and 5 model-routed agents in my harness

  • 347

    subagent runs in seven weeks

§04stage: serve

Writing it down so it doesn't depend on me.

I founded my team's standards repo and wrote its ingest standards and playbooks so that engineers and AI agents follow the same steps. I've interviewed and onboarded three engineers, and I run knowledge-transfer sessions until the systems I built no longer need me.

selected work · pages in progress

§05stage: lineage

Experience.

  1. 2024 — Now

    IRU(formerly Kandji)

    Senior Data Engineer II, Data Platform

  2. 2019 — 2023

    HouseCanary

    Software Engineer, Data Platform

  3. 2018 — 2019

    Platform Science

    Data Analyst

  4. 2018

    Occidental College

    B.A., Mathematics