edward plata

About

I'm Edward Plata, a data engineer in New York. I've spent seven years building analytics platforms, and the part I care about most is the unglamorous part: the standards, checks and observability that make a platform something other engineers can safely build on.

What I work on now#

Since February 2025 I've been a data engineer at Vytalize Health. I own the FHIR/BCDA claims ingestion (PySpark, Delta Lake) and a shared-savings payout pipeline, from ingestion through reconciliation to the payment figures that go out. Around them sit a medallion lakehouse on Databricks with Unity Catalog, the dbt governance standard (tests, docs, lineage, CI) for the analytics layer, pipeline observability with OpenTelemetry, and ingestion across Azure and AWS.

Before that#

From 2021 to 2025 I was a data engineering consultant at Accenture, building a self-service analytics platform for 200+ analysts, streaming pipelines with Kafka and Spark Structured Streaming, and dbt standards that 50+ engineers adopted. From 2019 to 2021 I was a data engineer at Verisk (Argus), automating ETL and building data-quality reporting over production warehouses.

I have a BS in Statistics from Baruch College, CUNY, and I went through the Galvanize Data Science Immersive. The statistics shows up more than you might expect: most of what I write about is some version of "is that number telling you what you think it is?"

What I write about here#

The projects on this site are things I built on my own time, and the posts are what they taught me:

  • Warehouses and pipelines: contracts, schema ownership, and what belongs in a silver layer.
  • Agents over data: making LLM tools reliable, safe against untrusted input, and honest about what they did.
  • Self-hosting: running my own services unattended, and the many ways a system can report success while doing nothing.
  • Autonomous ML research: why most experiments should be discarded, and how to tell a loop that is working from one that is only green.

I try to include the trade-offs and the mistakes, because those are the parts I find most useful in other people's writing.

Get in touch#