I wanted to know what data engineers argued about this month. I did not want what vendors said they argued about. So I ran a research script over public threads and read what came back. This post is the log. It is field notes, not a survey.
The method#
The tool is last30days, an open-source skill for coding agents. It is a research script that searches Reddit, Hacker News, GitHub and dev.to for the last 30 days. It groups the posts into clusters, ranks them by engagement and relevance, and saves the raw results to a file.
I ran it on 2026-11-03. The window was 2026-10-04 to 2026-11-03. This post uses five runs, all on data engineering topics:
- single-node vs distributed query engines
- distributed query engine cost benchmarks
- Databricks cost, governance and platform migration pain
- onchain data indexing and analytics pipelines
- onchain data platforms (Dune, Allium, Goldsky)
Two gaps shape everything below. First, X/Twitter was missing from the runs. Much data engineering talk happens there, and none of it is in this log. Second, Reddit blocked my fetches when I tried to check the threads for this post. The script read Reddit, but I could not confirm any thread myself. So I quote no Reddit thread here. Every source I cite below is one I opened and read.
Five things people argued about#
1. Single-node vs cluster#
This was the loudest argument of the month. It started with a release. DuckDB 2.0.0 shipped on 2026-10-21 as a long-term support release, "with support until at least October 2027" (DuckDB). It adds a server mode, async I/O for remote reads, and triggers.
The Hacker News thread had 463 points and 63 comments when I checked. One commenter, RobinL, wrote that DuckDB 2.0 is "much more likely to see 100% utilisation up to about 128vCPUs or 192vCPUs", where before "speed would top out at more like 64vCPU" (comment). Another, nicoritschel, said it is "very fast, especially on a decent sized cloud VM and when querying Iceberg", and that they run it in production for a small organization (comment). Those are opinions from people who use it. They are not benchmarks.
The day after the release, Firat Celik published a benchmark: DuckDB 2.0.0, Polars 2.0.0 and PySpark 4.2.0 on one 8 vCPU VM, with a watchdog that killed any run over 4.5 GiB of memory (Celik). At 150M rows, DuckDB and Polars finished the filter and join queries in under 1.5 seconds (median of 3 runs). Spark took 10 to 16 seconds. On the window dedup query, Polars was killed in all three runs, and Spark finished two of three.
The comments are the part I found most useful. They were about the method, not the result. One reader asked: "Did you test with a cold cache between runs? Our fastest-looking benchmark turned out to be measuring the OS page cache more than the engine." Another pointed out that spark.driver.memory "caps the JVM heap, not the process", so Spark may have been set up to fail under an RSS cap.
Then Jacob reran the same benchmark on an Apple M3 Pro laptop and added SQE, a SQL engine for Iceberg built on DataFusion (jverhoeks). His geomean on Parquet was 1.88 s for SQE, 2.89 s for DuckDB and 15.11 s for Spark. His last line sums up the month: "If your job fits on one machine, measure it on one machine before you start a cluster for it."
Two benchmarks on two machines is not a result I can generalize. It is a good reason to run the test again.
2. Cost that moves#
The second theme was that a feature can remove a cost from one place and put it somewhere else. The clearest written case was Celik again, on Databricks Auto CDF, which became generally available on 2026-09-01 (Celik). In his open-source Delta and Spark test, reading one commit's changes took 0.21 s with legacy CDF and 4.73 s with his row-tracking reconstruction. That is about 22 times slower. His summary: "Auto CDF does not remove the cost of change data." The writer stops paying. Each reader pays on each read.
His test rebuilds the read path by hand, so it is not the managed feature itself. But the shape of the argument appeared in several Reddit threads in my runs too, with titles about table maintenance bills and compute management. I could not open those threads to check them, so I do not quote them.
3. Silent wrong data#
The third theme was data that is wrong while every job reports success. Two dev.to posts made the case well.
In the first, an incremental ERP extract got 30.8% of rows wrong (5,762 of 18,707) (trcollinson). A nightly batch changed quantities without moving the LastModified field, so the incremental read never fetched those changes. A nightly full sweep against the source caught it. The author is clear that the ERP is a simulation with nine faults put in on purpose. It is a constructed example, and a good one. The key line: "a wrong copy never throws an error."
In the second, a project over 1 billion rows of Brazilian crime data found that the newest month always looks low (crime_brasil). In São Paulo, daily reports dropped from 2,459 on 2026-08-26 to 994 on 2026-08-31. Crime did not drop. The paperwork was late. "A report filed today may describe a crime from last week, so the newest month keeps growing for weeks after it 'ends'."
Neither problem needs a new tool. Both need a check that compares the copy to something outside the pipeline.
4. Vendor risk#
The fourth theme came from one incident. Allium, an onchain data vendor, detected unauthorized access to some of its systems on 2026-10-25 (Unchained). These are the facts as Unchained reported them:
- Datashare tables in every region received no new data for just under 24 hours, from 2026-10-26 to 2026-10-27.
- The Allium App was down for about 61 hours.
- On 2026-10-28, Allium disclosed the incident and revoked existing API keys.
- On 2026-10-30, Allium said its Snowflake, BigQuery and Databricks datashares "have passed historical verification checks for all chains across all regions."
- An outside security firm is investigating. At the time of the article, Allium had not yet posted its findings.
I do not know more than that, and I will not guess. The engineering point is narrow. If one vendor feeds your warehouse, a pause at that vendor is a pause in your warehouse. And you may learn about it from the data going stale, not from an alert.
5. Iceberg read cost#
The fifth theme was smaller but specific. A dev.to post explained why a CDC-fed Iceberg table gets slower to read each week (kyien). Deletion vectors point at one data file. "Equality deletes carry no such reference", so a reader must check every older file in the partition for each delete key. Compaction resets the growth but does not stop it. The post names a fix in Iceberg 1.12.0 that converts equality deletes into deletion vectors, for format version 3 tables. The one measured number is an anecdote: a dashboard query that went from about 4 seconds to about 30 seconds over three weeks.
This is the same pattern as theme 2. The write is cheap, and the read pays later.
What the evidence can and cannot tell you#
The script labels each cluster. In my runs, the five topics gave 139 clusters. Only 4 of them drew on more than one source type. The other 135 came from one source type each, and the script flagged 3 as "thin-evidence". So each argument above rests on one or two posts, plus the people who replied to them.
That limits what I can say:
- It can tell you what people wrote about. These five themes came up, and the posts are real and dated.
- It cannot tell you how common a problem is. Upvotes and points measure attention in one community on one day. They are not a sample of data teams.
- It cannot tell you who is right. A benchmark with 3 runs on one VM is a data point. A thread full of agreement is a thread full of agreement.
- It has blind spots I know about. X was missing, and I could not check Reddit myself. Private Slack groups and vendor forums are not in it at all.
I trust the dev.to posts most, because they show their method and the comments test it. I trust the counts least.
What I will do with it#
I will rerun the 150M-row benchmark myself. Before I run anything, I will publish the method card: hardware, engine versions, cache state for each run, the Spark memory settings, and the median of 3 runs. The comments on Celik's post are the checklist. If my numbers differ from his, the card should show why.
I will add a source comparison to my own pipelines. The ERP post and the crime data post describe the same failure. A job that reports success is no proof that the copy matches the source. A nightly count and checksum against the source is cheap, so I will add one wherever a job has none.
I will run the same research again in 30 days, with X included if I can get it working. Then I can see which of these five arguments was a month's noise and which one stayed.