Parallel-Run: How to Replace a Monitoring System Without Trusting It Yet

I have two monitoring systems logging the same house right now. The old one is RRDtool, which has been running for years and which I still believe. The new one is the asyncio → SQLite(WAL) → FastAPI → Chart.js stack I wrote about in the migration post. The new one is better in every way I can articulate, and I have still not turned the old one off.

That is not indecision. It’s that “the new system works” is a claim I hadn’t yet built a test for, and until I have the test, the old system is my only evidence that the new one is lying or not.

Every parallel-run guide you’ll find is written for enterprise migrations — BI platforms, transaction monitoring, data warehouses. They all say the same thing: feed both systems the same input, compare the outputs, cut over when the discrepancies are explained. That advice is fine and it is also not directly usable here, because it assumes something that is false for sensors: that you can feed the same input to both systems.

You cannot duplicate a physical read

A transaction record can be fanned out to two consumers for free. A DS18B20 cannot. There is one 1-Wire bus, one pull-up, one shared time-slotted protocol, and every read occupies it exclusively. Two independent pollers on one bus are not two observers of the same thing — they are two contenders for a resource, and they change the thing they’re measuring.

I have numbers for how expensive that bus is, from a 24-hour instrumented run on my own chain (8 DS18B20s, baseline 4k7 pull-up, as-installed cabling):

A full sweep of the chain takes a median of 3.9 seconds and occasionally 17.4. That is the bus busy, end to end. Now imagine a second daemon that also wants a sweep, on its own schedule, unaware of the first. The two interleave, transactions get stepped on, and the CRC failure rate — already 6.67 per 1,000 in the quiet case — goes up. Both systems get worse data, and crucially they get differently worse data. You would then spend weeks investigating a discrepancy that you created by running the comparison.

There’s a related detail in the per-sensor timings that I want to flag rather than smooth over. Median single-sensor read time on my chain splits into two clusters: four sensors around 0.128–0.168 s, four around 0.800–0.830 s. The DS18B20 datasheet (Maxim/ADI DS18B20) gives a maximum temperature conversion time of 750 ms at 12-bit resolution, halving with each bit of resolution dropped. The ~0.8 s cluster lines up with a full 12-bit conversion being triggered and waited on per read. The ~0.13 s cluster does not, and the honest answer is that I have not yet confirmed why — the plausible causes are a different configured resolution on those parts, or the kernel w1_therm driver returning a value from a conversion that was already in flight. I haven’t isolated it, so I’m not going to claim which.

The fix: one reader, two writers

The correct topology for a sensor parallel run is not two pollers. It is one poller and a tee. The collector that owns the bus reads once, then hands the identical sample to both sinks.

This preserves the property the enterprise guides actually care about — both systems see the same input — while respecting the property they never had to think about, which is that the input is a scarce physical resource.

async def poll_and_tee(bus, sinks):
    """One read of the chain. Same sample object to every sink."""
    sample = await bus.read_chain()          # the only physical access
    results = await asyncio.gather(
        *(sink.write(sample) for sink in sinks),
        return_exceptions=True,
    )
    for sink, r in zip(sinks, results):
        if isinstance(r, Exception):
            # A failing sink must never take down the poller or its peer.
            log.warning("sink %s failed: %r", sink.name, r)
    return sample

Two rules in that snippet are load-bearing. return_exceptions=True means a sink that throws — RRDtool’s binary refusing an out-of-order timestamp, say — cannot cancel the sibling write or kill the poll loop. And the sinks receive the same sample, not two re-reads, so any disagreement downstream is genuinely a storage or processing difference and not a sampling difference. That distinction is the entire point of the exercise.

The legacy sink is just a subprocess call:

class RRDSink:
    name = "rrd"
    async def write(self, sample):
        for metric, value in sample.metrics.items():
            proc = await asyncio.create_subprocess_exec(
                "rrdtool", "update", self.path_for(metric),
                f"{int(sample.ts)}:{value}",
                stderr=asyncio.subprocess.PIPE,
            )
            _, err = await proc.communicate()
            if proc.returncode:
                raise RuntimeError(err.decode().strip())

Why “the two graphs should match” is the wrong exit criterion

Here is the trap I nearly walked into: comparing values from the two stores and expecting equality.

They will not be equal, and not because either is wrong. RRDtool does not store your samples. It stores consolidated samples. Data arrives, gets normalised onto fixed step boundaries, and is folded into round-robin archives by a consolidation function — AVERAGE, MIN, MAX — over a defined number of primary data points. What comes back out of rrdtool fetch is a resampled, averaged view aligned to step boundaries you did not choose per-query. SQLite, meanwhile, has my raw rows at the exact timestamps they were taken.

So a naive diff shows disagreement everywhere, and every one of those disagreements is an artefact of the comparison, not a defect. If you set “graphs match” as your exit criterion you’ll either never cut over, or you’ll get tired and cut over for bad reasons.

What I compare instead:

Only the third of those touches values, and it only does so after I’ve applied the same consolidation to the SQLite side that RRDtool applied to its own. The comparison query is a GROUP BY on the step boundary with AVG(), then a join against the fetched RRD series on the aligned timestamp. If those disagree by more than float noise, something real is wrong. If I skip the consolidation step, the disagreement means nothing.

The gap census is the one that actually earns its keep, and it’s cheap: bucket both series by the RRD step, and list the buckets where one side has rows and the other doesn’t. Missing data is the failure mode that a new collector actually has — a wedged event loop, a service that didn’t come back after a reboot, a token that expired at 3 a.m. Wrong-by-0.2-degrees is not.

What went wrong

The comparison itself became the biggest risk. My first instinct was two independent pollers, because that’s what every parallel-run write-up describes. On a shared 1-Wire bus that is actively harmful, for the reasons above. I switched to the tee before I’d generated much bad data, but the design mistake was real and it was caused by importing advice from a domain where inputs are free to duplicate.

A failing legacy sink took down the new one. Early on the RRD subprocess write and the SQLite write lived in the same unguarded coroutine. rrdtool update rejects a timestamp that is not strictly newer than the last one in the file, which happens after a clock adjustment. That exception propagated and killed the poll iteration — so the legacy system’s pickiness caused a gap in the new system’s data, which is exactly backwards. Hence return_exceptions=True and the per-sink try boundary. The system under test must not be able to be failed by the system it’s replacing.

Both systems on one Pi share a failure domain. Running old and new on the same Pi 4 means an SD card problem, a kernel oops, or a power cut takes out your control and your treatment together. That’s a real limitation of my setup and I’m not going to pretend otherwise. The parallel run validates the software; it does not validate against host failure, and a “we ran for months with no data loss” claim from a single-host parallel run is weaker than it sounds.

Disk is not free. WAL-mode SQLite plus a full set of RRDs plus doubled write traffic on the same card, indefinitely. RRD files are fixed-size, which helps, but the SQLite side grows without bound by design — that was the point of migrating. An open-ended parallel run needs a decision about how long, made up front.

The actual criterion

Mine is boring and I’d recommend it: the new system has to survive a period containing at least one unplanned reboot, one network outage, and one Nest token refresh, with no gaps that the legacy system doesn’t also have. Not “matches the old graphs.” Not a fixed number of weeks on a calendar. A list of specific failures it must be observed absorbing.

The enterprise guides say one to three months and I understand why they need a number — they’re writing for people who have to justify a schedule. I don’t. I have a list, and the old system stays up until the list is done. The cost of leaving RRDtool running is a few megabytes and some CPU. The cost of cutting over early and discovering in February that I lost January is history I cannot regenerate, because the house only produces that data once.

Related reading

• Migrating RRDtool to SQLite on Raspberry Pi: Keep Every Historical Data Point

  How I migrated a two-year RRDtool logging setup to SQLite in WAL mode on a Raspberry Pi 4 — rescued 145,742 legacy rows, kept every historical data point, and used a parallel-run cutover to verify.

• An asyncio Collector Pattern for Mixed-Protocol Sensor Fleets

  Why asyncio does not make 1-Wire faster, and the collector pattern I actually run: per-plugin loops, to_thread for blocking buses, and one batched SQLite writer.

• DHT11 After 18 Months of Continuous Logging: An Honest Accuracy Review

  18 months and 26,318 rollup rows from one DHT11 on a Pi: 38% of them below the datasheet's 20 %RH floor, what that actually looks like, and where the part stops being usable.