I’m testing a small Python pipeline (pandas + PI AF exports) to roll up daily kg CH4/boe using Subpart W factors and compressor run-hours, with the goal of a 10% vent reduction and clear audit trails… Any open-source calculators or templates you’ve used that handle pneumatic bleed rates and flare efficiency assumptions cleanly, and can auto-ingest SCADA for near-daily updates?
OPGEE v3 might be heavy; still, it handles pneumatics/flare well. https://github.com/Stanford-EAO/OPGEE — want PI Web API snippet?
Quick win: swap the PI AF exports for PI Web API pulls into TimescaleDB, then have pandas resample to daily kg CH4/boe while reading Subpart W pneumatics/flare assumptions from a YAML so the audit trail is diffable — , spreadsheet overrides get lost. For flares, model efficiency as a time series (say 98% baseline plus downtime tags) and weight with compressor run-hours; that keeps SCADA-driven updates clean. @olivia_carter2005, mind posting that Web API snippet?
What’s worked well for us is keeping Subpart W assumptions in a Git‑versioned YAML (pneumatic bleed scfh and a site-level ‘flare_eff’ rule) and joining them by effective date so the daily rollup always applies the right factors with clear diffs; we also auto-drop flare_eff to 0.90 on days with high wind or no pilot signal and validate fields with pandera (GitHub - unionai-oss/pandera: A light-weight, flexible, and expressive statistical data testing library). @stevens_k98 would that config‑by‑date pattern slot into your flow?
But i’ve had good luck stamping each daily record with an ‘assumption_id’ (SHA256 of the pneumatics/combustor config) and writing outputs to partitioned Parquet so audits show exactly which rates drove each number. To keep the stream clean, add pandera checks on SCADA joins and cap forward-fills on run‑hours to a small window; here’s the lib: https://pandera.readthedocs.io. Want a tiny snippet to compute the hash and attach it to the dataframe?
Pair your config YAML (@g_blackman89’s point) with Great Expectations and log each daily calc’s inputs/outputs as a tiny “manifest” row (git SHA of code, factor set, flare assumption, device list) so you’ve got a breadcrumb trail an auditor can follow. Then land the results in Delta Lake for time‑travel and diffs when assumptions change. For near‑daily ingest from SCADA, Telegraf’s OPC UA plugin → Kafka keeps the pandas step simple — would that be overkill for your volume?