Hildebrand Glow, Home Assistant and MQTT - Getting visibility into my energy data
Since Home Assistant announced energy monitoring was being added, I’ve been super interested in what sort of data I could get out of my energy system. We have nothing fancy here. Just fairly standard dual-fuel (electricity and gas) system provided by the grid with a smart meter. No solar panels or batteries or returning power to the grid. My provider (E.ON Next) don’t make that data easily accessible (or at least, they didn’t when I looked).
So, I searched around and found Hildebrand and their Glow meter. Which was out of stock. Boo.
I signed up for stock alerts for the product and started wondering if I could write a bot to log in to my account on the E.ON website, then scrape the data and push it back into Home Assistant in the meantime.
But then I got an email…
DCC? Bright? What?
A few days after signing up for stock alerts, I got an email from Jane at Hildebrand & Glowmarkt Support.
They thanked me for my interest in the product, and let me know about their Bright App (Android - iOS) which would give me half-hourly data.
Well, that saves me some work then! 😅
Hildebrand are a “DCC Other User” meaning that you can grant them access to your energy data:
Our obligations as a DCC Other User mean that SMETS2 customers will have to verify their identity before we can retrieve their data; this is all managed through our Bright App. We are able to join your Display and CAD to your SMETS2 meters without your supplier’s involvement because we are a SECAS (Smart Energy Code Administrator) approved DCC Other User. Please be assured that your verification data is stored in a separate, secure system and Hildebrand are ISO 27001 Certified.
This seemed promising. I downloaded the app, registered an account, added my meter details and just like that the app had my energy data available to it!
Okay, that’s cool. I wasn’t aware such a thing existed. It seems that SMETS2 allows easier access of data to other parties (which makes sense as when you change suppliers with SMETS2 equipment you don’t have to have a new meter installed)
They have an API?!
I thanked Jane for the information, and they replied that while they can only fetch data every 30 minutes without their equipment. Once I buy one of their meters the electricity stats will be updated every 10 seconds and gas continues every 30 minutes. The data would also be available via an API automatically, and MQTT if I request it.
Oh. My. God. Yes.
Until then, I could still use the API, it would just only update every 30 minutes. Well, that’s fine by me for now!
API access is automatic; you use your Bright credentials. Further information here : Repository URL - https://bitbucket.org/ijosh/brightglowmarkt/src/master/
The Bright Application ID is
b0f1b774-a586-4f72-9edd-27ead8aa7a8dYou may find our forums helpful, they are found here: https://forum.glowmarkt.com - several people on there have integrated with HA and I believe some have shared their code.
What’s that? A Postman collection ready to go to fill in my details and poke around with how the API works?! Oh, these people really get it. This is wonderful. So, now to get that data into Home Assistant.
I poked around with the API in Postman, and wrote some PHP scripts to get my head around the data their API was providing. It seemed straight forward enough, but as I was told in the email, people had already integrated with Home Assistant so maybe there was something out there already?
Integration into Home Assistant
I found an addon for Hildebrand Glow. I added it into Home Assistant and got… nothing.
Booo. It turns out that addon looks for MQTT data, which I didn’t have yet (and wouldn’t be setup until I had their in-home device)
Thankfully, HandyHat had created another addon that worked with the DCC data without an in-home device.
I installed and configured this new addon. It gave me data! I added the electricty and gas readings into my Home Assistant energy configuration and saw data in lovely graphs. Gas wasn’t in m³, it was in kWh (the same as electricity) so electricity and gas both appeared in the same chart (gas as a darker blue bar), but that wasn’t a deal-breaker for me. Just having visibility into hourly usage was lovely.
Guess when we started cooking dinner…
A week or so later, my readings weren’t available anymore.
It seems that the token you request for access to API endpoints had expired and the addon hadn’t refreshed it. I removed the addon and reinstalled it and it worked again. Annoying, but not the end of the world.
Other people had the same problem and (at the time of writing) there’s an outstanding pull request to fix this issue.
There’s also outstanding PRs for converting gas to m³ and getting tarrif data so hopefully those will be merged for people soon.
Back in stock
I checked back on the Glowmarkt website one day and noticed that the product was back in stock! (Though, I didn’t get an email alert, lucky I checked eh?). I ordered one immediately and it turned up just a couple of days later.
Setup was super simple. I plugged it into power, configured my WiFi details and it just did the rest itself.
The bright app now showed me the power that our house was consuming right now (well, updated every 10 seconds but close enough!)
So, about that MQTT data then…
EDIT: Thanks to Robin Hawkes on Twitter I learned that the Glow IHD now supports communicating with a local MQTT broker! The data you get from the local MQTT messages is also easier to understand!
Having checked through the original addon I found, it didn’t provide much. So I decided to look into pulling the data out of MQTT myself.
I emailed Jane back and asked them to setup my account for MQTT access. They replied saying it had been setup and gave some pointers on where to look for the data format:
The data is delivered using SEP (zigbee Smart Energy Profile specification) cluster identifiers encoded in JSON (https://www.json.org/json-en.html)
Detail on the SEP is available in this document, we suggest reviewing the Simple Metering Cluster section: https://zigbeealliance.org/wp-content/uploads/2019/12/docs-07-5356-19-0zse-zigbee-smart-energy-profile-specification.pdf
Holy intimidating PDF Batman!
I started looking through the documentation, as well as the sample data I was sent by Jane:
{
"elecMtr": {
"0702": {
"03": {
"01": "000001",
"05": "43",
"04": "9B",
"02": "0003E8",
"07": "XXXXXXXXXXXX",
"08": "17P3111676",
"03": "43",
"00": "00",
"06": "00"
},
"00": {
"07": "25FB6E37",
"01": "000000000000",
"00": "0000002347A4",
"14": "02",
"02": "000000000AC4"
},
"04": {
"01": "0008CA",
"00": "FFFBC1"
},
"02": {
"00": "00"
}
}
},
"gasMtr": {
"0702": {
"03": {
"01": "000001",
"05": "43",
"04": "9B",
"02": "0003E8",
"07": "YYYYYYYYYY",
"08": "G4P63020111800",
"03": "43",
"00": "00",
"06": "80"
},
"00": {
"07": "25FAE6F4",
"01": "000000000000",
"00": "000000E02D06",
"14": "02",
"02": "000000000000"
},
"04": {
"01": "003E76",
"00": "000000"
},
"02": {
"00": "00"
}
}
},
"ts": "2020-03-11 09:39:08",
"hversion": "EHZBWIFI0v4",
"time": "5E68B1BC",
"gmtime": 1583919548,
"pan": {
"status": "joined",
"nPAN": "00",
"join": "0",
"lqi": "FF"
},
"gts": "2020-03-11 00:01:56",
"ets": "2020-03-11 09:39:03",
"error": {
"lastCommand": "00",
"errorCode": "00"
},
"gid": "ZZZZZZZZZZZZZZZZ"
}
This was going to take some time to grok. Let me just Google some things to see if anyone’s condensed this down into the need-to-know bits for me.
Well would you look at that. Frédéric Sagnes posted 2 gists that detail the sensor config in Home Assistant and the type of data contained in each key within the payload. Amazing. Thank you Frédéric!
# Fields gathered from the ZigBee Smart Energy Standard document
# 0702: Metering
# - 00: Reading Information Set
# - 00: CurrentSummationDelivered: meter reading
# - 01: CurrentSummationReceived
# - 02: CurrentMaxDemandDelivered
# - 07: ReadingSnapshotTime (UTC time)
# - 14: Supply Status (enum): 0x2 is on
# - 02: Meter Status
# - 00: Status (bit map): 10 means power quality event
# - 03: Formatting
# - 00: UnitofMeasure (enum): 00 means kWh, 01 means m3
# - 01: Multiplier
# - 02: Divisor
# - 03: SummationFormatting (bit map):
# 2B means 3 digits after the decimal point, 2 digits before the decimal point
# FB means 3 digits after the decimal point, 16 digits before the decimal point,
# no leading zeros
# - 04: DemandFormatting
# - 06: MeteringDeviceType: 00 means Electric Metering, 80 means Mirrored Gas Metering
# - 07: SiteID: MPAN encoded in UTF-8
# - 08: MeterSerialNumber (string)
# - 12: AlternativeUnitofMeasure (enum)
# - 04: Historical Consumption
# - 00: InstantaneousDemand (signed): current consumption
# - 01: CurrentDayConsumptionDelivered
# - 30: CurrentWeekConsumptionDelivered
# - 40: CurrentMonthConsumptionDelivered
# - 0C: Alternative Historical Consumption
# - 01: CurrentDayConsumptionDelivered
# - 30: CurrentWeekConsumptionDelivered
# - 40: CurrentMonthConsumptionDelivered
# 0705: Prepayment
# - 00: Prepayment Information Set
# - 00: PaymentControlConfiguration (bit map)
# - 01: CreditRemaining (signed)
# 0708: Device Management
# - 01: Supplier Control Attribute Set
# - 01: ProviderName (string)
I already have an MQTT broker running in the house, so I set up a bridge to the Glowmarkt MQTT broker:
connection bridge-glowmarkt
address glowmarkt.energyhive.com:8883
remote_username {EMAIL_HERE}
remote_password {PASSWORD_HERE}
bridge_capath /etc/ssl/certs/
topic SMART/HILD/{DEVICE_ID} in 0
I set the bridge up so that it only pulls in things on the topic (topic SMART/HILD/{DEVICE_ID} in 0). I don’t need to write to that topic, and I’m not sure if they’d allow me anyway!
I connected to my local broker and success! I could see the data in there from the bridged connection to the Glowmarkt broker!
With that, I was able to use Frédéric’s sensor config to get them into Home Assistant:
- platform: mqtt
name: "Home Electricity Meter"
state_topic: "SMART/HILD/DEVICE_ID"
last_reset_topic: "SMART/HILD/DEVICE_ID"
last_reset_value_template: homeassistant.util.dt.utc_from_timestamp(0)
unit_of_measurement: 'kWh'
device_class: energy
state_class: total_increasing
value_template: "{{ value_json['elecMtr']['0702']['00']['00']|int(base=16) * value_json['elecMtr']['0702']['03']['01']|int(base=16) / value_json['elecMtr']['0702']['03']['02']|int(base=16) }}"
icon: 'mdi:flash'
- platform: mqtt
name: "Home Gas Meter"
state_topic: "SMART/HILD/DEVICE_ID"
last_reset_topic: "SMART/HILD/DEVICE_ID"
last_reset_value_template: homeassistant.util.dt.utc_from_timestamp(0)
unit_of_measurement: m³
device_class: gas
state_class: total_increasing
value_template: "{{ value_json['gasMtr']['0702']['00']['00']|int(base=16) * value_json['gasMtr']['0702']['03']['01']|int(base=16) / value_json['gasMtr']['0702']['03']['02']|int(base=16) }}"
icon: 'mdi:fire'
- platform: mqtt
name: "Home Current Electricity Usage"
state_topic: "SMART/HILD/DEVICE_ID"
unit_of_measurement: 'W'
value_template: "{{ value_json['elecMtr']['0702']['04']['00']|int(base=16) }}"
icon: 'mdi:speedometer'
I re-configured my energy settings in Home Assistant to use these new sensors (the Home Current Electricity Usage isn’t in there, that’s just for my own little graph)
They appear in their own graphs now!
So now what?
I don’t really have a use-case right now other than “this is cool to see”. I’m on a fixed-tarrif energy plan so I get charged the same money regardless of when energy is used.
I’m not using the data to determine when to turn things on and off.
There are some providers who offer low-cost energy at certain times of day, so you could pull tariff data, hook into this and turn on higher-power devices (EVs etc) to charge during those times I’d imagine. But not having a flexible tariff or an EV, that’s of no use to use at the minute!
For now, this is just about visibility. It’s interesting seeing how the graphs progress during the day (working from home and turning on work equipment to living room media usage in the evening to gas usage for heating & cooking etc).