MQTT configuration for automatic polling messages (persistent)

[tnn85]

Hi,

I am trying to understand the concept of ebusd in regards to how data/messages can periodically be acquired. I already read the available documentation (Wiki) on Github. However, at some point the information given is not enough for me to fully understand how I could achieve my goal of monitoring certain messages / fields available for my heating system over time.

What I want to do is this:

1. Periodically acquire data from the Vaillant heating control / burner unit, mixer control via eBUS.
2. Make this data available via MQTT (in ebusd?)
3. Use Influx Telegraf + InfluxDB to store this data in a timeseries database
4. Create Dashboards with Grafana to visualize the data in a graph

Lets forget about 3)+4) because that is my problem to implement.

I have difficulties to understand how I am supposed to configure ebusd to read+poll and then send the data I am interested in every -for example- 5s via MQTT and how to make this configuration persistent in case of a power loss / restart (ebusd runs on a dedicated Raspberry Pi).

I have already achieved to read data (ebus messages) from the bus and publish them on MQTT.
However, before most messages are polled regularly I have to execute a "ebusctl read -pX -c CIRCUIT MESSAGENAME" (where X = 1..9) command on the local machine for each message I want to be polled
OR
send a specific "/get" topic suffix as well as "X?" as the value of an MQTT topic to the host ebusd is running on.
I just can't do this with my the backend (telegraf + influxdb) because it does not publish MQTT itself. I can only subscribe to topics.

AFAIK I can also download all necessary CSV message definition files for my system (Vaillant ecovit exclusiv VKK 226/4 + calorMATIC 470 VRC 470 + Mixer VR61) and change the first column to r1..r9 for every message I want to be read (&polled) automatically without further ado.

In this case I would have to modify the original message definition source files (CSV and INC) from and redirect the ebusd argument "--configpath=PATH" to the local path where I have saved the downloaded message definition files from https://github.com/john30/ebusd-configuration/tree/master/ebusd-2.1.x

Is this the way to go?

If this is the way:
I just didn't want to mess around in the original message definition files and would have preferred to use latest updated message definitions from https://cfg.ebusd.eu/ .
I thought maybe there is another way to configure ebusd to poll certain selected eBUS messages.

[ghhausi]

You may want to give the "--mqttinit=/etc/ebusd/mqtt-hassio.cfg" (and "--mqttretain") a chance ... or adapt the ".cfg" to your needs .... all the ebus readout parameters are going into mqtt .

[tnn85]

Thank you for your answer. In fact I have already taken a look at the MQTT integration configurations before. I just didn't know it would suit my needs because MQTT integration configurations are even harder to understand because the missing documentation and the comments in the file are a little confusing to me:

As it seems in the MQTT integration config the filter filter-seen = X is probably what I am looking for but it has 2 purposes:

1. If (filter-seen = 1): Messages that have not appeared on the bus because they are either not sent by other masters (QQ != ebusd) and are not broadcast messages and have no poll priority (r1-r9) set in the message definition files (= all messages when https://cfg.ebusd.eu/ is the selected source for message definition files) will be filtered. So basically no difference to if no MQTT integration is used.
2. If (filter-seen > 1): For all messages listed in the message definition files (which were chosen after the initial scan) the poll priority is set to 2 to 9 (depends on which value above 1 is set here ).

I set filter-seen = 2. Those messages are claimed to be polled with at least the poll priority of 2 (according to the information given in the comments of those MQTT integration config files).

What I have read in the other discussions/documentation poll priority 2 means those messages are only polled every second poll cycle?
The poll cycle interval can be set by the ebusd option --pollinterval=SEC in seconds, right? Default pollinteral is 5 seconds?.

I have set --pollinterval=1 (1 second) already. However, even if i only filter one message (filter-name = Flow1Sensor) with the MQTT integration configuration this message is sent with a frequency of approx. 0.1 msg/s (=10 seconds between subsequent messages) at most. 0.5 msg/s is expected for poll prio 2 -> 1 message every 2 seconds).
If i don't filter any messages with other filters at all the poll interval or lets better say the MQTT publish frequency is approx. 15 min for each message (in MQTT Explorer).
I assume this is because of the heavy load for a bus with only 2400 Baud rate ? Or what is the limiting factor here?
If I use the message definition modify method and set the messages I want to poll with r1 - r9 I have similar behavior.

Why is the poll priority value 1 not possible with automatic polling in the integration configuration? Wouldn't it be better to use a switch to enable polling for all messages and use a filter for messages that are not sent by ebusd? Maybe I have not understood the concept completely.

It also seems that the MQTT integration config doesn't stop broadcast or messages sent by other ebus masters (which appear on the bus without having a poll priority set for them) from being published as MQTT topics if those messages are filtered by filter-name = MESSAGE_NAME. Just like if I wouldn't use the MQTT integration config at all. I thought I could filter those messages with the config as well.

[VivLloyd]

Some clarification:
pollinterval [seconds] = frequency between each message read cycle in seconds. The default is once read/poll interval every 5 seconds.
priority [0,1-9] = message poll priority. Where 0 = don't poll. Messages with priority 1 are polled in each poll cycle, priority 2 messages only in every second poll cycle and so on.

Example (assuming pollinterval of 5 seconds):
If you had 100 messages each with a poll priority of 1, each message would be polled once every 500s.
If you had 100 messages each with a poll priority of 2, each message would be polled once every 1000s.

In summary, a message polling frequency is a function of the pollinterval, poll priority and total number of messages to be polled i.e. you cannot configure a fixed polling frequency for an individual message, but you can calculate it.

[toerni]

@tnn85
Did you eventually figure this out, since I am struggling with the same problem.
I want some specific sensors to be polled frequently and others should not be polled at all, since they just clutter the traffic

[VivLloyd]

You have a number of options.

1. Modify the configuration files by deleting all entries except the ones that you want. However, this means that you won't be able to perform ad-hoc reads/writes of other values as the messages won't be defined. Based on what remains you can tune the poll rate to meet your needs.
2. Modify the configuration files by setting the poll interval of each unrequired message to zero to disable polling. This way you can still perform ad-hoc reads/writes. Based on the an what remains enabled, you can tune the poll rate to meet your needs.
3. Leave configuration files unmodified and disable polling completely and reimplement polling in Node-RED. This gives you complete control of polling rates and is much simpler to modify/maintain. This is what I have done.

[tnn85]

@toerni
Not really. I think right now i am using these configuration options:
EBUSD_OPTS="--scanconfig -d /dev/ttyUSB0 -p 8888 --httpport=8889 --htmlpath=/var/ebusd/html --pollinterval=1 --accesslevel=* --configpath /etc/ebusd --mqttport=1883 --mqttjson=short

I am only interessted in a few messages (~20) anyway. I am not sure right now because I am not into the whole configuration of ebusd anymore (it was 1 year ago when i setup my configuration and it took a lot of time to understand everything).
I think i modified the configuration files for my needs (set the polling interval for single messages).
I didn't use the configuration for home assistant because it made it more difficult for the integration with InfluxDB / Telegraf.

Sorry I can't help you more.

[toerni]

@tnn85 tnx for the reply 🙏

[dietersiebs]

Modify the configuration files by setting the poll interval of each unrequired message to zero to disable polling. This way you can still
perform ad-hoc reads/writes. Based on the an what remains enabled, you can tune the poll rate to meet your needs.

How can I do this? With r0,,Name... it will nevertheless be polled.

I want to hold all findings in the csv files to see lesser not "decrypted" messages at the bus. But I want only some data...

[VivLloyd]

@dietersiebs I assume that you have cloned the ebusd configuration files (ebusd-2.1.x/en or ebusd-2.1.x) locally and updated them accordingly. Also you have configured your path so the ebusd loads from your local files. If you've not already done so, I suggest you temporarily rename one of the previously loaded configuration files and restart ebusd to verify it can't be found proving that it is being correctly referenced locally. Once this is done, take a copy of one of the loaded configuration files, rename the original and replace with a simplified version with only small number of sensors messages. Restart ebusd and verify only these sensors are being polled. Once this done, modify this file to add a poll priority e.g. change from 'r' to 'r5' and verify that that poll interval is 5 times longer. Then change the poll priority to r0 and verify it no longer gets polled. Alternatively comment out the sensor with '#' to disable polling (but you then won't be able to perform ad-hoc reads as the sensor message definition will be missing). Let me know how you get on.

[dietersiebs]

@VivLloyd I tested it, like you described.

• change config path to http//:ebusd.eu/config
• start ebusd (HA addon)
• the definition files are found, definitions are polled, some OK, some Invalid...
• stop ebusd
• I cloned the files from John30
• change config path to /config/ebcfg/de
• start ebusd
• the definition files are found, definitions are polled, some OK, some Invalid...
• stop ebusd and rename 08.hmu.csv, start ebusd
• [main error] unable to load scan config 08: no file from vaillant with prefix 08 matches ID "hmu00", SW0901, HW5103
• -> config path is correct
• stop ebusd, edit 08.hum.csv:

-#- type (r[1-9];w;u),circuit,name,[comment],[QQ],ZZ,PBSB,[ID],field1,part (m/s),datatypes/templates,divider/values,unit,comment,field2,part (m/s),datatypes/templates,divider/values,unit,comment,field3,part (m/s),datatypes/templates,divider/values,unit,comment,field4,part (m/s),datatypes/templates,divider/values,unit,comment
*r,,,,,,"B511",,,,,,,,,,,,,,,,,,,,,,,,,
r,,State,,,,,07,energy,,UCH,,,,,,energy,,kWh,,onoff,,UCH,0=off;1=on,,,state,,UCH,0x01=ready;0x0b=error;0x09=heating;0x11=cooling;0x81=heating_water,,
-#- Heatpump statistics,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
*r,,,,,,"B51A","05FF32",,,IGN:3,,,,,,,,,,,,,,,,,,,,,
r5,,EnergyHc,,,,,00,,,energy,,,,,,,,,,,,,,,,,,,,,
r5,,EnergyCool,,,,,01,,,energy,,,,,,,,,,,,,,,,,,,,,
r5,,Hours,,,,,40,,,energy,,,,,,,,,,,,,,,,,,,,,
r5,,HoursHc,,,,,41,,,energy,,,,,,,,,,,,,,,,,,,,,
r5,,HoursCool,,,,,43,,,energy,,,,,,,,,,,,,,,,,,,,,
-#- some more statistics from Live Monitor,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
r,,EnergyIntegral,,,,,21,,,integral,,°min,current energy integral in °min,,,,,,,,,,,,,,,,,,
r5,,CurrentYieldPower,,,,,23,,,D1B,10,kW,current yielded energy from the environment in kW,,,,,,,,,,,,,,,,,,
r5,,CurrentConsumedPower,,,,,24,,,D1B,10,kW,current power consumption of the heatpump in kW,,,,,,,,,,,,,,,,,,
r5,,CurrentCompressorUtil,,,,,25,,,D1B,,%,current compressor utilization in percent,,,,,,,,,,,,,,,,,,
-#- Total heating yields,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
-#-*r,,,,,,"B516","18",,,IGN:1,,,,,,,,,,,,,,,,,,,,,
-#-r,,YieldTotal,,,,,01,,,energy,,,,,,,,,,,,,,,,,,,,,
-#- Total electric power consumption,,,,,,,,,,,,,,,,,,,[...]

• start ebusd

• log entry -> [main notice] found messages: 39 (0 conditional on 0 conditions, 8 poll, 10 update)

• log:

2024-12-07 20:31:32.252 [update notice] sent poll-read hmu EnergyIntegral QQ=31: -68
2024-12-07 20:32:08.281 [update notice] sent poll-read vr_71 currenterror QQ=31: -;-;-;-;-
2024-12-07 20:32:20.218 [update notice] sent poll-read hmu EnergyHc QQ=31: 329
2024-12-07 20:32:26.190 [update notice] sent poll-read hmu EnergyCool QQ=31: 0
2024-12-07 20:32:32.229 [update notice] sent poll-read hmu Hours QQ=31: 4968
2024-12-07 20:32:38.232 [update notice] sent poll-read hmu HoursHc QQ=31: 1235
2024-12-07 20:32:44.180 [update notice] sent poll-read hmu HoursCool QQ=31: 0
2024-12-07 20:32:50.236 [update notice] sent poll-read hmu CurrentYieldPower QQ=31: 2.2
2024-12-07 20:32:56.242 [update notice] sent poll-read hmu CurrentConsumedPower QQ=31: 0.5
2024-12-07 20:33:02.221 [update notice] sent poll-read hmu CurrentCompressorUtil QQ=31: -112
2024-12-07 20:33:08.184 [update notice] sent poll-read hmu currenterror QQ=31: -;-;-;-;-
2024-12-07 20:33:14.224 [update notice] sent poll-read hmu DateTime QQ=31: nosignal;-:-:-;-.-.-;-
2024-12-07 20:33:20.213 [update notice] sent poll-read hmu EnergyIntegral QQ=31: -64
2024-12-07 20:33:32.229 [update notice] sent poll-read hmu Status01 QQ=31: 29.5;28.5;-;-;-;on
2024-12-07 20:33:56.177 [update notice] sent poll-read vr_71 currenterror QQ=31: -;-;-;-;-
2024-12-07 20:34:08.246 [update notice] sent poll-read hmu EnergyHc QQ=31: 329
2024-12-07 20:34:14.172 [update notice] sent poll-read hmu EnergyCool QQ=31: 0
2024-12-07 20:34:20.193 [update notice] sent poll-read hmu Hours QQ=31: 4968
2024-12-07 20:34:26.246 [update notice] sent poll-read hmu HoursHc QQ=31: 1235

I expected, that "EnergyIntegral" should polled every cycle, "EnergyHc" ando so on, only every 5th cycle, and in this case it should be lesser logged than "EnergyIntegral", but all definitions are polled in the same cycle.

Next test: I changed all definitions above to "r0,"

• log entry "[main notice] found messages: 39 (0 conditional on 0 conditions, 0 poll, 10 update)"

2024-12-07 20:39:19.189 [update notice] sent poll-read hmu CurrentConsumedPower QQ=31: 0.5
2024-12-07 20:39:31.289 [update notice] sent poll-read hmu CurrentYieldPower QQ=31: 2.1
2024-12-07 20:39:37.159 [update notice] sent poll-read hmu DateTime QQ=31: nosignal;-:-:-;-.-.-;-
2024-12-07 20:39:43.277 [update notice] sent poll-read hmu EnergyCool QQ=31: 0
2024-12-07 20:39:49.222 [update notice] sent poll-read hmu EnergyHc QQ=31: 331
2024-12-07 20:39:55.274 [update notice] sent poll-read hmu EnergyIntegral QQ=31: -47
2024-12-07 20:40:07.373 [update notice] sent poll-read hmu Hours QQ=31: 4969
2024-12-07 20:40:13.183 [update notice] sent poll-read hmu HoursCool QQ=31: 0
2024-12-07 20:40:19.253 [update notice] sent poll-read hmu HoursHc QQ=31: 1235

Both configurations "r5," and "r0," are ignored...

I don't know why.

[dietersiebs]

@VivLloyd is it possible, that the configuration with r0, r1, r2 and so on not works with the ebusd-Version (23.2.6) in the HA Addon (LukasGrebe)? Is it a feature of the 24.x?

[dietersiebs]

@VivLloyd : found out what's wrong!

For update only the configuration is not "r0" - I had to set it like this

*r,,,,,,B514,05,,,IGN:2,,,,,,,,,,,,,,,,,,,,,
*u,,,,,,B514,05,,,IGN:2,,,,,,,,,,,,,,,,,,,,,
r,,Fan1Sensor,,,,,1103FFFF,,,percent3,,,Gebläsedrehzahl 1 (Prozent der max.),,,,,,,,,,,,,,,,,,
r,,Fan2Sensor,,,,,1203FFFF,,,percent2,,,Gebläsedrehzahl 2 (Prozent der max.),,,,,,,,,,,,,,,,,,

u,,CondensateTrayHeaterStatus,,,,,1303FFFF,,,onoff,,,Kondensatwannenheizung (ein/aus)),,,,,,,,,,,,,,,,,,

u,,4PortValveStatus,,,,,1403FFFF,,,onoff,,,Vierwegeventil (ein/aus)),,,,,,,,,,,,,,,,,,
u,,ElectronicExpansionValveSensor,,,,,1503FFFF,,,percent3,,,Position EEV (Prozent),,,,,,,,,,,,,,,,,,
u,,HeatingCoilCompressorStatus,,,,,1703FFFF,,,onoff,,,Heizschlangenkompressor (an/aus),,,,,,,,,,,,,,,,,,

The "other character" has to be set in a top line AND in the definition line.

[VivLloyd]

@dietersiebs I'm running ebusd 23.3. I checked the behaviour on my system and it's definitely working as I explained. Here's a sample from one of my configuration files. Only sensors that have the prefix r5 are polled at the poll interval (default 5s)

Heatpump statistics

*r,,,,,,"B51A","05FF32",,,IGN:3,,,,,,,,,,,,,,,,,,,,,
r5,,EnergyHc,,,,,00,,,energy,,,,,,,,,,,,,,,,,,,,,
r0,,EnergyCool,,,,,01,,,energy,,,,,,,,,,,,,,,,,,,,,
r5,,Hours,,,,,40,,,energy,,,,,,,,,,,,,,,,,,,,,
r5,,HoursHc,,,,,41,,,energy,,,,,,,,,,,,,,,,,,,,,
r0,,HoursCool,,,,,43,,,energy,,,,,,,,,,,,,,,,,,,,,

Some more statistics from Live Monitor

r5,,EnergyIntegral,,,,,21,,,integral,,°min,current energy integral in °min,,,,,,,,,,,,,,,,,,
r0,,CurrentYieldPower,,,,,23,,,D1B,10,kW,current yielded energy from the environment in kW,,,,,,,,,,,,,,,,,,
r0,,CurrentConsumedPower,,,,,24,,,D1B,10,kW,current power consumption of the heatpump in kW,,,,,,,,,,,,,,,,,,
r5,,CurrentCompressorUtil,,,,,25,,,D1B,,%,current compressor utilization in percent,,,,,,,,,,,,,,,,,,

One other thing to consider is the ebusd HA integration which auto generates the HA configuration file. I believe this file may also contain the polling interval for each sensor. I'm not sure about this as I don't currently use HA. Refer to "https://github.com/john30/ebusd/wiki/MQTT-integration" for details,

[Dobie007]

Hi, First of all big thanks to john30 for his work!
My solution to improve the poll rate is the following: (I use the mqtt explorer to see what messages are published and how often.)

To reduce the overhead:
Copy the mqtt-hassio.cfg and change some things using visual studio (its easier to see what is active):
filter-name = status|temp|humidity|count|power|runtime|hours|starts|mode|curve|^load$|^party$|sensor|speed
In this line i removed "timer" and some other texts and added "speed" because i don't use the timers and wanted the "FanSpeed" which was missing.
Next i changed this line:
filter-non-name = timer|Hc3|Hc2|Z2|Z3
so that i don't get any entities starting with timer and none from Heating circuit 2,3 and Zone 2,3 (not used in my setup).
Store the cfg file under a different name and change the reference in the ebus config file.
I have not understood much yet but these two helped to reduce the number of polled messages quite a lot.
I assume you can filter with more intelligence to refine things even further.

To me it is unclear what is meant with "circuit" and what with "fields". Those have separate filter settings that i haven't made use of yet..