Feature Request: First half of day and Second half of day

[Noodleyman]

I am currently using automations to decide if I should charge to full, then discharge my battery based upon the PV forecast. When the Forecast is > 46kWh I have the battery discharge to near empty. This discharge process is dynamic and the time is based upon historical PV genration data. I do this because I have a large PV generation capacity (16kWh), and I want to make use of maxing exports and charging the battery using clipping (feature of the inverter)

In some cases, the overall daily forecast may be 50kWh for example. However, the majority of that generation may come from the last few hours in the afternoon and the morning could possibly be cloudy/raining etc. This means the battery potentialy gets discharged and there is not enough generation in the morning to charge the battery via clipping.

To make a better choice for the automations, it would be really helpful to have a half day forecast. First half and second half. In my use case, I would only be evaluating the first half.

The "day" being the total generation period, from the first time slice with a forecast > 0, to the last non-zero slice. The split between first and last half being the mid-section of that timeframe.

It would be really helpful if additional attributes could be made available to provide the data as above.

Describe alternatives you've considered
Considered attempting a custom sensor to achieve what I would like, however I am unsure on how to do that for the forecast data.

[autoSteve]

I don't think we should, @Noodleyman.

It's not a terrible idea, so don't get me wrong. The integration could easily calculate this, as attributes, but needs would vary. The data is there for the taking for those into a bit of Jinja2.

Here's something I use that kind-of does a similar thing to what you are looking for, and this actually features in a brand shiny new TEMPLATES.md which is yet to be merged to the MASTER branch. I'm making 'merge' noises for the approved PR #271 in prep for a v4.3.3 release proposed for late next week.

I participate in a virtual power plant. This is not fully automatic (probably because the power retailer looks like they run on the smell of an oily rag, and don't want to charge high prices, hence don't develop something which suits me fine.)

What I do between 11:00-14:00 is to charge 28kWh of battery from the grid for free (and from available PV from 14.5kW max). Then in the early evening when everyone is cooking dinner the feed-in tariff is generous, rewarding a good 'power dump' back for others to use, and so I do so keeping my electric bill in credit (I've not paid for power since September when I switched to this retailer).

I don't want to pay for any power in the morning should I go too hard with the dump. So enter my early hours forecast of kWh from sunrise and for the next two hours...

This template works out how much I need to leave in the tank to not run out by morning. It does this by utilising an average power consumption between sunset and sunrise, which is quite predictable for my situation.

Recently (with shittier weather here) this template has been getting it right to 6% (+/- 3%) remaining at the time in the morning when PV generation can sustain my (not inconsiderable) house base load of ~700W 24x7.

I expect to tweak it further in terms of the variables as I get a greater body of experience. (Example: During Winter the house heating fans kick on and off like they do not in Summer/shoulder consuming more power for gas heating overnight, and so the overnight average is probably going to need to be altered seasonally.)

If you want a morning and arvo forecast then grab the core concept of this template and have at it. If you get stuck, holler for help.

(edit: Revised code slightly.)
(This uses both the Sun and Sun2 integrations, BTW.)

template:
  - sensor:
      - name: "Battery dump minimum remaining"
        unique_id: "battery_dump_minimum_remaining"
        device_class: energy
        unit_of_measurement: "kWh"
        state: >
          {% set next_rise = states('sensor.sun_next_rising') | as_datetime %}
          {% set from_interval = next_rise.replace(minute=int(next_rise.minute / 30)*30, second=0) | as_local %}
          {% set to_interval = from_interval + timedelta(hours=2) %}
          {% set tomorrow = state_attr('sensor.solcast_pv_forecast_forecast_tomorrow', 'detailedForecast') %}
          {% set ns = namespace(total=0) %}
          {% for interval in tomorrow %}
            {% if interval['period_start'] > from_interval and interval['period_start'] <= to_interval %}
              {% set ns.total = ns.total + interval['pv_estimate'] * 0.5 %}
            {% endif %}
          {% endfor %}

          {% set sunset_to_sunrise_hours = (states('sensor.sun_next_rising') | as_datetime - states('sensor.home_sun_setting') | as_datetime).total_seconds() / 3600 %}
          {% set avg_hourly_use_overnight = 1.345 %}
          {% set base_minimum = sunset_to_sunrise_hours * avg_hourly_use_overnight %}

          {% set must_generate_at_least = avg_hourly_use_overnight * 2 %}
          {% if ns.total < must_generate_at_least %}
            {{ [base_minimum + (must_generate_at_least - ns.total), 0] | max }}
          {% else %}
            {{ [base_minimum, 0] | max }}
          {% endif %}
        availability: >
          {% set now_minutes = now().hour * 60 + now().minute %}
          {{ now() > states('sensor.home_sun_rising') | as_datetime and now_minutes > 5 and now_minutes < 1435 }}

[autoSteve]

Month-to-date, by the way, with getting closer to zero charge without net import the goal and the path to a bigger net-negative bill. I've clipped an optimal zero on just three days for April, but that's quite okay given forecast accuracy and evening power use variance.

It's a cash cow in Summer, and I'm yet to road test this in full-on Winter conditions. But like John Snow would rasp with dread, "Winter is coming..."

[autoSteve]

When you're done with your templates @Noodleyman, I'd love to include them in TEMPLATES.md as yet another real-world example of the smart stuff that folks do with a detailed solar forecast. That body of knowledge is scenario-based, laying out a use case and suggested solution.

A stretch goal for yourself is factoring the gap between 'confidence' levels of 10%/50%/90%. I've not yet approached that, but have it on the to-do list with my after sunrise generation calculation template to gauge how likely it is that I'll hit the zero target and adjust aggresively or conservatively.

Solcast can be wildly out even at 10% confidence, but they do get it fairly right most of the time.

[Noodleyman]

Cheers. I'm away for a few days as of tonight, so I'll review the feedback in more detail once back :) Thank you for the examples, I'll have to experiment with it and see what I can come up with. I can document my whole bespoke solution if you wanted to capture it?

[BJReplay]

This means the battery potentialy gets discharged and there is not enough generation in the morning to charge the battery via clipping.

Having spent about 8 years fiddling with charge algorithms, I've found that the easiest approach is the simplest: looking back and finding out at what time of day solar output exceeds house load. Obviously, this changes over the solar year, but it can be condensed down to seasons or months if you want. I just split it into daylight savings and non-daylight savings time, as it is good enough for me.

The next thing I determine is the battery level that I typically have at that time of morning after overnight discharge has met house loads.

I add 10% to that as my buffer to allow for a particularly cloudy morning.

That gives me my target morning state of charge.

One figure, at one time (two, if you take into account daylight saving).

Working backwards, I then have a target evening state of charge.

That is how much charge I need the battery to have to see me through until the next morning.

I can then use that target as my target for automations, and it makes for very simple automations: if remaining solar will exceed that, then I can do things like choose to discharge. If remaining consumption will mean that I have excess charge when the sun goes down, then I can discharge into high prices.

My automations are really exceptionally simple now: in the evening, discharge down to a target state of charge to earn income from high feed in prices if I have sufficient state of charge.

In the day, delay charging the battery while feed in is worth more (and feed in) if remaining solar is going to top me up (though that last one is currently disabled as I can charge from free every day).

[autoSteve]

I like simple, and I also like the 10% buffer idea, BJ. In my case I reckon I'm going to go a little more "complicated" by considering expected early morning generation rate and confidence level, and see if that allows for a fine tune.

Lower generation forecast and higher divergence between 10% & 50% = more buffer. Strong expected generation with less divergence = less buffer. I'm going to investigate a template sensor to calculate a buffer factor to use based on early morning forecast.

This likely won't suit many folks, and that's where simple = better. But I reckon it will suit my very predictable consumption pattern; I'm a creature of habit, and I'm the only one in the house.

This morning was a little too close for comfort for my set up. Still didn't pull from the grid, but SOC got down to 0.1kWh remaining... (😅)

[BJReplay]

Still didn't pull from the grid, but SOC got down to 0.1kWh remaining...

Yeah, I've got a backstop - if it is within three hours of dawn, and SoC is < 10%, reduce reserve from 5% to 1%.

I don't reduce to 0% because if it hits zero, it then consumes about 1kWh to return to 0%, whereas if it hits 1%, it just switches load to the grid, which is likely to use less grid.

That automation has fired twice - on very overcast mornings.

alias: Reduce Reserve
description: Drain battery if already low and dawn soon.
triggers:
  - trigger: numeric_state
    entity_id:
      - sensor.powerwall_percentage_charged
    below: 10
conditions:
  - condition: sun
    after: sunrise
    after_offset: "-3:00:00"
actions:
  - device_id: <id of powerwall gateway in telsa fleet>
    domain: number
    entity_id: <id of backup reserve percentage>
    type: set_value
    value: 1
mode: single

[autoSteve]

TEMPLATES.md merged into MASTER, @Noodleyman.

If you've got any scenario-based Jinja2 to contribute then we're all ears and would welcome PRs.

[autoSteve]

How'd you go, @Noodleyman?

[autoSteve]

This has been converted to a discussion. As a FR it will almost certainly never fly given there are simpler alternatives.

[Noodleyman]

Andddddd, back from a nice holiday! so can revisit this thread :)

I am doing almost the same thing as @BJReplay currently, but with some extras. I use PredBat to determine what my battery charge % should be overnight. I use EON Drive which gives me midnight -> 7AM at 6.7p per kWh. I have a 3.68kWh export limit (very rural) with 15.5kWp over 36 panels. I currently have a single battery but have a 2nd on order. thus the content below is for 1 battery. I'm using solar edge, SE8000H inverter.

the time my items trigger is dynamic based upon previous events.

I have some bespoke helpers in HA.

• Override Predbat (Toggle on/off)
• "PV generation last hit export+needs at what time?" as time.

I also have some bespoke sensors, those are:

      solar_generation_higher_than_needs:
        friendly_name: "Solar Generation higher than house needs + max export"
        value_template: >
          {% set pv_generation_now = (states('sensor.power_solar_generation_2')|int / 1000) %}
          {% set pv_target = (3.6 + (states(('sensor.power_consumption'))|int / 1000)) %}
          {{ pv_generation_now >= pv_target }}
      solar_start_battery_dump_at_time_when_in_override_mode:
        friendly_name: "On high forecast days, start dumping the battery to grid and set PredBat to override at what time"
        value_template: > 
          {% set original_time = states('input_datetime.pv_generation_last_hit_export_needs_at_what_time') %}
          {% if original_time != 'unavailable' %}
            {% set time_obj = strptime(original_time, '%H:%M:%S') %}
            {% set new_time = (time_obj - timedelta(minutes=185)) %}
            {{ new_time.strftime('%H:%M:%S') }}
          {% else %}
            unavailable
          {% endif %}
      solar_start_battery_dump_at_time_when_in_override_mode_ts:
        friendly_name: "On high forecast days, start dumping the battery to grid and set PredBat to override at what time as a unix timestamp"
        device_class: timestamp
        value_template: > 
          {% set original_time = states('input_datetime.pv_generation_last_hit_export_needs_at_what_time') %}
          {% if original_time != 'unavailable' %}
            {% set time_obj = strptime(original_time, '%H:%M:%S') %}
            {% set new_time = (time_obj - timedelta(minutes=180)) %}
            {% set current_date = now().date()|string %}
            {% set new_datetime_str = current_date + ' ' + new_time.strftime('%H:%M:%S') %}
            {% set new_datetime = strptime(new_datetime_str, '%Y-%m-%d %H:%M:%S') %}
            {% set new_datetime_with_timezone = new_datetime | as_datetime %}
            {% set new_datetime_utc = new_datetime_with_timezone.astimezone(now().tzinfo) %}
            {{ new_datetime_utc.isoformat() }}
          {% else %}
            {% set original_time = '10:00:00' %}
            {% set time_obj = strptime(original_time, '%H:%M:%S') %}
            {% set new_time = (time_obj - timedelta(minutes=180)) %}
            {% set current_date = now().date()|string %}
            {% set new_datetime_str = current_date + ' ' + new_time.strftime('%H:%M:%S') %}
            {% set new_datetime = strptime(new_datetime_str, '%Y-%m-%d %H:%M:%S') %}
            {% set new_datetime_with_timezone = new_datetime | as_datetime %}
            {% set new_datetime_utc = new_datetime_with_timezone.astimezone(now().tzinfo) %}
            {{ new_datetime_utc.isoformat() }}            
          {% endif %}

The "Override Predbat" toggle allows me to flip a switch, which triggers some automations. I can then use the custom helper in additional status checks within automations.

This will enable or disable read only in PredBat

alias: PredBat - Enable Read Only when Override Predbat switch is on
description: ""
mode: single
triggers:
  - entity_id:
      - input_boolean.override_predbat_today
    to: "on"
    trigger: state
conditions: []
actions:
  - target:
      entity_id: switch.predbat_set_read_only
    data: {}
    action: switch.turn_on

alias: PredBat - Disable Read Only when Override Predbat switch is off
description: ""
mode: single
triggers:
  - entity_id:
      - input_boolean.override_predbat_today
    to: "off"
    trigger: state
conditions: []
actions:
  - target:
      entity_id:
        - switch.predbat_set_read_only
    data: {}
    action: switch.turn_off

I have a safeguard hard coded to 14:30 to set the system back to the Max Self Consumption Mode, which will prioritize battery charging over export. If the battery isn't full by this time, it will focus on topping it up.

alias: >-
  Predbat - Disable Read Only  mode at 14:30 and set back to Maximize Self
  Consumption
description: ""
triggers:
  - at: "14:30:00"
    trigger: time
conditions:
  - condition: state
    entity_id: input_boolean.override_predbat_today
    state: "on"
actions:
  - metadata: {}
    data: {}
    target:
      entity_id: switch.predbat_set_read_only
    action: switch.turn_off
  - target:
      entity_id: input_boolean.override_predbat_today
    data: {}
    action: input_boolean.turn_off
  - delay:
      hours: 0
      minutes: 3
      seconds: 0
      milliseconds: 0
  - action: select.select_option
    metadata: {}
    data:
      option: Maximize Self Consumption
    target:
      entity_id: select.solaredge_i1_storage_default_mode
mode: single

I cross reference the PV forecast "today" to see if the forecated solar amount is > 46kWh. If it is, then I know I can dump my battery to the grid early in the morning to boost export income. This is more useful to me with a 3.68kWh export limit. The time at which the battery dump starts is based upon what happened last time we started this process, and at what time the house needs + max export was met by PV generation. I then start 3 hours earlier than the previously noted time. I add a delay here in case PredBat is in a round of processing, it gets time to wrap up a cycle after read only was enabled. I cross reference the bespoke sensor for my start time.

The 3 hours allows for the battery to get down to about 15%, as the PV generation ramps up during that window of time as well. Thus there is always some battery power to make coffee!

alias: >-
  Predbat/Solar - Enable Read Only mode as early as is viable IF PV forecast is
  more than 46
description: ""
triggers:
  - trigger: time
    at: sensor.solar_start_battery_dump_at_time_when_in_override_mode_ts
conditions:
  - condition: numeric_state
    entity_id: sensor.solcast_pv_forecast_forecast_today
    above: 46
  - condition: state
    entity_id: input_boolean.override_predbat_today
    state: "off"
actions:
  - metadata: {}
    data: {}
    target:
      entity_id: switch.predbat_set_read_only
    action: switch.turn_on
  - data: {}
    target:
      entity_id: input_boolean.override_predbat_today
    action: input_boolean.turn_on
  - delay:
      hours: 0
      minutes: 6
      seconds: 0
      milliseconds: 0
  - action: select.select_option
    metadata: {}
    data:
      option: Discharge to Maximize Export
    target:
      entity_id: select.solaredge_i1_storage_default_mode
  - delay:
      hours: 0
      minutes: 0
      seconds: 15
      milliseconds: 0
  - action: input_number.set_value
    metadata: {}
    data:
      value: 5000
    target:
      entity_id: number.solaredge_i1_storage_discharge_limit
mode: single

Once the PV generation covers max export + house needs, the following automation fires, which sets CLIPPED charging mode and updates the time of day. This mode will take PV generation, cover house needs, then export and whatever can't be export then goes into battery.

alias: >-
  Predbat/Solar - When PV generation is more than max export + home requirement,
  set to CLIPPED
description: ""
triggers:
  - trigger: state
    entity_id:
      - sensor.solar_generation_higher_than_needs
    from: "False"
    to: "True"
conditions:
  - condition: state
    entity_id: input_boolean.override_predbat_today
    state: "on"
  - condition: state
    entity_id: select.solaredge_i1_storage_default_mode
    state: Discharge to Maximize Export
actions:
  - action: input_datetime.set_datetime
    metadata: {}
    data:
      time: "{{ now().strftime('%H:%M:%S') }}"
    target:
      entity_id: input_datetime.pv_generation_last_hit_export_needs_at_what_time
  - delay:
      hours: 0
      minutes: 0
      seconds: 15
      milliseconds: 0
  - action: select.select_option
    metadata: {}
    data:
      option: Charge from Clipped Solar Power
    target:
      entity_id: select.solaredge_i1_storage_default_mode
mode: single

If the battery gets full, another automation fires, which will set the mode to max self consuption. aka, use the battery before export. Control is then given back to Predbat as well.

alias: >-
  Predbat - Disable Read Only mode when battery becomes full and reset to
  Maximize Self Consumption
description: We hit our goal, give back control.
triggers:
  - entity_id:
      - sensor.solaredge_b1_state_of_energy
    trigger: state
conditions:
  - condition: numeric_state
    entity_id: sensor.solaredge_b1_state_of_energy
    above: 96
  - condition: time
    after: "07:45:00"
    before: "14:29:00"
    weekday:
      - sun
      - sat
      - fri
      - thu
      - wed
      - tue
      - mon
  - condition: state
    entity_id: input_boolean.override_predbat_today
    state: "on"
actions:
  - metadata: {}
    data: {}
    target:
      entity_id: switch.predbat_set_read_only
    action: switch.turn_off
  - target:
      entity_id: input_boolean.override_predbat_today
    data: {}
    action: input_boolean.turn_off
  - delay:
      hours: 0
      minutes: 3
      seconds: 0
      milliseconds: 0
  - action: select.select_option
    metadata: {}
    data:
      option: Maximize Self Consumption
    target:
      entity_id: select.solaredge_i1_storage_default_mode
mode: single

There is an oddity with the above, in that it's almost better to just charge battery to full from midnight in preperation for battery dumping rather than let predbat do it. I liked that PredBat would choose to only charge battery to a lower % sometimes, but this doesn't play nice with the max export model. so to work around this.... another automation based on PV forecast.

alias: >-
  Solar/Predbat - Turn Off Predbat after 1:30AM forcast update and Charge
  Battery to 100% ready to be dumped later
description: ""
triggers:
  - trigger: time
    at: "01:35:00"
conditions:
  - condition: numeric_state
    entity_id: sensor.predbat_pv_today
    above: 46
actions:
  - action: switch.turn_on
    metadata: {}
    data: {}
    target:
      entity_id: switch.predbat_set_read_only
  - delay:
      hours: 0
      minutes: 6
      seconds: 0
      milliseconds: 0
  - action: select.select_option
    metadata: {}
    data:
      option: Charge from Solar Power and Grid
    target:
      entity_id: select.solaredge_i1_storage_default_mode
  - delay:
      hours: 0
      minutes: 0
      seconds: 15
      milliseconds: 0
  - action: input_number.set_value
    metadata: {}
    data:
      value: 5000
    target:
      entity_id: number.solaredge_i1_storage_discharge_limit
  - delay:
      hours: 0
      minutes: 0
      seconds: 15
      milliseconds: 0
  - action: input_number.set_value
    metadata: {}
    data:
      value: 5000
    target:
      entity_id: number.solaredge_i1_storage_charge_limit
  - delay:
      hours: 0
      minutes: 0
      seconds: 15
      milliseconds: 0
  - action: number.set_value
    metadata: {}
    data:
      value: "{{ states('input_number.minimum_battery_reserve') | float }}"
    target:
      entity_id: number.solaredge_i1_backup_reserve
  - delay:
      hours: 0
      minutes: 0
      seconds: 15
      milliseconds: 0
  - action: number.set_value
    metadata: {}
    data:
      value: "{{ states('input_number.minimum_battery_reserve') | float }}"
    target:
      entity_id: number.solaredge_i1_backup_reserve
mode: single

Everything above works great... when the PV generation is in the morning... but if it's a slow start to the morning the battery will get emptied and fall back to grid. Thus the request to evaluate the morning PV forecast.

However, I've just pulled the trigger on another battery, so when this arrives I will change my setup as in theory I'll not need to use the grid at all for charging until October onwards. an alternate approach would make more sense.

[Noodleyman]

I also had to add this, so the wife knows when she can go nuts with the high power appliances hah.

[BJReplay]

My... checks notes... 9.04kWh forecast for tomorrow quavers in fright.

That said, I too have handy tell-tales for WAF.

If the Charge number isn't Red, it's pretty much OK to run dishwasher or washing, but not Oven until it climbs back up.

Our load and charge time is now always in the middle of the day: sun comes out, power is free (as in beer and speech - 0¢/kWh between 11am and 2pm) so that's when everything runs, including car charging, household battery charging. Dump some excess to the grid to earn a few cents in the evening (15¢/kWh which, combined with a credit of $1/day for not using grid power between 5pm and 7pm means effectively no connection charge), and happy days.

[Noodleyman]

Oh, and I've also setup backup reserves using weather alerts as our power is flakey in most storms. we have the backup module so can run during a power outage. This essentially checks weather alerts, sets the reserve accordingly. I do need to boost this to use the PV forecasts to charge further in advance and hold the charge. currently it'll charge from grid a few hours before the weather warning comes into effect.

[BJReplay]

Oh, @Noodleyman I have a half day forecast solution for you:

Set up your last x hours sensor to 12 hours (or even, if you want to run this automation at 8pm each night, 16 hours): It won't be as dynamic as everything else, but it sounds like you need it in advance, in any case.

So, if you set it to 12 hours, then at around midnight, that sensor will give you the forecast to midday (your morning sensor), and the full day sensor will give you the forecast for the full day. A simple template subtracting the morning sensor from the full day sensor gives you the afternoon sensor.

Or pick a time that you know works year-round (in your strange land of sun is still shining at 10pm at night): say 11pm, 13-hour sensor, run your automation at 11pm, and use 13-hour sensor and next day for your two figures.

[Noodleyman]

Thanks for the suggestion, I'll check this out when I've a little more time to play around with it and check how that works. seems like it might do what I need though. I appreciate the suggestion