Support multiple local Tesla Energy Gateways (Powerwall/Inverters) using the server for local or cloud access

[Nexarian]

I'm referencing this line: TODO: Add support for multiple Powerwalls

As I have two side-by-side (SxS) Tesla 7.6 kw inverters, this feature is a must for me if I'm going to monitor my system using Powerwall dashboard, especially important is TEDAPI mode and a facility to alternate network routes to interleave data from the two.

This issue is to track development. I intend to tackle this myself.

[Nexarian]

The first step was to solve the ability to query two Tesla Energy Gateways on a LAN by using variations in NAT: #128 (comment)

The second is to get the pypowerwall proxy server to be able to handle multiple gateways. I am working on this, but it's not done yet: Draft PR

Essentially the proxy server needs to compartmentalize each inverter into its own separate proxy server and then route each request from the virtual IPs configured in the scripts above. That simply requires a ton of refactoring and modularization.

[jasonacox]

@Nexarian 🙇 The draft looks good! Thanks for kicking that off and for the much needed refactoring. I'll put my additional comments in the PR.

[Nexarian]

@jasonacox Still not quite finished! I've been going through it line-by-line, doing my best to preserve functionality while increasing modularity and maintainability. I also haven't actually tested this yet, because this refactor is so invasive I can't really do that until I'm finished.

The main points in the PR are:

• Remove all important state from global variables. Instead, things like config, pypowerwall, and the like will be accessed as class members of the Handler (or perhaps another class after an additional refactor) and/or function arguments.
• Because things are now compartmentalized/modularized, accessing more than one TEG is a matter of setting up a loop.

A few things things I'm still not sure yet is how to handle:

• proxystats. They seem to be a bit ad-hoc as to when they are manipulated and when they're not.
• How to actually read configuration for each TEG/Pypowerwall object. What I'm moving towards is an "INI"-like file format, that lists each:

[Powerwall 1]
var=value
...
[Powerwall 2]
var=value
...

However, we could conceivably move this to JSON or YAML as well, but that's likely to break many things.

Thanks for your support on this, it's taking me forever to get this honed in! Several late nights of coding so far!

Also: I understand that a massive refactor like this is a big pill to swallow. I'm happy to "part out" the finished version into more digestible chunks so we migrate the server to the final result slowly, while I continue to test the full version on my own systems.

[jasonacox]

Thanks @Nexarian - I haven't seen anything otherwise, but please keep these project design principles in mind:

• Simplicity over Cleverness - As a open source repo, our bias should be easy to read and maintain, simple as possible, not clever structures or approaches that may be more efficient or concise but difficult to read. We have a lot of first time contributors who are just learning python. I think that indicates we have done a good job at keeping it simple enough to understand for those learning to code.
• Maintain API - Changes should not require existing installations to change their approach or use of the tool/library. When API changes are needed, they should be justified, documented and as much as possible, avoided or mitigated for easier adoption.

Note on your suggested refactoring - for the most part I think these make it more readable, just to be clear.

A few things things I'm still not sure yet is how to handle:

• Proxystats - this is a troubleshooting tool and intent is to represent config, calls and errors (operational metrics) - open to new ways to capture and serve, but this is an expected feature we would want to maintain.
• Config settings - we should maintain the environmental variables as a way to pass the config settings into the container. That doesn't mean we can't include a new variable that has a JSON payload. If that variable is not null, it can be set to override the singleton approach from the existing env vars.

I'm happy to "part out" the finished version into more digestible chunks

We should target MVP so that each incremental is still a working proxy, backward compatible, but could slowly walk toward the final outcome. What is in scope for MVP? I'm having some difficulty thinking what would be broken out unless you are thinking about deeper refactoring.

[Nexarian]

For me, at least, the MVP is the proxy can handle two simultaneous calls (using the virtual IPs above) to retrieve internal TEG/string data in such a way that two sets of string data/internal TEG data can show up on the Powerwall dashboard. Each inverter has its own TEG rather than a standalone Gateway like most Powerwall setups have. Why Tesla chose that route, I'm not sure, but it's what makes my situation harder.

As you mentioned the other way to do this might be to simply have two containers that report into InfluxDB, or to multiplex them with my earlier solution that alternates static routes. My solution is certainly "fancier" and thus may violate simplicity over cleverness. Still, I don't think it unreasonable to say the system should support querying multiple TEGs simultaneously without fancy Docker manipulation (which might ALSO be a violation of that principle).

MVP is two because I have two, but really the longer-term goal should be N (multiple).

As to the config, I will write it such a way that a simple "one TEG" setup will work, but it will also read multiple as well. Similar to how a black & white TV can read a color signal :)

I am actually not planning on changing the API other than supporting multiple TEGs.

[Nexarian]

As to how to part it out, I think there are multiple phases:

1. Move global variables into a more structured format (Mainly config and proxy stats)
2. Change the post and get handlers to use functions with a dictionary selector rather than a gigantic if block.
3. Use this modularization to support multiple TEGs.

In no step would the API or functionality change, except for the expansion in the last one.

[jasonacox]

Yes, I agree with you. There are several in the community that have multiple gateways (very larger systems) so this enhancement would work for them as well.

The complexity concern is more related to the code readability than the functions. I think your proposal is very reasonable.

[Nexarian]

The server.py refactor now works on my Raspberry PI; at least, it does for local access. I decided against using a config file given the way that environmental variables are used. You can simply do this:

PW_DEBUG=no
TZ=America/New_York

# Inverter 1
export [email protected]
export PW_PASSWORD_1=
export PW_GW_PWD_1=GATEWAY_PASSWORD
export PW_HOST_1=VIRTUAL_IP
export PW_PORT_1=8675
export PW_TIMEZONE_1=America/New_York
export PW_STYLE_1=solar

# Inverter 2
export [email protected]
export PW_PASSWORD_2=
export PW_GW_PWD_2=GATEWAY_PASSWORD
export PW_HOST_2=VIRTUAL_IP
export PW_PORT_2=8676
export PW_TIMEZONE_2=America/New_York
export PW_STYLE_2=solar

I will continue testing. Feel free to check it out now!

[jasonacox]

I love that approach @Nexarian - I was worried about the config file as well.

[Nexarian]

Alright, I've made some progress on getting this to work on Powerwall-Dashboard, and... It's a bit trippy. See jasonacox/Powerwall-Dashboard#573

First, my crazy network script makes routing changes to eth0 (or eth1 or equivalent), which complicates the Docker setup. For now, I've changed the docker images in the compose to use network_mode: host but that's a bit of a hack. The "right" answer might be to have all the docker containers share the same networking interface and then have them all depend on a setup script that adds the routing to the docker network.

Because I'm flipping on network_mode: host, that means the containers can't reference them by their names, only by their ports.

Getting Telegraf to connect to multiple TEGs was trivial though.

Finally, as you mentioned earlier, we need to think about how to make the dashboard look good with multiple inverters. I thought it was a matter of doing something like this, but it's not. Looks like there are both InfluxDB queries and Grafana queries to change before that'll work correctly.

[jasonacox]

On telegraf, since the keys are all the same, are they both getting recorded?

.
For example solar_instant_power would come in from both gateways. II wonder if we could use telegraf to add a suffix to the key for the 2nd, 3rd, ... xth, gateways, and then build the dashboard with that. And it would make it easier to essentially line copy the influxdb CQ's for each gateway and rename the keys there by adding the suffix.

[[input.http]]
	urls = [
		"http://pypowerwall:8675/aggregates",
		"http://pypowerwall:8675/soe",
		"http://pypowerwall:8675/strings",
		"http://pypowerwall:8675/temps/pw",
		"http://pypowerwall:8675/freq",
		"http://pypowerwall:8675/pod"
	]
	method = "GET"
	insecure_skip_verify = true
	timeout = "4s"
	data_format = "json"
	name_suffix = "_pw2"

There may be an easier way... I'm pretty zapped today,.

[Nexarian]

It does seem like the keys are overwriting each other, yes, though all I know for sure is I simply see one set of 4 strings.

Is there a script that builds the dashboard? Editing each of the charts looks to be very tedious.

[jasonacox]

Each one is hand crafted inside the Grafana UI, I'm afraid to say. BUT I would only use dashboard.json, you can ignore the rest.

[Nexarian]

My question was more about this: https://github.com/jasonacox/Powerwall-Dashboard/blob/main/influxdb/influxdb.sql

Where did this come from? Also, the strings 1-11 queries confuse me, are there some Tesla systems that have 6 strings? (E & F?)

Also: Inverters 1-6?

Does this need to change to support multiple TEGs or should I leave it alone?

[jasonacox]

influxdb.sql Where did this come from?

Also a work of art.. I mean, hand crafted by me and the community. :)

Also, the strings 1-11 queries confuse me, are there some Tesla systems that have 6 strings? (E & F?)

Yes, Powerwall 3 have 6 strings (A-F). Over time, the community has expanded this to accomodate several different configurations (which I don't have) like 3 Phase services and Powerwall 3's.

Does this need to change to support multiple TEGs or should I leave it alone?

If we want one InfluxDB for all the TEGs, then yes. It would need to have CQs set up for each. Basically copy each row and duplicate it for each TEG but with the related suffix on the SELECT data elements (e.g. site_instant_power would be site_instant_power_pw2, site_instant_power_pw3, etc). Those would project into (SELECT INTO) things like home, home_pw2, home_pw3, etc. We would then create dashboard panels that use those new elements.

Yes, tedious. And someone nice may want to write a script that builds this all and we would love that person. 😀 However, I also have another idea, which may be more complex on the proxy side but easier on the user side. We could have the proxy do these aggregation, essentially sum up all the TEG values into a combined output for at least these key APIs used by the dashboard:

• "http://pypowerwall:8675/aggregates",
• "http://pypowerwall:8675/soe",
• "http://pypowerwall:8675/strings",
• "http://pypowerwall:8675/temps/pw",
• "http://pypowerwall:8675/freq",
• "http://pypowerwall:8675/pod"
• "http://pypowerwall:8675/alerts/pw"

Some would need to be augmented (.e.g strings and alerts) to somehow identify the separate TEGs. This would likely require some logic change in the proxy, or perhaps we just add another endpoint (e.g. port 9675) that is the sum of all the TEGs.

[Nexarian]

I've created #133 as a separate issue to resolve this. I think, given how complex these queries are, we DO need to generate them. Even modifying this file to handle my 2x inverter setup feels super tedious and error prone. I may need to solve that before I continue here; though I don't think it'll be hard. Something like Jinja2 makes short work of most templating problems.

As to modifying the proxy to have some endpoints that aggregate and some endpoints that report metrics per TEG -- Yes, we need that. My Powerwall Dashboard "works" -- But many graphs and values are empty or zero, and the animation is not correct either, and that's because of this tension.

The Tesla public app/API effectively does the aggregation for us, but we can't rely on that in TEDAPI local mode.

[jasonacox]

Agree on all of that. Thanks @Nexarian . Also, I would love to see a screenshot of your dashboard, even current state. 🙏

[Nexarian]

I'm experimenting with appending the DIN to the alerts, this breaks color scheme :)

[Nexarian]

The waves in the graph are my heat pump kicking in. It's cold here!

[jasonacox]

That's awesome! Thanks @Nexarian ... and keep warm! ;-)