Skip to content
This repository has been archived by the owner on Apr 22, 2024. It is now read-only.

IP change not detected #373

Open
madninja opened this issue Aug 16, 2021 · 150 comments
Open

IP change not detected #373

madninja opened this issue Aug 16, 2021 · 150 comments

Comments

@madninja
Copy link
Member

Many EU ISPs change consumer IP addresses every 24-48 hours.

There have been a number of reports where the IP address changes, and while the API has a recent peerbook record the IP address was not updated to reflect the change either never, or it took a very long time which put it past the time for the next IP update.

Restarting the hotspot seems to fix it every time this was observed.

The theory is that IP change detection is failing, not being done often enough or not being done at all?

@maco2035
Copy link

This occurs in the helium OG, and rak's. This happens on occasion. With about 50+ miners this has only occurred a few times with me. I have use a wifi outlet that I can remotely turn off and on. It is a minor inconvenience for me, but for others this might be worse as the hotspots aren't able to be adjusted properly.

@leyuansugar
Copy link

This problem also happens with bobcat.

@wghewitt
Copy link

wghewitt commented Aug 19, 2021

NO BEACONS

I've recently experienced the exact problem with 2 of my hosts and miners where they did not send any beacons for over 9 days which was concerning, when we rebooted their routers beacons started to happen again in the next few hours.

This problem has now become more of a recent report of events by several people in the South African Helium Community, as our ISP also changes IPs every 24-48 hours.

And there are now several hotspots where you would see they have not sent beacons at all in the last week or more and they are not even aware of this problem or how to resolve it.

This is becoming a problem now for us as we are starting to witness less as beacons are not being sent due to this problem.

And we have seen this with Bobcat and Sensecap miners.

Hope this will get as much attention to be resolved as soon as possible, please.

@Vagabond
Copy link
Contributor

Can anyone affected help me troubleshoot this? I need access to an affected hotspot, or at least the logs from it.

@chillbreak
Copy link

what exactly is needed, i can provide it because i am affected by this problem.

@abhay
Copy link

abhay commented Aug 21, 2021

@chillbreak a hotspot name would be helpful. It’s easiest for the core team if it’s an OG or Rak

@chillbreak
Copy link

Prehistoric pearl dinosaur

@radicale
Copy link

I've seen the issue and reproduced it with the following test:

  1. Hotspot behind CG NAT (shared IP) and thus relayed
  2. Route hotspot through a VPN to a public IP
  3. Forward port 44158
  4. Peerbook "listen ip" is updated from the relay address to the CG NAT shared IP, and not the VPN public IP that it's actually using.
    It is as if the hotspot caches the IP it initially discovered and never bothers to check it again.

@Vagabond
Copy link
Contributor

I know what the problem is, I just don't know why it happens. Prehistoric pearl dinosaur is not a hotspot I have support access too, unfortunately.

@westerwalder
Copy link

A simple reboot of the miner/hotspots resolves the issue, as during reboot it checks for an public ip change. This has to be implemented periodically, as it is nothing high sophisticated...
But this issue is quite critical, as in some cases this would result in an unreachable miner for several days/months.

@radicale
Copy link

I know what the problem is, I just don't know why it happens. Prehistoric pearl dinosaur is not a hotspot I have support access too, unfortunately.

Perhaps try to reproduce it by changing the outward facing IP of a hotspot or test setup? Are you having trouble reproducing the issue?

@shawaj
Copy link

shawaj commented Aug 22, 2021

@Vagabond if you DM me and we have a willing volunteer from NebraLtd/helium-miner-software#108 I can give you logs / info about the HS. I know @dakky221 has this issue

@dakky221
Copy link

Yeah, I'm having the same issue, and I "fixed" it by plugging the miner into a smart-plug and making it reboot the miner on IP change, but that will only decrease the lifespan of the hotspot and must be other way around, like @westerwalder said - add periodically check for public ip, for example every 5 minutes, it would fix the issue entirely.

@dakky221
Copy link

btw @shawaj unfortunatley my Nebra is now behind CGNAT on a semi-static IP so my logs won't really help. I can share the Sensecap's logs though if there is a way to get them.

@ghost
Copy link

ghost commented Aug 22, 2021

Yep, plugged my hs on a timer as well to reboot every 24h. Currently takes 2-3 hrs to reflect updated IP, but lag seems to increase (week before: ~1hr) due to network growth(?).

@westerwalder
Copy link

By (unsafely) rebooting it that often, you can directly say goodbye to the sd-card/filesystem..

@ghost
Copy link

ghost commented Aug 22, 2021

I have cloned SD-Cards as backup should that happen. But I'm about to switch to tunneling a static IP instead.

@dakky221
Copy link

By (unsafely) rebooting it that often, you can directly say goodbye to the sd-card/filesystem..

Well, hopefully the libp2p gets updated to periodically check for external IP before the sd-card goes kaboom ;)

@Vagabond
Copy link
Contributor

I should have a fix for this today or tomorrow that we can test.

@wghewitt
Copy link

wghewitt commented Aug 23, 2021

I should have a fix for this today or tomorrow that we can test.

That is fantastic, will this fix be rolled out in an OTA on all miners or how will the update be released?

@dakky221
Copy link

I should have a fix for this today or tomorrow that we can test.

I can renew the IP as many times as you wish on my VDSL2 connection, but the hotspot is Sensecap, so if they can deploy the updated firmware we can test.

@ci-work
Copy link
Contributor

ci-work commented Aug 24, 2021

possible related: #376

@Vagabond
Copy link
Contributor

Has this improved any now both PRs are merged and deployed?

@dakky221
Copy link

There is no way to be sure if the fix worked or it didn't. Listen addr doesn't change on external IP change, but I guess that's normal because it gets it from the API and API is too slow to be of any use. I also guess manufacturers should add to the dashboard some kind of a log if the hotspots sent the new IP to the peers to be sure...

@leyuansugar
Copy link

My isp updated the ip address about 10 hours ago, but looking at the address on the api it still shows the previous one. but when I just checked inside peerbook it shows the correct ip address, of course I had rebooted the hotspot before that so not sure if it was the reboot that made the peerbook change. but as of the time of the comment the api still shows the wrong address.

@Vagabond
Copy link
Contributor

Rebooting the hotspot will find the new address, yes. The API may not have a perfectly up to date view of the system so it may take a while to see the IP change (if the hotspot notices it).

@dakky221
Copy link

OK, just to add: I didn't reboot the hotspot since the IP change. 10 hours already passed and it's not updated on peerbook or API. Listen addr. is still the yesterday's IP. Will reboot hotspot now, I guess 10 hours is enough of waiting to see that fix isn't working.

@aflinfo
Copy link

aflinfo commented Dec 15, 2021

Same issue here, changed my router and ISP assigned new IP, still not changed in the API... Tried to reset the miner but no luck still old IP displayed and the miner status shows offline API and explorer, despite this is online and synced as per local diagnostics tool via Bluetooth. :(

@Passiveminer
Copy link

i have sensecap and have switched ISP to get static public IP, peace of mind now, would suggest same for everyone as its a well known bug and there is no fix yet though almost 2,3 weekly updates are beings pushed for other issues.

@aflinfo
Copy link

aflinfo commented Dec 16, 2021

Thanks! I have requested my ISP a static one. Issue resolved, but not all ISP will grant that or they will even charge that. Agree with all this needs to be resolved at some point . Thanks!

@SuchTorment
Copy link

I’ve not been earning anything in a decent location ,So on helium geek app it checks my ports for me but my ip on there is not the same as what it says it is on my router is this the same issue? Regards

@aflinfo
Copy link

aflinfo commented Dec 17, 2021

Not sure. I was able to change to a static IP and I will monitor over the time now. The only thing when I query my hotspot via API it still shows the old IP :( and port . When I test new IP and port forwarding this is working fine (you can use https://www.canyouseeme.org/) also when you run local miner diagnostic if inbound and outbound is fine (green) I would say you should be fine. Last but not least after IP change I saw a significant decrease in my rewards… looks likes this kind of changes take longer to reflect…

@SuchTorment
Copy link

I can’t see my address when I click the api tab in the diagnoser hmmmm

@techygeekshome
Copy link

I've got this problem, my external IP changed by my ISP about 8 days ago yet not updated on my SenseCap M1. How is this fixed?

@old-square-eyes
Copy link

old-square-eyes commented Dec 24, 2021

Got two hotspots. Set up my second location one at home so not to bother the host too much. It took an IP of my home, and I synced. Powered down gracefully and connected it at my friends. Paired to their WiFi. All day online not a single block synced (ports open Senscap dashboard up and device online). API has different IP to the Senscap dashboard.

Key point... it was powered off, and on again at the new host/location. Then it was asserted a location (for the first time). Still not syncing anything and still incorrect IP after 12 hours. I'm dead in the water until I can return to safely reboot it - assuming that works.

@quarky42
Copy link

I've seen this problem with multiple SenseCAPs that I own. It can take days for them to report the correct listening IP address even after multiple reboots. Sometimes a reboot seems to help (once or twice), however I rebooted one today 4 hours ago that is still showing the wrong listening IP.

@old-square-eyes
Copy link

old-square-eyes commented Dec 30, 2021

Mine came right, after a couple of reboots (and a couple of stressful days). I also think the Helium API is mega slow which is another vector for issues. I was fully synced for nearly a week before the Helium mapper recognised it. Seems like major issues with the network at both ends. As an experienced "miner" - GPU, and also Chia - the barrier to entry and lack of real-time updates/debug feedback is a major issue. You change one thing and there's no telling if it works for days. In the meantime you change 5 other things out of desperation. If this technology is to become mainstream it HAS to scale better.

@quarky42
Copy link

quarky42 commented Dec 30, 2021

I've had several reboots at this point (2 using the reboot button on the port 80 local interface and once with a hard power down because I moved the miner.

Using this interface:
https://api.helium.io/v1/hotspots/name/hotspot-name-here

It shows me that the listen_addr reported by the API is "null". A miner of mine that is working shows the proper address and port there instead of null.

Yes, I understand / agree that it is impacted by the API itself, but sure seems weird what updates relatively quickly and something like the listen address isn't showing the right IP for days.

I suspect this will be impacted by light hotspots. Not sure though.

@adityaacharya1989
Copy link

adityaacharya1989 commented Jan 17, 2022

Any resolution for this issue? I have the same problem. Listen address was last monitored for my hotspot on "timestamp":"2022-01-08T20:03:38.611000Z". My Public IP changed on 14th Jan. Have been waiting since to get this updated on "https://api.helium.io/", earnings have taken a serious hit. Hotspot Name: Fast Smoke Rhino.
This will be a recurring issue for folks who do not have a provision for applying for Static Public IP's. Request the devs to provide a fix for this. Even if there is a possibility to force assert IP(similar to Assert Antenna Gain), I believe the community will appreciate it.

@irdeto383
Copy link

irdeto383 commented Jan 18, 2022

The Question is.... why dont regognize the miner that the public ip has changed? or do the miner regognize it and the time to write it in peer /api needs to long? more then 24 h?
fact is that the most isp in germany have dynamic public ips and they change it all 24 hours...
my miners...sensecap m1 with newest firmware 2022.01.13.1. public ip doesnt match with listen ip ...follows no beacons send and challenges are allways 0 after few days
Maybe the bug with erlang-libp2p again in 2022.01.xxxx ?

thanks

@helium helium deleted a comment from manipayment Feb 8, 2022
@danielcharrua
Copy link

Any updates on this issue? I have read on some help articles that the update on the Helium API can take from hours to days. In my case 48 hours has passed without updating the listen_addrs value in the API.

On the miner I can see is connected through my public IP and port 44158 but on the API this is not updated.

I even do a "status check" on hotspotty and all seems good, "not relayed". But on the Helium API, the listen_addrs is through p2p instead of /ip4/xxx.xxx.xxx.xxx/tcp/44158 so I get the "relayed" alert.

@ci-work
Copy link
Contributor

ci-work commented Feb 11, 2022

some observations from the past few months - and monitoring several thousand devices - would indicate that some issues in libp2p somewhere

  • some hotspots do not send out gossip of their peer status regularly (sometimes days go past without any), this can randomly hit any device
    -- n.b. we are validating this by requesting peer book entries from multiple randomized sources in batches, not relying on propagation through network
  • relayed hotspots tend not to gossip new listen_address when they switch relayers
  • outbound_gossip_connections may benefit from an increase to 3-5 rather than the current 2, to aid propagation

@techygeekshome
Copy link

My hotspot has not updated its IP for 10 days now and its showing as reIayed either though the dashboard for the device is showing it as not reIayed. Earning virtuaIIy nothing. Needs sorting.

@quarky42
Copy link

quarky42 commented Feb 12, 2022 via email

@techygeekshome
Copy link

techygeekshome commented Feb 12, 2022 via email

@quarky42
Copy link

quarky42 commented Feb 12, 2022 via email

@Passiveminer
Copy link

Passiveminer commented Feb 12, 2022

Static IP is the only solution, got static IP after figuring out issue, at least this pain is gone for good. one miner per network, if more than one miner on same network, one will be fine ,one that is connected to ethernet cable, others will be in relay mode, unless vpn is setup for relayed ones. ABOVE FEEDBACK IS FOR UK WITH XLN.

@Passiveminer
Copy link

Port 44158 shows as open, using a Sky Fibre (UK) NAT type is none, tried with static too, aIso tried with uPNP on and off, but currentIy off. HeIiumstatus.io and hotspotty show it as onIine and not reIayed. From: quarky42 @.> Sent: 12 February 2022 02:59 To: helium/erlang-libp2p @.> Cc: TechyGeeksHome @.>; Comment @.> Subject: Re: [helium/erlang-libp2p] IP change not detected (#373) What does a port check on 44158 show? (Portchecker.co) Do you have an Asus router? On mine I had to change the NAT type from Symmetric to Fullcone. Do you have UPnP turned on? If so, turn it off. Have you rebooted your router since making the port forward? What does HeliumStatus.io site say? (I think they recently switched to a small fee model, but it's a great site...worth it)
On Fri, Feb 11, 2022, 7:40 PM TechyGeeksHome @.mailto:***@***.***> wrote: My hotspot has not updated its IP for 10 days now and its showing as reIayed either though the dashboard for the device is showing it as not reIayed. Earning virtuaIIy nothing. Needs sorting. — Reply to this email directly, view it on GitHub <#373 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALQQATEKX2JZ7P4XFRQV7FTU2XCCPANCNFSM5CIMUQUQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.mailto:***@***.***>
— Reply to this email directly, view it on GitHub<#373 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJ3EUUID4BZ2Q5CP72THSXTU2XEIBANCNFSM5CIMUQUQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.@.>>

if connected through wifi ,connect it through Ethernet cable, resolved issue in my case with this.

@quarky42
Copy link

That brings up a good point: ONLY connect via one or the other, not both! use the app to make the miner forget the wifi if you are going from wifi to Ethernet.

Having both connected can and will cause major problems, if that was the case.

And I agree with those before me: wired is way better.

@danielcharrua
Copy link

danielcharrua commented Feb 12, 2022

Anyone solve this issue by switching from WiFi to Ethernet? I don't think this is related

@techygeekshome
Copy link

I've had the miner on wifi and ethernet (one at a time obviously) port forwarded, dashboard for the miner says everything is absolutely fine, heliumgeek app has the wrong public IP for the hotspot and helium app shows it as relayed. I've rebooted the miner, the router etc. I've tried all configurations with my router, i.e. uPNP, static, symmetric, none NAT etc. I've worked in IT for 25 years so I know what I'm doing. It just seems that the helium p2p network is not updating to the correct IP. Last update on the heliumgeek app is now showing as 12 days ago!

@quarky42
Copy link

I've had the miner on wifi and ethernet (one at a time obviously) port forwarded, dashboard for the miner says everything is absolutely fine, heliumgeek app has the wrong public IP for the hotspot and helium app shows it as relayed. I've rebooted the miner, the router etc. I've tried all configurations with my router, i.e. uPNP, static, symmetric, none NAT etc. I've worked in IT for 25 years so I know what I'm doing. It just seems that the helium p2p network is not updating to the correct IP. Last update on the heliumgeek app is now showing as 12 days ago!

The Helium network will NOT update right away. It had take multiple days for the IP up update.

You've tried a bunch of things, but some of them were flat out wrong, others unnecessary, and I get the impression that you didn't wait near long enough.

"No NAT" = wrong.

"Symmetric NAT" = less than ideal if you have full cone available as a choice, use Fullcone instead.

"Port shows as open" - this is what you need and you need to LEAVE it alone for days, maybe even a week for the network to get it updated. Again, the Helium network is VERY slow to update.

"Wifi" you should have used wired if at all possible and avoided wifi. I myself have a couple miners on wifi because I have no other option but I have also done some work to make sure I have very clear wifi channels for them and they work fine. In a congested environment they would be screwed over, bad.

I have 25 plus years in networking, worked at an ISP, and most importantly I have 5 miners that areall working properly and I have had several that had problematic IP / P2P status that I have fixed. So either listen to someone that has gotten this to work or continue to spin your wheels and try things that you really should have simply avoided.

The worst problem I've fixed with my helium miners is caused by Comcast and their advanced security bullshit that is on the account. It literally has to be turned off in two places and one of them is hard to find. I f-ing hate Comcast and their crap equipment. Over the years (other networks) they've caused nothing but issues because their hardware is flat out garbage.

The longest one of my miners has taken to get an updated / correct IP is over a week... There is pretty much nothing you can do about that except maintain the correct configuration and wait it out.

@Chewy-13
Copy link

Unfortunately I think I'm having the same issue.
In messing with using UPNP or Port Forwarding, I noticed my WAN IP has changed, which I thought was odd - because for the longest time my DHCP lease on my public IP hadn't expired, or I kept getting the same one.
Doing a Peerping command from GetMNTD's test your hotspot channel shows a very old WAN IP that I haven't see in a while. (At work my DYNDNS doesn't work, so for a while I was accessing some things by IP.)
So something still isn't updating unfortunately. Logging into the diagnoser of the Bobcat, the correct Public IP is shown.

@ArgyDr
Copy link

ArgyDr commented Apr 17, 2022

Same issue here or even better...

My provider changes my IP all the time. Each time sensecap dashboard has the ability to change and match the public ip with the listen ip. So far, so good.
On the other hand, the sensecap app always keeps the old listen ip and the miner doesn't look relayed , but it does absolutely nothing!!!!! No beacons, no witnesses, no activity until the listen ip matches the public ip on the APP. Matching on the dashboard obviously is not enough!!!
So, I'm up 4 days and it only worked the first minutes because then my provider decided to change my public ip again! And every time, the app has the previous listen ip and I'm always without activity even though the dashboard is ok!!!!

I'm going nuts! Please suggest me a solution...

I forgot to mention that I have selected the SENSECAP API on the app settings...just in case this could help, but nothing seems to make it work properly

@quarky42
Copy link

Light hotspots update is a couple/few weeks away at this point. They aren't going to make a big change to the firmware just before releasing another even bigger change that has to be in place for 100% of the hotspots to work.

Their focus is going to be on that.

Also Light Hotspots is going to change how things work quite a bit, so give it some time for now.

Two ways, in the meantime, to keep your IP address from changing:

  1. ask your ISP for a static IP. If they ask why tell them that you have some home security equipment that you need to be able to login remotely and with the IP always changing it's a pain in the butt.

Or

  1. get a vpn service that supports static IPs and have your hotspot connect through that. Some routers support VPNs or they make little VPN boxes that you can connect the miner through. It will need to manually test your port forward to make sure it works.

@ArgyDr
Copy link

ArgyDr commented Apr 17, 2022

Thanks for the suggestions! I might try the second one since the static ip on my provider is not for free.
Let's hope the light hotspots solve this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests