[offtopic, not a technical issue] google charging for network traffic from US to EMEA #83

kurti500 · 2024-01-01T22:08:40Z

Hi all,
quick question, maybe I have made setup mistake or conditions have been changed: since a while I get network traffic from US to EMEA charged, not massively (below 1€/month), but as I am located in Germany there is a charge coming in every month and the idea was that the setup is for free ;-) :
Network Internet Data Transfer Out from Americas to EMEA
Network Internet Data Transfer Out from Americas to China

Anybody else in EMEA experiencing this? Any idea how this can be prevented? Also wondering where the transfer from US to China is coming from.

Thanks

turnah · 2024-01-01T22:31:04Z

I also had this, I ended up using cloudflare (moved my domain here) as a proxy to geoban regions and the problem has gone away. I couldn't find a way to do it elegantly within Google, even restricting by ip. I even had the regions blocked within the bitwarden gcloud setup too.

…

________________________________ From: kurti500 ***@***.***> Sent: Monday, January 1, 2024 10:08:54 pm To: dadatuputi/bitwarden_gcloud ***@***.***> Cc: Subscribed ***@***.***> Subject: [dadatuputi/bitwarden_gcloud] [offtopic, not a technical issue] google charging for network traffic from US to EMEA (Issue #83) Hi all, quick question, maybe I have made setup mistake or conditions have been changed: since a while I get network traffic from US to EMEA charged, not massively (below 1€/month), but as I am located in Germany there is a charge coming in every month and the idea was that the setup is for free ;-) : Network Internet Data Transfer Out from Americas to EMEA Network Internet Data Transfer Out from Americas to China Anybody else in EMEA experiencing this? Any idea how this can be prevented? Also wondering where the transfer from US to China is coming from. Thanks — Reply to this email directly, view it on GitHub<#83>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABBR63MKYQHYDI3E4KOB63TYMMXXJAVCNFSM6AAAAABBJKJSFSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA3DCNZZG42TMNQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

jtognazzi · 2024-01-02T11:03:10Z

Yeah, I also see some amount on the line
Network Internet Data Transfer Out from Americas to China

It is in the order of 0.01 USD

So I did not bother much (yet)

But I'm wondering what could be this traffic going out to China...
bots traffic probably.

dadatuputi · 2024-01-02T11:08:02Z

I've noticed the same, 2 months with $.02. Unfortunately, putting the blocking in gcloud will still not eliminate traffic from bot traffic; even the packets that are dropped by ipsec will be charged. Having it outside as some sort of web app firewall (ala Cloudflare @turnah) is the best approach to make it fool-proof.

This month I'll have some time to research some possible methods and write up. There is some prior work done here

killer23d · 2024-01-04T17:14:23Z

I have about CAD 0.02 every month, not a big deal but the CF approach seems to be a more elegant approach.

Looks like a websocket change for the next version that we need to be prepared for: dani-garcia/vaultwarden#4024

asardaes · 2024-01-04T17:35:58Z

FWIW, I already use the free tier of Cloudflare for DNS and proxying, and I also get a couple cents charged on some months, though not always. This doesn't include any blocking/firewall, but my understanding was that, through the Cloudflare network, the data center talking to the actual GCE VM would be in a region close to the VM.

However, do you see the "Network Internet Data Transfer" charges in the PDF invoices that Google sends? Because I've never seen anything so detailed there, mine just say "Fee for MONTH YEAR", and sometimes that's 0, sometimes not.

killer23d · 2024-01-09T17:06:15Z

FWIW, I already use the free tier of Cloudflare for DNS and proxying, and I also get a couple cents charged on some months, though not always. This doesn't include any blocking/firewall, but my understanding was that, through the Cloudflare network, the data center talking to the actual GCE VM would be in a region close to the VM.

However, do you see the "Network Internet Data Transfer" charges in the PDF invoices that Google sends? Because I've never seen anything so detailed there, mine just say "Fee for MONTH YEAR", and sometimes that's 0, sometimes not.

When I check the Cost Breakdown, I always have:

Network Internet Data Transfer Out from Americas to China - $0.02

The CF implementation would be great if the traffic can be proxied.

asardaes · 2024-01-09T17:40:35Z

Ah I found it now under "Cost table", same as yours. Since I already use the free CF proxy, I suppose that's not enough.

I did configure Countryblock as documented in the wiki, but I guess some requests always slip through, not sure if it can be 100% avoided.

dadatuputi · 2024-05-23T11:24:56Z

I didn't have the capacity I thought I would to look at this in January. There shouldn't be any charges for inbound data to Google. Any firewall rule added by countryblock will DROP packets, so there should be no outbound to IPs from the countries.

There will be IP subnets not in ipdeny.com's lists that Google will charge exit fees to.

Cloudflare might eliminate some traffic that's using your DNS to scan, but I believe most of the traffic is from systems that scan the entire IPv4 space (think Shodan), so DNS will have little to do with it.

The most airtight solution will require something like Cloudflare Zero Trust VPN (free I think for these purposes), or only whitelisting cloudflare IPs from Google.

dadatuputi · 2024-11-04T17:25:51Z

I made some changes to countryblock: dadatuputi/bwgc_countryblock#4. iptables changes relevant to this thread are:

The countryblock chain is now the very first rule; ssh connection attempts weren't following the rule
The countryblock rules block src and dst - before they only blocked src

New default iptables config now:

Chain INPUT (policy DROP)
target     prot opt source               destination
countryblock  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
...
Chain countryblock (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             match-set AU src,dst
DROP       all  --  anywhere             anywhere             match-set HK src,dst
DROP       all  --  anywhere             anywhere             match-set CN src,dst
RETURN     all  --  anywhere             anywhere

Let's see how this affects things, although I haven't had charges since this thread started.

killer23d · 2024-11-13T19:59:53Z

Just implemented the changes, let's see how this will turn up at the end of the month.

crespire · 2025-01-27T16:57:33Z

Just implemented the changes, let's see how this will turn up at the end of the month.

Any update on how these changes played out? I've noticed I have a lot of network being charged to South America, and am now considering moving my DNS over to CF over from my domain registrar's.

killer23d · 2025-01-27T17:15:21Z

Just implemented the changes, let's see how this will turn up at the end of the month.

Any update on how these changes played out? I've noticed I have a lot of network being charged to South America, and am now considering moving my DNS over to CF over from my domain registrar's.

Nothing to update as I recently shutdown this instance because I went back with BitWarden paid plan; like you said I have racked up quite a bit of network charges (unsure where), and it is cheaper for me to to pay BitWarden than Google.

The other option you can consider is moving the instance to Oracle, then it's completely free but you will need to manage the Linux VM and docker rather than Google's container OS.

dadatuputi · 2025-01-27T21:17:21Z

I had $0.14 in December and $0.91 in January.

Breaking down January, it's
1 $0.41 for 3.14 gibibyte to South America
2 $0.25 Networking Telemetry
3 $0.25 Vended Logs Storage

Regarding 1 - according to https://cloud.google.com/free/docs/free-cloud-features#compute we get 1GB outbound for free. So how is there 3.14 gibibyte to South America? I've enabled VPC Flows in my compute instance for further analysis but it's odd that there's so much egress.

A band-aid would be to include South American countries to the banned list.

The last 30 days has over 1 million connections to Brazil, here's a quick analysis of the last 10,000:

Almost entirely from 443, with a few from 22. Is it a scan? I don't see any banned IPs that are related.

Looking at my last hour of sshd logs, I see a bunch of scans for valid usernames:

Jan 27 19:04:38 bitwarden sshd[114014]: Invalid user crystal from 189.50.215.253 port 40452
Jan 27 19:04:38 bitwarden sshd[114014]: Received disconnect from 189.50.215.253 port 40452:11: Bye Bye [preauth]
Jan 27 19:04:38 bitwarden sshd[114014]: Disconnected from invalid user crystal 189.50.215.253 port 40452 [preauth]
Jan 27 19:06:14 bitwarden sshd[114472]: Invalid user ubuntu from 189.50.215.193 port 55828
Jan 27 19:06:15 bitwarden sshd[114472]: Received disconnect from 189.50.215.193 port 55828:11: Bye Bye [preauth]
Jan 27 19:06:15 bitwarden sshd[114472]: Disconnected from invalid user ubuntu 189.50.215.193 port 55828 [preauth]
Jan 27 19:07:47 bitwarden sshd[114930]: Received disconnect from 189.50.215.253 port 48398:11: Bye Bye [preauth]
Jan 27 19:07:47 bitwarden sshd[114930]: Disconnected from authenticating user root 189.50.215.253 port 48398 [preauth]
Jan 27 19:09:22 bitwarden sshd[115374]: Invalid user qwert from 189.50.215.253 port 46704
Jan 27 19:09:22 bitwarden sshd[115374]: Received disconnect from 189.50.215.253 port 46704:11: Bye Bye [preauth]
Jan 27 19:09:22 bitwarden sshd[115374]: Disconnected from invalid user qwert 189.50.215.253 port 46704 [preauth]
Jan 27 19:11:01 bitwarden sshd[115877]: Invalid user deploy from 189.50.215.253 port 57988
Jan 27 19:11:01 bitwarden sshd[115877]: Received disconnect from 189.50.215.253 port 57988:11: Bye Bye [preauth]
Jan 27 19:11:01 bitwarden sshd[115877]: Disconnected from invalid user deploy 189.50.215.253 port 57988 [preauth]
Jan 27 19:12:41 bitwarden sshd[116343]: Invalid user server from 189.50.215.155 port 47542
Jan 27 19:12:41 bitwarden sshd[116343]: Received disconnect from 189.50.215.155 port 47542:11: Bye Bye [preauth]
Jan 27 19:12:41 bitwarden sshd[116343]: Disconnected from invalid user server 189.50.215.155 port 47542 [preauth]
Jan 27 19:12:44 bitwarden sshd[116345]: Invalid user dist from 59.92.227.21 port 55980
Jan 27 19:12:44 bitwarden sshd[116345]: Received disconnect from 59.92.227.21 port 55980:11: Bye Bye [preauth]
Jan 27 19:12:44 bitwarden sshd[116345]: Disconnected from invalid user dist 59.92.227.21 port 55980 [preauth]
Jan 27 19:14:14 bitwarden sshd[116793]: Invalid user anwar from 189.50.215.253 port 55666
Jan 27 19:14:14 bitwarden sshd[116793]: Received disconnect from 189.50.215.253 port 55666:11: Bye Bye [preauth]
Jan 27 19:14:14 bitwarden sshd[116793]: Disconnected from invalid user anwar 189.50.215.253 port 55666 [preauth]
Jan 27 19:15:26 bitwarden sshd[117129]: Received disconnect from 59.92.227.21 port 41937:11: Bye Bye [preauth]
Jan 27 19:15:26 bitwarden sshd[117129]: Disconnected from authenticating user root 59.92.227.21 port 41937 [preauth]
Jan 27 19:15:48 bitwarden sshd[117249]: Invalid user stephen from 189.50.215.253 port 34992
Jan 27 19:15:49 bitwarden sshd[117249]: Received disconnect from 189.50.215.253 port 34992:11: Bye Bye [preauth]
Jan 27 19:15:49 bitwarden sshd[117249]: Disconnected from invalid user stephen 189.50.215.253 port 34992 [preauth]
Jan 27 19:18:01 bitwarden sshd[117889]: Received disconnect from 59.92.227.21 port 53197:11: Bye Bye [preauth]
Jan 27 19:18:01 bitwarden sshd[117889]: Disconnected from authenticating user root 59.92.227.21 port 53197 [preauth]
Jan 27 19:20:32 bitwarden sshd[118608]: Invalid user applmgr from 59.92.227.21 port 36227
Jan 27 19:20:32 bitwarden sshd[118608]: Received disconnect from 59.92.227.21 port 36227:11: Bye Bye [preauth]
Jan 27 19:20:32 bitwarden sshd[118608]: Disconnected from invalid user applmgr 59.92.227.21 port 36227 [preauth]
Jan 27 19:22:58 bitwarden sshd[119313]: Invalid user hero from 59.92.227.21 port 46106
Jan 27 19:22:58 bitwarden sshd[119313]: Received disconnect from 59.92.227.21 port 46106:11: Bye Bye [preauth]
Jan 27 19:22:58 bitwarden sshd[119313]: Disconnected from invalid user hero 59.92.227.21 port 46106 [preauth]
Jan 27 19:25:40 bitwarden sshd[120090]: Invalid user wesley from 59.92.227.21 port 58938
Jan 27 19:25:40 bitwarden sshd[120090]: Received disconnect from 59.92.227.21 port 58938:11: Bye Bye [preauth]
Jan 27 19:25:40 bitwarden sshd[120090]: Disconnected from invalid user wesley 59.92.227.21 port 58938 [preauth]
Jan 27 19:28:05 bitwarden sshd[120781]: Invalid user zhouhao from 59.92.227.21 port 41348
Jan 27 19:28:06 bitwarden sshd[120781]: Received disconnect from 59.92.227.21 port 41348:11: Bye Bye [preauth]
Jan 27 19:28:06 bitwarden sshd[120781]: Disconnected from invalid user zhouhao 59.92.227.21 port 41348 [preauth]
Jan 27 19:30:38 bitwarden sshd[121498]: Invalid user zz from 59.92.227.21 port 52791
Jan 27 19:30:38 bitwarden sshd[121498]: Received disconnect from 59.92.227.21 port 52791:11: Bye Bye [preauth]
Jan 27 19:30:38 bitwarden sshd[121498]: Disconnected from invalid user zz 59.92.227.21 port 52791 [preauth]
Jan 27 19:31:15 bitwarden sshd[121688]: Connection closed by authenticating user root 143.198.96.196 port 55958 [preauth]
Jan 27 19:31:35 bitwarden sshd[121790]: Invalid user ftp from 194.0.234.38 port 54820
Jan 27 19:31:36 bitwarden sshd[121790]: Connection closed by invalid user ftp 194.0.234.38 port 54820 [preauth]
Jan 27 19:33:05 bitwarden sshd[122212]: Invalid user jenkins from 59.92.227.21 port 35341

I think adding a sshd jail to fail2ban is a good idea. The 189.50.215.253 is one of the Brazil connections.

IP-wide scanning is so common that any server with an open port 22 and 443 are going to get hammered. So much so that 1GB egress won't cover just the scanning that happens.

Caddy logs don't show anything unusual, so I believe it's simply connection attempts to 443 that get shut down before anyone gets to the Bitwarden login page - as my domain is pretty obscure, it would take some clever dns bruteforcing to discover what it is.

Regarding 2 above, I had logging turned on for my vm instance and with all the scanning traffic it was filling up, so I turned if off, which should take care of #2.

Edit: I also disabled the logging API here entirely: https://console.cloud.google.com/apis/library/logging.googleapis.com?inv=1&invt=Abn_xQ&project=api-project-1048835024069

To reduce/eliminate network telemetry charges for 2 above, I turned off logging for all the subnets with this script in the Cloud Shell:

gcloud compute networks subnets list --format="csv(name,region)" | tail -n +2 | while IFS=, read name region; do
    echo "Disabling flow logs for subnet $name in $region"
    gcloud compute networks subnets update $name \
        --region=$region \
        --no-enable-flow-logs
done

I also found this setting, set to Premium - I don't know what the default is but I'm changing it to Standard:

I am starting to feel like Google Cloud isn't a great home for this, and I looked into @killer23d's comment about Oracle - they have a fantastic free tier. Under their Always Free offering:

10TB egress!
200GB storage
1/8 OCPU AMD or 4 OCPU ARM

I think I'll spin up an Oracle version of this and try it out on ARM.

Regarding the future of this project, I think I'll make it more cloud-agnostic, it mostly is really. This docker-compose will work on any docker host.

dadatuputi · 2025-01-27T21:38:59Z

Adding Brazil to my .env:

Testing a fail2ban config now that will ban sshd attempts.

asardaes · 2025-01-27T22:43:46Z

I was also looking at network service tier. From what I can tell, the standard tier's free egress is relatively recent (since 2023?), and I think premium has always been the default. I'll give standard a try as well, see if I notice any significant latency being outside the US.

killer23d · 2025-01-28T02:59:57Z

I am starting to feel like Google Cloud isn't a great home for this, and I looked into @killer23d's comment about Oracle - they have a fantastic free tier. Under their Always Free offering:

10TB egress!

200GB storage

1/8 OCPU AMD or 4 OCPU ARM

I think I'll spin up an Oracle version of this and try it out on ARM.

Regarding the future of this project, I think I'll make it more cloud-agnostic, it mostly is really. This docker-compose will work on any docker host.

Regarding Oracle (OCI), I have been using them for a while and there are a few minor annoyances that you need to know before signing up. First, they need a credit card verification even for the free tier to ensure you do not have duplicate accounts. They are very strict on this. Second, the A1.Flex VM are scarce and almost never available to free tier users because of "high demand". The ONLY way to get this is to upgrade the account to Pay As You Go, once approved you will have immediate access. The upgrade approval process can take hours to days depending on your risk level. My first account took 7 days, requiring some back and forth with a sales rep to provide access. My second account (using my wife's info) took 3 hours. OCI console lacks some features, mainly web SSH and the performance of the network are not as quick as GCP or AWS. Once all these are over, the A1 performance trumps the E1 Micro in every way.

I just checked my GCP console, I have about $0.6 for egress to South America as well.

dadatuputi · 2025-01-28T14:23:01Z

Second, the A1.Flex VM are scarce and almost never available to free tier users because of "high demand".

I'm running into this right now. I like the safety of being in the "Free Tier" account, meaning you are limited to only the free products, but none of the domains in my midwest US region had availability.

dadatuputi · 2025-01-29T12:43:16Z

I'm documenting the Oracle process here: https://github.com/dadatuputi/bitwarden_gcloud/wiki/Oracle-Cloud-Free-Tier

I'm currently at the stage where I'm trying to 'brute force' my way into a vm, but not very optimistic about it. I'll give it a couple days then see how it is to upgrade my account.

@killer23d After you upgraded your account, how hard was it for you to stay within the free tier limits?

Edit: Oracle PAYG is no different than Google's free tier, so that might just be the easiest way forward, if it's not a pita to upgrade for everyone like it was for you.

killer23d · 2025-01-29T17:53:32Z

I'm documenting the Oracle process here: https://github.com/dadatuputi/bitwarden_gcloud/wiki/Oracle-Cloud-Free-Tier

I'm currently at the stage where I'm trying to 'brute force' my way into a vm, but not very optimistic about it. I'll give it a couple days then see how it is to upgrade my account.

@killer23d After you upgraded your account, how hard was it for you to stay within the free tier limits?

Edit: Oracle PAYG is no different than Google's free tier, so that might just be the easiest way forward, if it's not a pita to upgrade for everyone like it was for you.

I have not yet exceeded the free tier, with OCI free tier, you can have up to A1 Flex with 24GB RAM that can be split with 4 VMs. 20GB storage and 10TB of data egress is plenty. I did not use OCI for VaultWarden, I am using it for OpenVPN and Tailscale, the A1 VMs are great for this. I just don't like I have to maintain the host OS that's all. I am not too proficient with Linux.

asardaes · 2025-01-29T19:16:35Z

Have you seen Oracle's reclamation policy for idle compute instances? I wonder if the whole setup would use so little resources that it would be counted as idle.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[offtopic, not a technical issue] google charging for network traffic from US to EMEA #83

[offtopic, not a technical issue] google charging for network traffic from US to EMEA #83

kurti500 commented Jan 1, 2024

turnah commented Jan 1, 2024 via email

jtognazzi commented Jan 2, 2024

dadatuputi commented Jan 2, 2024 •

edited

Loading

killer23d commented Jan 4, 2024 •

edited

Loading

asardaes commented Jan 4, 2024

killer23d commented Jan 9, 2024

asardaes commented Jan 9, 2024

dadatuputi commented May 23, 2024

dadatuputi commented Nov 4, 2024 •

edited

Loading

killer23d commented Nov 13, 2024

crespire commented Jan 27, 2025

killer23d commented Jan 27, 2025

dadatuputi commented Jan 27, 2025 •

edited

Loading

dadatuputi commented Jan 27, 2025

asardaes commented Jan 27, 2025

killer23d commented Jan 28, 2025

dadatuputi commented Jan 28, 2025

dadatuputi commented Jan 29, 2025 •

edited

Loading

killer23d commented Jan 29, 2025

asardaes commented Jan 29, 2025 •

edited

Loading

[offtopic, not a technical issue] google charging for network traffic from US to EMEA #83

[offtopic, not a technical issue] google charging for network traffic from US to EMEA #83

Comments

kurti500 commented Jan 1, 2024

turnah commented Jan 1, 2024 via email

jtognazzi commented Jan 2, 2024

dadatuputi commented Jan 2, 2024 • edited Loading

killer23d commented Jan 4, 2024 • edited Loading

asardaes commented Jan 4, 2024

killer23d commented Jan 9, 2024

asardaes commented Jan 9, 2024

dadatuputi commented May 23, 2024

dadatuputi commented Nov 4, 2024 • edited Loading

killer23d commented Nov 13, 2024

crespire commented Jan 27, 2025

killer23d commented Jan 27, 2025

dadatuputi commented Jan 27, 2025 • edited Loading

dadatuputi commented Jan 27, 2025

asardaes commented Jan 27, 2025

killer23d commented Jan 28, 2025

dadatuputi commented Jan 28, 2025

dadatuputi commented Jan 29, 2025 • edited Loading

killer23d commented Jan 29, 2025

asardaes commented Jan 29, 2025 • edited Loading

dadatuputi commented Jan 2, 2024 •

edited

Loading

killer23d commented Jan 4, 2024 •

edited

Loading

dadatuputi commented Nov 4, 2024 •

edited

Loading

dadatuputi commented Jan 27, 2025 •

edited

Loading

dadatuputi commented Jan 29, 2025 •

edited

Loading

asardaes commented Jan 29, 2025 •

edited

Loading