tfstate file corruption when user is not allowed to access resource

[robo-cap]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version and Provider Version

Terraform version: 4.81.0
Provider version: 4.81.0

Affected Resource(s)

affected_resources = terraform

Terraform Configuration Files

Initial deployment using admin's user_ocid:

provider "oci" {
  tenancy_ocid     = ".."
  user_ocid        = ".."
  fingerprint      = ".."
  private_key_path = ".."
  region           = ".."
}

resource "oci_core_vcn" "admin" {
    #Required
    compartment_id = "<admin_compartment>"
    cidr_blocks = ["10.0.0.0/16"]
    display_name = "admin"
    dns_label = "admin"
}

Secondary deployment using John's user_ocid. John is not allowed to list resources in admin's compartment, he is only allowed to deploy resources within his own comparment.

provider "oci" {
  tenancy_ocid     = ".."
  user_ocid        = ".."
  fingerprint      = ".."
  private_key_path = ".."
  region           = ".."
}

resource "oci_core_vcn" "admin" {
    #Required
    compartment_id = "<admin_compartment>"
    cidr_blocks = ["10.0.0.0/16"]
    display_name = "admin"
    dns_label = "admin"
}

resource "oci_core_vcn" "test" {
    #Required
    compartment_id = "<john_comparment>"
    cidr_blocks = ["10.0.0.0/16"]
    display_name = "test"
    dns_label = "test"
}

Expected Behavior

tfstate file should preserve existing resources (VCN created by admin) and include new created resources (VCN created by John).

Actual Behavior

When John is trying to create test VCN, admin's VCN gets removed from tfstate file as John is not allowed to list existing resources in admin's compartment. Service API return Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. for resource deployed in admin's compartment.

Steps to Reproduce

1. Using admin's user_ocid create admin VCN in admin compartment.
2. Check tfstate file.
3. Using John's user_ocid try to add test VCN in John's compartment.
4. Check tfstate file; admin's VCN is removed from tfstate file and only John's VCN is present.

[or-shachar]

Bump.

Note that this issue makes this provider super fragile when there are regional/global issues with Infrastructure Identity and Access Management services (as happened just recently), not only when user-supplied bad credentials.

The current behavior removes the created objects from the state, and then we must re-import them manually.
At scale with many DCs - this becomes a huge pain.

[rramge]

This is normal behaviour within Terraform and a error in the personal provisioning concept. John is not supposed to write the admin's state; depending on the secrets contained within, he should not even have read access. Those are different people / roles, both technical as well as organizational, and need to stay separated on Terraform level as well. In Terraform Enteprise / Cloud terms, those are workspaces and they are isolated from each other and they MUST stay isolated from each other. This isoliation can be on organizational level (like different company departements, e.g. network and database teams), it can be on a legal level (e.g. HR or financial infrastructure vs. hosting engineers), or security levels, or simply the most common use case: Global cloud administration vs. individual Value Streams who can only access their own small partition of the entire cloud and Infrastructure-as-Code partition, but are not supposed to interfere which each other.

Instead, the admin needs John to grant read-only access to the admin tfstate, and John needs to access it read-only via the terraform_remote_state datasources. Depending on the secret management (and other sensitive data in the admin statefile which John may not be suppossed to see), it often makes sense to deploy proxy states, which means that John cannot access the original admin statefile, its bucket, compartment or even tenancy in any way, but the only an additional state which serves as a proxy and only contains the specific John-relevant data and nothing else. This can be achieved easily even with Terraform Open Source. Documentation can be found here: https://www.terraform.io/language/state/remote-state-data

Long story short: This is not a provider issue, but intended Terraform behaviour.

[or-shachar]

@rramge We recently experience many false and inconsistent 401 errors from OCI (unlike the author of this issue - we used the same exact credentials and on 3/10 runs - we got 401).

I think that in our case this is an issue with the provider. Given that the OCI resources are only part of a big terraform run, and that we couldn't just exclude OCI resources while the service was down - it kept corrupting our remote state and forced us to manually restore it.

Do you think we should open a different issue for that ? (same issue in Pulumi)

[uristernik]

Resurfacing this as we are getting 503 due to the service distruption.
We are again seeing things deleted from state when running plan while the provider is unavailable

[ravinitp]

Thank you for reporting the issue. We observed the affected resources are not provided in the description or it's incorrect. We request you to add it in issue description as mentioned in below format.
Example: affected_resources = oci_core_instance , oci_core_instances

If it's not related to any particular resource then mention affected resource as terraform.
Example: affected_resources = terraform

As this works through automation, request you to follow exact syntax.