See GLOSSARY.md.
- Common issues
- Seed Project missing APIs - The Seed Project is missing required APIs.
- Seed Service Account missing roles - The Seed Service Account has insufficient permissions.
When the Project Factory is used with a misconfigured Seed Project, it will partially generate a new Target Project, fail, and enter a state where it can no longer generate Target Projects.
Error message:
Error: Error refreshing state: 1 error(s) occurred:
* module.project-factory.data.google_compute_default_service_account.default: 1
error(s) occurred:
* module.project-factory.data.google_compute_default_service_account.default:
data.google_compute_default_service_account.default: Error reading GCE service
account not found: googleapi: Error 403: Project 449949713944 is not found and
cannot be used for API calls. If it is recently created, enable Compute Engine
API by visiting
https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=449949713944
then retry. If you enabled this API recently, wait a few minutes for the
action to propagate to our systems and retry., accessNotConfigured
Cause:
The Project Factory has generated a new Target Project but could not enable the
compute.googleapis.com API. This causes Terraform to get jammed, with the
following causal chain:
terraform plantries to query the default GCE service account.- The query fails because the
compute.googleapis.comAPI is not enabled on the Target Project. - The
compute.googleapis.comAPI is not enabled on the Target Project because it does not have an associated billing account. - The Target Project does not have an associated billing account for one of
the following causes:
- The Seed Project does not have the
cloudbilling.googleapis.comAPI enabled, so Terraform cannot enable billing on the Target Project. - The Seed Service Account does not have the
roles/billing.userrole on the associated billing account, and cannot link the Target Project with the billing account.
- The Seed Project does not have the
This issue is confusing because the error indicates that the
compute.googleapis.com API is disabled on the Target Project, but the absence
of the Google Compute Engine API is a symptom of an issue with configuring
billing, rather than the cause of the issue itself.
Solution:
In order to recover the Terraform configuration, the required APIs need to be enabled on the Seed Project and Target Project.
- Enable billing on the Seed Project:
- Enable the
cloudbilling.googleapis.comAPI on the Seed Project:# Requires `roles/serviceusage.admin` on $SEED_PROJECT gcloud services enable cloudbilling.googleapis.com \ --project "$SEED_PROJECT" - Associate a billing account with the Seed Project:
# Requires `roles/billing.projectManager` on $SEED_PROJECT and # `roles/billing.user` on the billing account gcloud alpha billing projects link "$SEED_PROJECT" \ --billing-account=[billing-account]
- Enable the
- Enable
compute.googleapis.comon the Target Project:This should be run in the context of the Seed Service Account. You can add the Seed Service Account to your list of authentication credentials by issuing the following command and importing the Seed Service Account key:# Requires `roles/serviceusage.admin` on $TARGET_PROJECT gcloud services enable compute.googleapis.com --project $TARGET_PROJECT
gcloud auth activate-service-account \ --key-file=path-to-service-account-credentials.json
Error messages:
Error: Error applying plan:
1 error(s) occurred:
* module.project-factory.gsuite_group.group: 1 error(s) occurred:
* gsuite_group.group: Error creating group: googleapi: Error 403: Not Authorized to access this resource/api, forbidden
Error: Error applying plan:
1 error(s) occurred:
* module.project-factory.gsuite_group_member.service_account_sa_group_member: 1
error(s) occurred:
* gsuite_group_member.service_account_sa_group_member: Error creating group
member: Post
https://www.googleapis.com/admin/directory/v1/groups/thebo-pf-service-accounts/members?alt=json:
oauth2: cannot fetch token: 401 Unauthorized
Response: {
"error" : "unauthorized_client",
"error_description" : "Client is unauthorized to retrieve access tokens using this method."
}
Cause:
By default service accounts cannot manage G Suite resources, because the G Suite directory API requires a user account with admin rights. A service account can impersonate a user account with admin rights to interact with the directory API, but the service account has to be granted domain wide delegation rights to impersonate users, and must be granted OAuth scopes for the functionality it needs to interact with.
If you encounter a 403: Not Authorized/forbidden, likely it means you
provided a service account instead of a user account to the
impersonated_user_email variable in the gsuite provider.
If you encounter a 401: Unauthorized, likely it is because domain wide
delegation is not granted to the Seed Service Account, and it will fail to
obtain the access token needed to interact with the Directory API and fail.
See the README G Suite documentation for more information.
Solution:
Provide a user account (not a service account) to the gsuite provider via
impersonated_user_email.
Enable domain wide delegation on the Seed Service Account with the following scopes:
- https://www.googleapis.com/auth/admin.directory.group
- https://www.googleapis.com/auth/admin.directory.group.member
Error messages:
When running terraform validate
Error: Failed to instantiate provider "gsuite" to obtain schema: Incompatible
API version with plugin. Plugin version: 4, Client versions: [5]
Cause:
This is likely caused by an older version of the gsuite provider, for example the 0.1.10 version which is compatible with Terraform 0.11. Terraform 0.12 produces this error message because of the new plugin API version.
Solution:
Replace the old plugin with a version compatible with 0.12 from the gsuite provider releases. For example:
cd ~/.terraform.d/plugins/darwin_amd64
curl -LO https://github.com/DeviaVir/terraform-provider-gsuite/releases/download/v0.1.22/terraform-provider-gsuite_0.1.22_darwin_amd64.zip
unzip terraform-provider-gsuite_0.1.22_darwin_amd64.zipThe Project Factory requires the following services to be enabled on the Seed Project. If these APIs are not enabled, the Project Factory may fail to generate resources, or generate incomplete resources.
A canonical list of required APIs is available in the README.
- cloudresourcemanager.googleapis.com
- cloudbilling.googleapis.com
- iam.googleapis.com
- appengine.googleapis.com (when managing an App Engine instance on the Target Project)
- admin.googleapis.com (when managing G Suite groups and group membership)
Error message:
Error: Error refreshing state: 1 error(s) occurred:
* module.project-factory.data.google_organization.org: 1 error(s) occurred:
* module.project-factory.data.google_organization.org:
data.google_organization.org: Error reading Organization Not Found :
[organization-id]: googleapi: Error 403: Cloud Resource Manager API has not
been used in project [seed-project-number] before or it is disabled. Enable it
by visiting
https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=[seed-project-number]
then retry. If you enabled this API recently, wait a few minutes for the
action to propagate to our systems and retry., accessNotConfigured
Cause:
The Seed Project does not have the cloudresourcemanager.googleapis.com API
enabled. This prevents the Project Factory from looking up the GCP organization.
Solution:
- Option 1: enable the required API with
gcloud:# Requires `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT gcloud services enable cloudresourcemanager.googleapis.com --project "$SEED_PROJECT" - Option 2: create a new Seed Service Account and enable the required APIs:
# requires `roles/resourcemanager.organizationAdmin` on the organization and # `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT ./helpers/setup-sa.sh [organization id] "$SEED_PROJECT"
Error message:
Error: Error applying plan:
1 error(s) occurred:
* module.project-factory.google_project.project: 1 error(s) occurred:
* google_project.project: Error setting billing account "[billing-account]" for
project "projects/[target-project-id]": googleapi: Error 403: Cloud Billing
API has not been used in project [seed-project-number] before or it is
disabled. Enable it by visiting
https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=[seed-project-number]
then retry. If you enabled this API recently, wait a few minutes for the
action to propagate to our systems and retry., accessNotConfigured
Cause:
The Seed Project does not have the cloudbilling.googleapis.com API enabled.
This prevents the Project Factory from enabling APIs that may result in billing
charges, such as the Google Compute Engine API.
Solution:
- Option 1: enable the required API with
gcloud:# Requires `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT gcloud services enable cloudbilling.googleapis.com \ --project "$SEED_PROJECT" - Option 2: create a new service account and enable the required APIs:
# requires `roles/resourcemanager.organizationAdmin` on the organization # and `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT ./helpers/setup-sa.sh [organization id] "$SEED_PROJECT"
Notes:
This error will occur once when applying a Terraform plan when the Seed Project does not have the Cloud Billing API enabled. On subsequent Terraform runs, Terraform will generate an error indicating that the Compute Engine API is not enabled on the Target Project. Watch out for this!
Error message:
Error: Error applying plan:
3 error(s) occurred:
* module.project-factory.null_resource.delete_default_compute_service_account:
Error running command
'./.terraform/modules/9a383cbc6c7042259f4a632606c52cf4/scripts/delete-service-account.sh
[target-project-id] ./credentials.json
[email protected]': exit status 1. Output:
API [iam.googleapis.com] not enabled on project [[seed-project-number]].
Would you like to enable and retry (this will take a few minutes)? (y/N)?
ERROR: (gcloud.iam.service-accounts.list) User [[service-account]] does not have
permission to access project [[target-project-id]] (or it may not exist):
Identity and Access Management (IAM) API has not been used in project
[seed-project-number] before or it is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=[seed-project-number]
then retry. If you enabled this API recently, wait a few minutes for the action
to propagate to our systems and retry.
- '@type': type.googleapis.com/google.rpc.Help
links:
- description: Google developers console API activation
url: https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=[seed-project-number]
* google_service_account.extra_service_account: 1 error(s) occurred:
* google_service_account.extra_service_account: Error creating service account:
googleapi: Error 403: Identity and Access Management (IAM) API has not been
used in project [seed-project-number] before or it is disabled. Enable it by
visiting
https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=[seed-project-number]
then retry. If you enabled this API recently, wait a few minutes for the
action to propagate to our systems and retry., accessNotConfigured
* module.project-factory.google_service_account.default_service_account: 1
error(s) occurred:
* google_service_account.default_service_account: Error creating service
account: googleapi: Error 403: Identity and Access Management (IAM) API has
not been used in project [seed-project-number] before or it is disabled.
Enable it by visiting
https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=[seed-project-number]
then retry. If you enabled this API recently, wait a few minutes for the
action to propagate to our systems and retry., accessNotConfigured
Cause:
The Seed Project does not have the iam.googleapis.com API enabled. This
prevents the Project Factory from deleting the default GCE service account and
assigning IAM roles to groups, service accounts, etc.
Solution:
- Option 1: enable the required API with
gcloud:# Requires `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT gcloud services enable iam.googleapis.com --project "$SEED_PROJECT" - Option 2: create a new Seed Service Account and enable the required APIs:
# requires `roles/resourcemanager.organizationAdmin` on the organization and # `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT ./helpers/setup-sa.sh [organization id] "$SEED_PROJECT"
Error message:
$ gcloud app describe --project [target-project-id] --format=json
API [appengine.googleapis.com] not enabled on project [[seed-project-number]].
Would you like to enable and retry (this will take a few minutes)?
(y/N)?
ERROR: (gcloud.app.describe) User [[service-account]] does not have permission
to access app [[target-project-id]] (or it may not exist): App Engine Admin API
has not been used in project [seed-project-number] before or it is disabled.
Enable it by visiting
https://console.developers.google.com/apis/api/appengine.googleapis.com/overview?project=[seed-project-number]
then retry. If you enabled this API recently, wait a few minutes for the action
to propagate to our systems and retry.
- '@type': type.googleapis.com/google.rpc.Help
links:
- description: Google developers console API activation
url: https://console.developers.google.com/apis/api/appengine.googleapis.com/overview?project=[seed-project-number]
Cause:
The App Engine API is not enabled on the Seed Project, which prevents creation of an App Engine instance on the Target Project.
Solution:
- Option 1: enable the required API with
gcloud:# Requires `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT gcloud services enable appengine.googleapis.com --project "$SEED_PROJECT" - Option 2: create a new Seed Service Account and enable the required APIs:
# requires `roles/resourcemanager.organizationAdmin` on the organization and # `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT ./helpers/setup-sa.sh [organization id] "$SEED_PROJECT"
Notes:
The absence of the appengine.googleapis.com API will not cause Terraform to
fail, but any interactions with the App Engine instance will fail. Once this API
is activated, it might take a few minutes for Terraform to detect that the App
Engine instance is not present and then create it.
Error message:
Error: Error applying plan:
1 error(s) occurred:
* module.project-factory.gsuite_group_member.service_account_sa_group_member: 1
error(s) occurred:
* gsuite_group_member.service_account_sa_group_member: Error creating group
member: googleapi: Error 403: Access Not Configured. Admin Directory API has
not been used in project 454152376537 before or it is disabled. Enable it by
visiting
https://console.developers.google.com/apis/api/admin.googleapis.com/overview?project=454152376537
then retry. If you enabled this API recently, wait a few minutes for the
action to propagate to our systems and retry., accessNotConfigured
Cause:
The Directory Admin API is not enabled on the Seed Project, which prevents management of G Suite resources.
See the README G Suite documentation for more information.
Solution:
- Option 1: enable the required API with
gcloud:# Requires `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT gcloud services enable admin.googleapis.com --project "$SEED_PROJECT" - Option 2: create a new Seed Service Account and enable the required APIs:
# requires `roles/resourcemanager.organizationAdmin` on the organization and # `roles/serviceusage.serviceUsageAdmin` on $SEED_PROJECT ./helpers/setup-sa.sh [organization id] "$SEED_PROJECT"
The Seed Service Account must have the following roles in order to fully create a Project Factory.
A canonical list of required roles is available in the README
- Organizational roles
- roles/resourcemanager.organizationViewer
- Required for looking up the domain name associated with the GCP organization ID.
- roles/resourcemanager.projectCreator
- Required for creating GCP projects within the organization.
- roles/compute.xpnAdmin (when using a shared VPC) - Required for associating the target project with the host VPC.
- roles/compute.networkAdmin (when using a shared VPC) - Required for managing shared VPC subnetwork IAM policies.
- roles/resourcemanager.organizationViewer
- Shared VPC project roles (when using a shared VPC)
- roles/resourcemanager.projectIamAdmin - Required for managing shared VPC project IAM policies.
- roles/browser - Required for enumerating shared VPC resources.
- Billing account roles
- roles/billing.user - Required for associating the billing account with a project.
Error message:
* module.project-factory.data.google_organization.org: 1 error(s) occurred:
* module.project-factory.data.google_organization.org:
data.google_organization.org: Error reading Organization Not Found :
[organization-id]: googleapi: Error 403: The caller does not have permission,
forbidden
Cause:
The Seed Service Account does not have
roles/resourcemanager.organizationViewer on the active organization. The
organizationViewer role grants the Seed Service Account the ability to look up
the GCP organization by the organization ID and fetch the associated domain
name.
Solution:
Use helpers/setup-sa.sh to create a new Seed Service Account with the necessary
roles.
OR
gcloud organizations add-iam-policy-binding \
"${ORG_ID}" \
--member="serviceAccount:${SA_ID}" \
--role="roles/resourcemanager.organizationViewer" \
--user-output-enabled false
Error message:
1 error(s) occurred:
* module.project-factory.google_project.project: 1 error(s) occurred:
* google_project.project: error creating project [target-project-id]
([target-project-name]): googleapi: Error 403: User is not authorized.,
forbidden. If you received a 403 error, make sure you have the
`roles/resourcemanager.projectCreator` permission
Cause:
The Seed Service Account does not have roles/resourcemanager.projectCreator on
the active organization. The projectCreator role allows the Seed Service Account
to generate new projects within the GCP organization.
Solution:
Use helpers/setup-sa.sh to create a new Seed Service Account with the necessary
roles.
OR
gcloud organizations add-iam-policy-binding \
"${ORG_ID}" \
--member="serviceAccount:${SA_ID}" \
--role="roles/resourcemanager.projectCreator" \
--user-output-enabled false
Error message:
Error: Error applying plan:
7 error(s) occurred:
* module.project-factory.google_compute_shared_vpc_service_project.shared_vpc_attachment:
1 error(s) occurred:
* google_compute_shared_vpc_service_project.shared_vpc_attachment: googleapi:
Error 403: Required 'resourcemanager.projects.get' permission for
'projects/[shared-vpc-project-id]', forbidden
* module.project-factory.google_project_iam_member.gke_host_agent: 1 error(s)
occurred:
* google_project_iam_member.gke_host_agent: Error retrieving IAM policy for
project "[shared-vpc-project-id]": googleapi: Error 403: The caller does not
have permission, forbidden
* module.project-factory.google_project_iam_member.controlling_group_vpc_membership[3]:
1 error(s) occurred:
* google_project_iam_member.controlling_group_vpc_membership.3: Error retrieving
IAM policy for project "[shared-vpc-project-id]": googleapi: Error 403: The
caller does not have permission, forbidden
* google_project_iam_member.additive_shared_vpc_role: 1 error(s) occurred:
* google_project_iam_member.additive_shared_vpc_role: Error retrieving IAM
policy for project "[shared-vpc-project-id]": googleapi: Error 403: The caller
does not have permission, forbidden
* module.project-factory.google_project_iam_member.controlling_group_vpc_membership[1]:
1 error(s) occurred:
* google_project_iam_member.controlling_group_vpc_membership.1: Error retrieving
IAM policy for project "[shared-vpc-project-id]": googleapi: Error 403: The
caller does not have permission, forbidden
* module.project-factory.google_project_iam_member.controlling_group_vpc_membership[0]:
1 error(s) occurred:
* google_project_iam_member.controlling_group_vpc_membership.0: Error retrieving
IAM policy for project "[shared-vpc-project-id]": googleapi: Error 403: The
caller does not have permission, forbidden
* module.project-factory.google_project_iam_member.controlling_group_vpc_membership[2]:
1 error(s) occurred:
* google_project_iam_member.controlling_group_vpc_membership.2: Error retrieving
IAM policy for project "[shared-vpc-project-id]": googleapi: Error 403: The
caller does not have permission, forbidden
Cause:
The service account is missing one of the following roles:
- Organization
roles/compute.xpnAdminroles/compute.networkAdminroles/iam.serviceAccountAdmin
- Shared VPC
roles/browserroles/resourcemanager.projectIamAdmin
These roles are required for associating the Target Project with the host VPC and managing access to the host VPC network resources.
Solution:
Use helpers/setup-sa.sh to create a new Seed Service Account with the
necessary roles.
OR
# Requires `roles/resourcemanager.organizationAdmin` on $ORG_ID
gcloud organizations add-iam-policy-binding \
"${ORG_ID}" \
--member="serviceAccount:${SA_ID}" \
--role="roles/compute.xpnAdmin" \
--user-output-enabled false
gcloud organizations add-iam-policy-binding \
"${ORG_ID}" \
--member="serviceAccount:${SA_ID}" \
--role="roles/compute.networkAdmin" \
--user-output-enabled false
gcloud organizations add-iam-policy-binding \
"${ORG_ID}" \
--member="serviceAccount:${SA_ID}" \
--role="roles/iam.serviceAccountAdmin" \
--user-output-enabled false
gcloud projects add-iam-policy-binding \
"${SHARED_VPC_PROJECT}" \
--member="serviceAccount:${SA_ID}" \
--role="roles/resourcemanager.projectIamAdmin" \
--user-output-enabled false
gcloud projects add-iam-policy-binding \
"${SHARED_VPC_PROJECT}" \
--member="serviceAccount:${SA_ID}" \
--role="roles/browser" \
--user-output-enabled falseError message:
Error: Error applying plan:
1 error(s) occurred:
* module.project-factory.google_project.project: 1 error(s) occurred:
* google_project.project: Error setting billing account "[billing-account]" for
project "projects/[target-project-id]": googleapi: Error 403: The caller does
not have permission, forbidden
Cause:
The Seed Service Account does not have the roles/billing.user role on the
billing account, and cannot link projects to the billing account. This will
prevent activation of GCP APIs (such as the Compute Engine API) that may incur
billing charges.
Solution:
Add the service account to the roles/billing.user role on the billing account.
- Fetch the billing account IAM policy.
# Requires `roles/billing.admin` on the billing account $ gcloud beta billing accounts get-iam-policy [billing-account] \ | tee policy.yml bindings: - members: - group:[email protected] role: roles/billing.admin - members: - group:[email protected] - serviceAccount:[service-account]@[seed-project-id].iam.gserviceaccount.com role: roles/billing.user etag: BwV3v6LFTjQ= - Update the policy to include the service account.
$ $EDITOR policy.yml - Verify that the policy is correct and only affects the service account.
$ cat policy.yml bindings: - members: - group:[email protected] role: roles/billing.admin - members: - group:[email protected] - serviceAccount:[service-account]@[seed-project-id].iam.gserviceaccount.com role: roles/billing.user etag: BwV3v6LFTjQ= - Update the billing account policy.
# Requires `roles/billing.admin` on the billing account $ gcloud beta billing accounts set-iam-policy [billing-account] policy.yml Updated IAM policy for account [[billing-account]].
Notes:
- Granting
roles/billing.useron the organization is not sufficient if the billing account is defined outside of the GCP organization. Make sure that the Seed Service Account has theroles/billing.userrole on the billing account. - If you encounter this error creating an initial project, e.g. using
simple_project, then you will also encounter the Unable to query status of default GCE service account issue on subsequent Terraform runs because the GCE API is not enabled.