google-cloud-platform - 如何在 Airflow 中使用 Google Operators 设置范围

我有一个使用 Airflow 中的 https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/suite/transfers/gcs_to_sheets/index.html 的任务,我试图将数据添加到工作表中。

googleapiclient.errors.HttpError: 
<HttpError 403 when requesting 
https://sheets.googleapis.com/v4/spreadsheets/<SHEET_ID>/values/Sheet1?valueInputOption=RAW&includeValuesInResponse=false&responseValueRenderOption=FORMATTED_VALUE&responseDateTimeRenderOption=SERIAL_NUMBER&alt=json 

returned "Request had insufficient authentication scopes.". 

    Details: "[{
'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 
'reason': 'ACCESS_TOKEN_SCOPE_INSUFFICIENT', 
'domain': 'googleapis.com', 
'metadata': {
    'service': 'sheets.googleapis.com', 
    'method': 'google.apps.sheets.v4.SpreadsheetsService.UpdateValues'}
    }]>

AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fopt%2Fairflow%2Fcredentials%2Fgoogle_credential.json

我尝试按照https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html 添加https://developers.google.com/identity/protocols/oauth2/scopes#sheets https://www.googleapis.com/auth/spreadsheets

AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fopt%2Fairflow%2Fcredentials%2Fgoogle_credential.json&extra__google_cloud_platform__scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets

google.api_core.exceptions.Forbidden: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/my-project/jobs?prettyPrint=false: Request had insufficient authentication scopes.

并且 GCSToGoogleSheetsOperator 运算符仍然会出现这样的错误:

google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/download/storage/v1/b/my-bucket/o/folder%2Fobject.csv?alt=media: Insufficient Permission: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

如何正确设置权限以同时使用 BigQuery、GCS 和 Sheets 运算符?

回答1

添加范围似乎忽略了 IAM 角色,因此它是一个或另一个。

服务帐户具有访问 GCS 和 BigQuery 所需的角色,但通过添加范围 https://www.googleapis.com/auth/spreadsheets,服务将忽略角色授予的权限并仅查看范围指定的权限。

因此,要恢复它,您必须同时添加 spreadsheetcloud-platform 范围(或更严格的范围)。 cloud-platform 将提供对 GCS 和 BigQuery 以及 spreadsheets 对 Google Sheets API 的访问。

如果您使用环境变量设置连接,则必须对参数进行 URL 编码,因此要创建 GOOGLE_CLOUD 连接,您必须执行类似的操作,但未编码...

AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=/abs/path_to_file/credential.json&extra__google_cloud_platform__scope=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/spreadsheets

要编码,这是您必须使用的版本,请替换 /,:

AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fabs%2Fpath_to_file%2Fcredentials%2Fgoshare-driver-c08e0904285b.json&extra__google_cloud_platform__scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets

相似文章

最新文章