[Deprecated] TFX Metadata Reviews

Diptimayee B. · Reviewed 12 months ago

Roberto G. · Reviewed 12 months ago

AI Platform doesn't show the created pipeline

Oğulcan C. · Reviewed 12 months ago

pipeline error out

jun f. · Reviewed 12 months ago

AMANDEEP S. · Reviewed 12 months ago

Kok Khuen Y. · Reviewed 12 months ago

Followed all the steps but pipeline deployment job is failing everytime. This happened multiple times. ModuleNotFoundError: No module named 'google.appengine' During handling of the above exception, another exception occurred: ModuleNotFoundError: No module named 'oauth2client.locked_file'

Jiji A. · Reviewed 12 months ago

Pravin T. · Reviewed 12 months ago

"file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth" ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/ml/v1/rest INFO:absl:TrainingInput={'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-686a4c694742/tfx_covertype_lab_04@sha256:84862572ea0faa96241cfbd498f591b8a37a69f27230bf97380f42e8270cf65b', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "13", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Transform/transformed_examples/12", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"state": {"string_value": "published"}, "producer_component": {"string_value": "Transform"}, "name": {"string_value": "transformed_examples"}}, "state": "LIVE", "create_time_since_epoch": "1728053526829", "last_update_time_since_epoch": "1728053613129"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"version": "INT", "split_names": "STRING", "span": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "12", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Transform/transform_graph/12", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1728053526826", "last_update_time_since_epoch": "1728053613127"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "17", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Tuner/best_hyperparameters/16", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-686a4c694742/tfx_covertype_lab_04@sha256:84862572ea0faa96241cfbd498f591b8a37a69f27230bf97380f42e8270cf65b\\"}, \\"project\\": \\"qwiklabs-gcp-01-686a4c694742\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-686a4c694742.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-q4k8x-3115704825", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'region': 'us-central1', 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-686a4c694742.iam.gserviceaccount.com', 'workerCount': 2, 'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard'} INFO:absl:Submitting job='tfx_tuner_20241004145358', project='qwiklabs-gcp-01-686a4c694742' to AI Platform. INFO:googleapiclient.discovery:URL being requested: POST https://ml.googleapis.com/v1/projects/qwiklabs-gcp-01-686a4c694742/jobs?alt=json INFO:googleapiclient.discovery:URL being requested: GET https://ml.googleapis.com/v1/projects/qwiklabs-gcp-01-686a4c694742/jobs/tfx_tuner_20241004145358?alt=json ERROR:absl:Job 'projects/qwiklabs-gcp-01-686a4c694742/jobs/tfx_tuner_20241004145358' did not succeed. Detailed response {'jobId': 'tfx_tuner_20241004145358', 'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard', 'workerCount': '2', 'region': 'us-central1', 'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-686a4c694742/tfx_covertype_lab_04@sha256:84862572ea0faa96241cfbd498f591b8a37a69f27230bf97380f42e8270cf65b', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "13", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Transform/transformed_examples/12", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"state": {"string_value": "published"}, "producer_component": {"string_value": "Transform"}, "name": {"string_value": "transformed_examples"}}, "state": "LIVE", "create_time_since_epoch": "1728053526829", "last_update_time_since_epoch": "1728053613129"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"version": "INT", "split_names": "STRING", "span": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "12", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Transform/transform_graph/12", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1728053526826", "last_update_time_since_epoch": "1728053613127"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "17", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Tuner/best_hyperparameters/16", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-686a4c694742/tfx_covertype_lab_04@sha256:84862572ea0faa96241cfbd498f591b8a37a69f27230bf97380f42e8270cf65b\\"}, \\"project\\": \\"qwiklabs-gcp-01-686a4c694742\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-686a4c694742.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-q4k8x-3115704825", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-686a4c694742.iam.gserviceaccount.com'}, 'createTime': '2024-10-04T14:53:58Z', 'startTime': '2024-10-04T14:54:13Z', 'endTime': '2024-10-04T14:54:44Z', 'state': 'FAILED', 'errorMessage': 'The replica master 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 1 exited with a non-zero status of 1. Termination reason: Error. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=359700377496&resource=ml_job%2Fjob_id%2Ftfx_tuner_20241004145358&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22tfx_tuner_20241004145358%22 ', 'trainingOutput': {'consumedMLUnits': 0.01}, 'labels': {'tfx_executor': 'ensions-google_cloud_ai_platform-tuner-executor-_workerexecutor', 'tfx_py_version': '3-7', 'tfx_runner': 'kfp', 'tfx_version': '0-25-0'}, 'etag': 'LDhHzlkwJDE=', 'jobPosition': '0'}. Traceback (most recent call last): File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in <module> main() File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main execution_info = launcher.launch() File "/tfx-src/tfx/orchestration/launcher/base_component_launcher.py", line 209, in launch copy.deepcopy(execution_decision.exec_properties)) File "/tfx-src/tfx/orchestration/launcher/in_process_component_launcher.py", line 72, in _run_executor copy.deepcopy(input_dict), output_dict, copy.deepcopy(exec_properties)) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/tuner/executor.py", line 121, in Do job_id) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/runner.py", line 305, in start_aip_training job_labels=job_labels) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/runner.py", line 194, in _launch_aip_training raise RuntimeError(err_msg) RuntimeError: Job 'projects/qwiklabs-gcp-01-686a4c694742/jobs/tfx_tuner_20241004145358' did not succeed. Detailed response {'jobId': 'tfx_tuner_20241004145358', 'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard', 'workerCount': '2', 'region': 'us-central1', 'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-686a4c694742/tfx_covertype_lab_04@sha256:84862572ea0faa96241cfbd498f591b8a37a69f27230bf97380f42e8270cf65b', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "13", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Transform/transformed_examples/12", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"state": {"string_value": "published"}, "producer_component": {"string_value": "Transform"}, "name": {"string_value": "transformed_examples"}}, "state": "LIVE", "create_time_since_epoch": "1728053526829", "last_update_time_since_epoch": "1728053613129"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"version": "INT", "split_names": "STRING", "span": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "12", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Transform/transform_graph/12", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1728053526826", "last_update_time_since_epoch": "1728053613127"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "17", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-686a4c694742-kubeflowpipelines-default/tfx_covertype_lab_04/1f352760-73d5-4804-b8e3-5bc06920786e/Tuner/best_hyperparameters/16", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-686a4c694742/tfx_covertype_lab_04@sha256:84862572ea0faa96241cfbd498f591b8a37a69f27230bf97380f42e8270cf65b\\"}, \\"project\\": \\"qwiklabs-gcp-01-686a4c694742\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-686a4c694742.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-q4k8x-3115704825", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-686a4c694742.iam.gserviceaccount.com'}, 'createTime': '2024-10-04T14:53:58Z', 'startTime': '2024-10-04T14:54:13Z', 'endTime': '2024-10-04T14:54:44Z', 'state': 'FAILED', 'errorMessage': 'The replica master 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 1 exited with a non-zero status of 1. Termination reason: Error. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=359700377496&resource=ml_job%2Fjob_id%2Ftfx_tuner_20241004145358&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22tfx_tuner_20241004145358%22 ', 'trainingOutput': {'consumedMLUnits': 0.01}, 'labels': {'tfx_executor': 'ensions-google_cloud_ai_platform-tuner-executor-_workerexecutor', 'tfx_py_version': '3-7', 'tfx_runner': 'kfp', 'tfx_version': '0-25-0'}, 'etag': 'LDhHzlkwJDE=', 'jobPosition': '0'}. time="2024-10-04T14:55:00.707Z" level=info msg="sub-process exited" argo=true error="<nil>" time="2024-10-04T14:55:00.707Z" level=error msg="cannot save artifact /mlpipeline-ui-metadata.json" argo=true error="stat /mlpipeline-ui-metadata.json: no such file or directory"

Billy B. · Reviewed 12 months ago

pipeline can not complete

Billy B. · Reviewed 12 months ago

Diogo S. · Reviewed 12 months ago

Not completable, cluster creation takes ages and fails.

Bartosz D. · Reviewed 12 months ago

Jesús A. · Reviewed 12 months ago

AMANDEEP S. · Reviewed 12 months ago

Castro D. · Reviewed 12 months ago

This lab isnt functional, please fix it, it doesnt make sense and it cant be completed

Isaac F. · Reviewed 12 months ago

Victor Hugo V. · Reviewed 12 months ago

cluster wont start

Min L. · Reviewed 12 months ago

wade a. · Reviewed 12 months ago

Kubernetes cluster deploys fail because of lack of resources. Again ..4 times

J. S. · Reviewed 12 months ago

Kubernetes cluster deploys fail because of lack of resources. Again ..

J. S. · Reviewed 12 months ago

Followed all the steps but pipeline deployment job is failing everytime. This happened multiple times. UnavailableError: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:7000: Failed to connect to remote host: Connection refused. Is the error im getting though I followed all the steps.

Aishwarya K. · Reviewed 12 months ago

Port forwarding cannot be configured due to outdated gcloud sdk which cannot easily be updated, lab cannot be completed

Patrick S. · Reviewed 12 months ago

Nathan L. · Reviewed 12 months ago

淳晰 施. · Reviewed 12 months ago