图片生成Gif任务 - II

创建 AWS Batch Job Definition

在本节中，我们将创建两个Job Definition:

  • 第一个Job Definition将从互联网提取图像并将其放置在选择的 S3 存储桶中。
  • 第二个将采用一组图像的 S3 前缀作为输入，以生成动画 GIF，然后将生成的动画上传到指定的 S3 位置。

Job Definition 1：将图像复制到的存储桶

创建Job Definition模板。请注意使用作业定义参数以及 pod 的 Kubernetes servica account来启用对私有 ECR 和 S3 资源的访问:

JD_CP_NAME="b4eks-jd-cp-$(uuidgen --random | cut -d'-' -f1)"
echo "export JD_CP_NAME=${JD_CP_NAME}" >> ~/.bash_profile

cat > ${JD_CP_NAME}.json <<EOF
{
  "jobDefinitionName": "${JD_CP_NAME}",
  "type": "container",
  "parameters": {
    "Source": "NULL",
    "Destination":  "NULL"
  },
  "eksProperties": {
    "podProperties": {
      "serviceAccountName": "${BATCH_EKS_S3ECR_SA}",
      "hostNetwork": true,
      "containers": [
        {
          "image": "${ECR_REPO_GIF}:latest",
          "command": [
            "copy-to-s3.sh",
            "Ref::Source",
            "Ref::Destination"
          ],
          "resources": {
            "limits": {
              "cpu": "1",
              "memory": "1024Mi"
            }
          }
        }
      ]
    }
  }
}
EOF

image-20240107191137481

使用 AWS CLI 注册 AWS Batch Job Definition:

aws batch register-job-definition --cli-input-json file://./${JD_CP_NAME}.json

应该看到如下所示的输出：

{
    "jobDefinitionName": "b4eks-jd-cp-403f8225",
    "jobDefinitionArn": "arn:aws:batch:us-east-1:111111111111:job-definition/b4eks-jd-cp-403f8225:1",
    "revision": 1
}

Job Definition 2：从源图像制作GIF

创建Job Definition模板:

export JD_MKGIF_NAME="b4eks-js-mkgif-$(uuidgen --random | cut -d'-' -f1)"
echo "export JD_MKGIF_NAME=${JD_MKGIF_NAME}" >> ~/.bash_profile

cat > ${JD_MKGIF_NAME}.json <<EOF
{
  "jobDefinitionName": "${JD_MKGIF_NAME}",
  "type": "container",
  "parameters": {
    "Source": "NULL",
    "Destination": "NULL"
  },
  "eksProperties": {
    "podProperties": {
      "serviceAccountName": "${BATCH_EKS_S3ECR_SA}",
      "hostNetwork": true,
      "containers": [
        {
          "image": "${ECR_REPO_GIF}:latest",
          "command": [
            "make-gif.sh",
            "Ref::Source",
            "Ref::Destination"
          ],
          "resources": {
            "limits": {
              "cpu": "1",
              "memory": "1024Mi"
            }
          }
        }
      ]
    }
  }
}
EOF

使用 AWS CLI 注册 AWS Batch Job Definition:

aws batch register-job-definition --cli-input-json file://./${JD_MKGIF_NAME}.json

应该看到如下所示的输出：

{
    "jobDefinitionName": "b4eks-js-mkgif-ac677469",
    "jobDefinitionArn": "arn:aws:batch:us-east-1:111111111111:job-definition/b4eks-js-mkgif-ac677469:1",
    "revision": 1
}

提交作业并查看结果

现在，我们将拉取一些图像作为单独的作业，然后发出作业将它们合并为 gif。然后我们将查看结果。

将图像复制到S3 存储桶:

JOB_IDS=()
JOB_IDS+=($(aws batch submit-job --job-name 'eks-copy-img-1' \
  --job-queue ${BATCH_EKS_JQ_NAME} \
  --job-definition "${JD_CP_NAME}:1" \
  --parameters Source='https://pingfan.s3.amazonaws.com/files/batch-101-make-gif-01.jpeg',Destination="s3://${BUCKET_NAME}/input/01.png" \
  --query "jobId" --output text))


JOB_IDS+=($(aws batch submit-job --job-name 'eks-copy-img-2' \
  --job-queue ${BATCH_EKS_JQ_NAME} \
  --job-definition "${JD_CP_NAME}:1" \
  --parameters Source='https://pingfan.s3.amazonaws.com/files/batch-101-make-gif-02.jpeg',Destination="s3://${BUCKET_NAME}/input/02.png" \
  --query "jobId" --output text))

JOB_IDS+=($(aws batch submit-job --job-name 'eks-copy-img-3' \
  --job-queue ${BATCH_EKS_JQ_NAME} \
  --job-definition "${JD_CP_NAME}:1" \
  --parameters Source='https://pingfan.s3.amazonaws.com/files/batch-101-make-gif-03.jpeg',Destination="s3://${BUCKET_NAME}/input/03.png" \
  --query "jobId" --output text))

JOB_IDS+=($(aws batch submit-job --job-name 'eks-copy-img-4' \
  --job-queue ${BATCH_EKS_JQ_NAME} \
  --job-definition "${JD_CP_NAME}:1" \
  --parameters Source='https://pingfan.s3.amazonaws.com/files/batch-101-make-gif-04.jpeg',Destination="s3://${BUCKET_NAME}/input/04.png" \
  --query "jobId" --output text))

echo "JOB_IDS = ${JOB_IDS[@]}"

image-20240107191639367

可以在AWS Batch 控制台( https://console.aws.amazon.com/batch/ )中检查作业的状态 或在终端中使用以下命令：

aws batch describe-jobs  --jobs "${JOB_IDS[@]}" --query "jobs[].[jobName, status]"

作业完成后，可以提交作业来提取图像并制作动画 GIF:

image-20240107192017176

创建动画 GIF

JOB_ID=$(aws batch submit-job --job-name 'eks-make-gif-1' \
  --job-queue ${BATCH_EKS_JQ_NAME} \
  --job-definition "${JD_MKGIF_NAME}:1" \
  --parameters Source="s3://${BUCKET_NAME}/input/",Destination="s3://${BUCKET_NAME}/output/myCoolGif.gif" \
  --output text --query jobId)

可以使用describe-jobs命令检查作业的状态:

aws batch describe-jobs  --jobs "${JOB_ID}" --query "jobs[].[jobName, status]"

image-20240107192154864

完成后，可以使用S3控制台下载并观看gif :

image-20240107192331868

清理资源

删除 S3 存储桶:

aws s3 rb s3://${BUCKET_NAME}  --force

删除 ECR 存储库:

aws ecr delete-repository --repository-name ${ECR_REPO_NAME_GIF} --force

取消注册 AWS Batch Job Definition:

aws batch deregister-job-definition --job-definition ${JD_CP_NAME}:1
aws batch deregister-job-definition --job-definition ${JD_MKGIF_NAME}:1

删除 EKS IAM service account和 IAM Policy：

eksctl delete iamserviceaccount \
  --name ${BATCH_EKS_S3ECR_SA_NAME} \
  --namespace ${BATCH_EKS_NAMESPACE} \
  --cluster ${BATCH_EKS_CLUSTER_NAME} \
  --wait

aws iam delete-policy --policy-arn ${BATCH_EKS_S3ECR_POLICY_ARN}