Troubleshooting Build and Deployment Process for DataHub Forked Version with CI/CD

all-things-deployment
1

Original Slack Thread

Hi all! We’ve been using DataHub for a while now and are really enjoying it! We’re now trying to deploy a forked version of the repo, and working on getting our CI/CD sorted. We’ve been able to modify our Helm chart to pull from ECS successfully, the only part we’re stuck on is how to build and deploy the docker images up into ECS. I’ve tried using this:

- create python env
- login to ecr...

- name: Gradle Build
  continue-on-error: true
  run: |
    ./gradlew clean
    ./gradlew build

- name: Deploy to ECS
  run: |
     ./gradlew quickstart --info
     {tag and upload images to ECS}```

The build is successful (once I exclude tests), but it always fails at the `./gradlew quickstart` step with varying errors. We haven’t made any changes to the codebase yet. I’m curious if anyone else has solved this problem already? Otherwise I’ll update here with what I did to get it working.
2

The real question here is: What is the most lightweight way to build the GMS and Frontend Images? That’s all we need for our deployment. I’ve tried using this:

(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml build datahub-frontend-react)
But the image isn’t everything needed for the DataHub frontend (I think?)

3

<@U05SKM6KGGK> might be able to speak to this!

4

GMS build: https://github.com/acryldata/datahub/blob/a02a45cba1f4679c8957afc80c2e5879ca203412/.github/workflows/docker-unified.yml#L77-L101
Frontend build: https://github.com/acryldata/datahub/blob/a02a45cba1f4679c8957afc80c2e5879ca203412/.github/workflows/docker-unified.yml#L309-L335

5

We managed to get this working via the following workflow:

    name: Build and Push DataHub Frontend Docker Image
    runs-on: [ self-hosted, &lt;our_runner&gt; ] # this could be ubuntu for others
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Login to ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - uses: actions/setup-python@v4
        with:
          python-version: "3.10"
          cache: pip

      - name: Set up JDK 11
        uses: actions/setup-java@v3
        with:
          distribution: "zulu"
          java-version: 11
          cache: 'gradle'

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v2
        with:
          registry: <http://ghcr.io|ghcr.io>
          username: ${{ github.actor }}
          password: ${{ secrets.GIT_TOKEN }}

      - name: Gradle Build
        continue-on-error: true
        run: |
          ./gradlew :datahub-frontend:dist -x test -x yarnTest -x yarnLint --parallel
          mv ./datahub-frontend/build/distributions/datahub-frontend-*.zip datahub-frontend.zip

      - name: Deploy to ECR
        env:
          IMAGE_REGISTRY: ${{ steps.login-ecr.outputs.registry }}/cko-dp
        run: |
          docker build -f ./docker/datahub-frontend/Dockerfile -t $IMAGE_REGISTRY/$FRONTEND_IMAGE_NAME:$IMAGE_TAG . 
          docker images
          docker push $IMAGE_REGISTRY/$FRONTEND_IMAGE_NAME:$IMAGE_TAG```
And for the gms build it was similar but looks like this:

``` - name: Gradle Build
        continue-on-error: true
        run: |
          ./gradlew :metadata-service:war:build -x test --parallel
          mv ./metadata-service/war/build/libs/war.war .

      - name: Deploy to ECR
        env:
          IMAGE_REGISTRY: ${{ steps.login-ecr.outputs.registry }}/cko-dp
        run: |
          docker build -f ./docker/datahub-gms/Dockerfile -t $IMAGE_REGISTRY/$GMS_IMAGE_NAME:$IMAGE_TAG .
          docker images
          docker push $IMAGE_REGISTRY/$GMS_IMAGE_NAME:$IMAGE_TAG```
6

So pretty similar to what you guys have! Just some small adjustments to get it to work with ECR