How to setup acceptance tests in Gitlab?

vasily.rodin

Hi community!

I'm trying to set up a pipeline with acceptance tests, for this I need a full multi-container setup of Spryker in CI.

If I'm using the config from https://docs.spryker.com/docs/ca/dev/configure-deployment-pipelines/configure-gitlab-pipelines.html#what-is-gitlab-pipelines , it gives me an error:

$ docker/sdk boot -v deploy.ci.yml
Docker Buildx plugin is not installed. Please, make sure Docker Buildx plugin is installed. How to install Docker Buildx(https://docs.docker.com/build/install-buildx/) 

I've also tested this configuration, which should include Buildx and the latest docker version:

test:end-to-end:
    only:
        refs:
            - develop
            - master
            - merge_requests
    stage: test
    image: docker:latest
    services:
        - docker:24.0.5-dind
    before_script:
        - apk update && apk upgrade && apk add bash curl git
    script:
        - docker info
        - docker-compose -v
        - git clone https://github.com/spryker/docker-sdk.git ./docker
        - bash docker/sdk boot -v deploy.ci.yml
        - bash docker/sdk up -t
        - bash docker/sdk testing console queue:worker:start --stop-when-empty

In that case, I get another error:

$ bash docker/sdk boot -v deploy.ci.yml
┌────╮       ┌─┐           ╭────┬────╮─┬─┐
│  ╮ │───┬───┤ ├─┬───┬─┬─┐ │ ───┤  ╮ │ ┌─┘
│  ╯ │ ┼ │ ├─┤───┤ ┼─┤ ┌─╯ ├─── │  ╯ │ └─┐
└────┴───┴───┴─┴─┴───┴─┘   └────┴────┴─┴─┘
                                          
Checking requirements... 0s [OK]
Checking docker version... 0s [OK]
Checking docker-compose version... 1s [OK]
Using deploy.ci.yml 
Building generator... 3s [DONE]
Copiyng assets... 0s [DONE]
Running generator 
PHP Fatal error:  Uncaught Symfony\Component\Yaml\Exception\ParseException: File "/data/deployment/project.yml" does not exist. in /data/vendor/symfony/yaml/Parser.php:54
Stack trace:
#0 /data/deploy-file-generator/src/Processor/Executor/PreExecutors/PrepareDeployFileTransferExecutor.php(45): Symfony\Component\Yaml\Parser->parseFile('/data/deploymen...')
#1 /data/deploy-file-generator/src/Processor/DeployFileProcessor.php(90): DeployFileGenerator\Processor\Executor\PreExecutors\PrepareDeployFileTransferExecutor->execute(Object(DeployFileGenerator\Transfer\DeployFileTransfer))
#2 /data/deploy-file-generator/src/Processor/DeployFileProcessor.php(38): DeployFileGenerator\Processor\DeployFileProcessor->execute(Array, Object(DeployFileGenerator\Transfer\DeployFileTransfer))
#3 /data/index.php(1402): DeployFileGenerator\Processor\DeployFileProcessor->process(Object(DeployFileGenerator\Transfer\DeployFileTransfer))
#4 /data/index.php(19): buildProjectYaml('/data/deploymen...')
#5 {main}
  thrown in /data/vendor/symfony/yaml/Parser.php on line 54

This looks to me like a permission issue? Or something else is wrong in the config?

There was a similar discussion before, but without resolution:

https://commercequest.space/discussion/1626/hey-maybe-anyone-has-encountered-such-problem-in-gitlab-ci-while-running-dind-uncaught-symfony-co

Any help is welcome :)

We're using a self-hosted Gitlab CI, maybe something to change on the server side?

ishakuta-spryker

I'd check (var_dump or something) the result of writing this file, here \DeployFileGenerator\Processor\Executor\PostExecutors\ExportDeployFileTransferToYamlExecutor::execute

or ls -lha to check if the file was generated after \DeployFileGenerator\Command\ConfigCommand

looks like I've missed it is a part of "pre" executor, I would still try to log the file presence during different stages of execution to understand the state of the filesystem

Serhii Doroshenkov

• Hey-hey!

_1. What in deploy.ci.yml ?

It looks like boot generation is failed, that's why you have no project.yml. I mean it can be error not related to permission.

2. You should have the same dind version as in parent daemon (as I remember) to not have strange side effects.

Maybe documentation was created when docker/compose:latest was for docker 19.03. (BTW It's one of best practices for docker images to not use 'latest' tag).

I have example from old setup:

default:
  image: docker:24.0.4
  services: 
    - name: docker:24.0.4-dind
  …

3. If you still have issues - my suggestion is to enable interactive session https://docs.gitlab.com/ee/ci/interactive_web_terminal/ to investigate it on the fly. Easiest way to put infinite sleep for pipeline script section and go into container (where is docker working, as it possible on self-hosted gitlab) and do the same (check on the fly what happens).

vasily.rodin

@ishakuta-spryker

looks like I've missed it is a part of "pre" executor, I would still try to log the file presence during different stages of execution to understand the state of the filesystem

Thanks for the hint! I've tried to add some additional commands in after_script section:

    after_script:
        - ls -la ./docker/deployment
        - ls -la ./docker/deployment/_tmp
        - cat ./docker/deployment/_tmp/project.yml

The file exists in a filesystem in tmp folder, which is expected if I look into bootstrap.sh

So my assumption is that something goes wrong with this file, when docker/sdk tries to mount it inside the container (so php from inside the container doesn't see the file, which exists outside)

@Serhii Doroshenkov

Thank you too! :)

1. What in deploy.ci.yml ?

deploy.ci.yml exists and I checked it locally - bootstrap happens without problems.

2. You should have the same dind version as in parent daemon (as I remember) to not have strange side effects.

I've tried with fixed versions, unfortunately the problem is the same

3. If you still have issues - my suggestion is to enable interactive session  https://commercequest.space/home/leaving?allowTrusted=1&target=https%3A%2F%2Fdocs.gitlab.com%2Fee%2Fci%2Finteractive_web_terminal%2F  to investigate it on the fly

Nice feature, didn't know about this. But it requires some special server configuration. Will have a look inside for more details and post here later!