Calling Developers!
We are reenergizing our code contribution process! Learn More

What are the Slack Archives?

It’s a history of our time together in the Slack Community! There’s a ton of knowledge in here, so feel free to search through the archives for a possible answer to your question.

Because this space is not active, you won’t be able to create a new post or comment here. If you have a question or want to start a discussion about something, head over to our categories and pick one to post in! You can always refer back to a post from Slack Archives if needed; just copy the link to use it as a reference..

Hi spryker developers & architects, we have an interesting point regarding spryker scaling and

Options
UMVENKCBF
UMVENKCBF Posts: 38 🧑🏻‍🚀 - Cadet
edited May 2022 in Help

Hi spryker developers & architects,

we have an interesting point regarding spryker scaling and rollouts of shops in Spryker Paas environments.

Let's assume we have the following setup:
• Spryker PaaS environment at AWS
15 shops (AT, DE, CZ, CH, ....)
shared database persistence
• as in Spryker PaaS designed 1 redis instance
• usage of Jenkins as scheduler
• many custom jobs which are triggered in Jenkins nearly at them same time because of dependencies to the backend systems (like SAP, PIM, ...)
• default Jenkins scale at Spryker PaaS with 2vCPU and 4GB memory
Let's go in deeper:
• each Jenkins job CAN CONSUME 1 vCPU caused by PHP architecture
• 15 shops will trigger 15 Jenkins Jobs for processing the queue (jobs like XX__queue-worker-start) with a maximum of 1 vCPU
◦ Savepoint: 15 jenkins jobs
• for concurrency it will be nice to have a second Jenkins job (worker) for another jobs which will be triggered (like imports, exports, ..., sync with external systems)
◦ Savepoint: 15 jenkins jobs
In summary we can plan (minimum worst case scenario) 30 concurrent running Jenkins jobs on one virtual machine.

Our thoughts are - this setup does not scale with additional shops and Jenkins jobs. There will be a problem in future with concurrency and it's not the task from development team to plan Jenkins job concurrency - system infrastructure should scale with new shops.
Jenkins jobs will block each other or jobs can consume the amount a maximum of cpu power.
In addition we should note the default usage of jenkins which consumes cpu power....

Whats your opinion? What are your experiences regarding this topic in Spryker Paas? How many shops running in parallel with the default setup?

Comments

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    Options

    My personal experience and recommendation would be to have 1 vCPU per store.
    So in case of 15 stores 16 vCPU instance should be enough to deal with the load if the application is optimized (like imports run only for deltas, etc).

  • UMVENKCBF
    UMVENKCBF Posts: 38 🧑🏻‍🚀 - Cadet
    Options

    In theory: yes. Means one jenkins job will block or reserve the cpu power from 1 vCPU.
    If you have long running job(s) you might ran in problems.
    Worst case: if 15 workers for importing products will ran with full cpu power there is no power left for running the queue worker jobs.

    Scaling in Spryker PaaS for jenkins is in dependency to rolled out shops, right? Or are there any other scaling options like jenkins master / slave concept with multiple virtual machines (ec2 instances)?

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    Options

    Didn't see a case where there are 15 import jobs in parallel.
    Also queue:worker console command has -s option so it does't run at idle for 1 minute if the queue is empty.

    The only option of scaling for Jenkins in the cloud is vertical.
    Locally you have Jenkins that curls to zed\backoffice and zed\backoffice executes php.
    In PaaS Jenkins has application code inside itself and runs php on its own.

  • U013EK8RD7A
    U013EK8RD7A Posts: 127 🧑🏻‍🚀 - Cadet
    Options

    @valerii.trots 😄, special for You

  • U013EK8RD7A
    U013EK8RD7A Posts: 127 🧑🏻‍🚀 - Cadet
    edited May 2022
    Options

    we have 20 executors, so 5 are available for regular jobs (such as queue consumer) all these jobs consume 60% - 70% of CPU power of t3.xlarge EC2 instance.

    @UMVENKCBF, for Jenkins queue worker You could configure this:

    $config[QueueConstants::QUEUE_ADAPTER_CONFIGURATION] = [
        EventConstants::EVENT_QUEUE => [
            QueueConfig::CONFIG_QUEUE_ADAPTER => RabbitMqAdapter::class,
            QueueConfig::CONFIG_MAX_WORKER_NUMBER => 5,
        ],
        PublisherConfig::PUBLISH_QUEUE => [
            QueueConfig::CONFIG_QUEUE_ADAPTER => RabbitMqAdapter::class,
            QueueConfig::CONFIG_MAX_WORKER_NUMBER => 10,
        ],
    ];
    
  • UMVENKCBF
    UMVENKCBF Posts: 38 🧑🏻‍🚀 - Cadet
    Options

    Okay thanks @U013EK8RD7A - how it's working with your case / configuration.
    Nice to see that there is some tweaking and that instance typ assigned to your landscape.

  • UP8T3GKFH
    UP8T3GKFH Posts: 56 🧑🏻‍🚀 - Cadet
    Options

    Hello @U013EK8RD7A, could you please explain to me how this works with the 10 workers, when a t3.xlarge only has 4 vCPUs available. My knowledge is that you should not enter more workers than CPU cores, as a worker reserves a core for itself. This would create a massive queue of workers, wouldn't it?