Calling Developers!
We are reenergizing our code contribution process! Learn More

What are the Slack Archives?

It’s a history of our time together in the Slack Community! There’s a ton of knowledge in here, so feel free to search through the archives for a possible answer to your question.

Because this space is not active, you won’t be able to create a new post or comment here. If you have a question or want to start a discussion about something, head over to our categories and pick one to post in! You can always refer back to a post from Slack Archives if needed; just copy the link to use it as a reference..

[SOLVED] Hi everyone 🤝, I have some configuration for queue: ```// ---------- Queue $conf

Options
giovanni.piemontese
giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
edited November 2021 in Help

[SOLVED] Hi everyone 🤝,

I have some configuration for queue:

// ---------- Queue
$config[QueueConstants::QUEUE_ADAPTER_CONFIGURATION_DEFAULT] = [
    QueueConfig::CONFIG_QUEUE_ADAPTER => RabbitMqAdapter::class,
    QueueConfig::CONFIG_MAX_WORKER_NUMBER => 12,
];

$config[QueueConstants::QUEUE_ADAPTER_CONFIGURATION][ProductPageSearchConfig::PUBLISH_PRODUCT_ABSTRACT_PAGE][QueueConfig::CONFIG_MAX_WORKER_NUMBER] = 6;

As u see I defined a specific CONFIG_MAX_WORKER_NUMBER for queue PUBLISH_PRODUCT_ABSTRACT_PAGE because it has an high usage of memory (known issue!).
Since this configuration the worker does not start more the task event for this specific queue.
I debugged \Spryker\Zed\Queue\Business\Worker\Worker::startProcesses to see if the right number of maxWorkerNumbers is read and calculated and everything is fine, but no processes are started.

Can anyone help me to understand the problem? Thank u in advance.!!!

«1

Comments

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    PUBLISH_PRODUCT_ABSTRACT_PAGE looks like the key for the queue messages - it’s not a queue itself.

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    I think this config only works with the queue names that you see in the RabbitMQ UI queues page, eg sync.search.product

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    the queue name is right... and the queue is also there..

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    ah interesting - I’ve not seen the publish queue split like this before

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    and the configuration is according the name of queue too...

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    I have every queue splitted ... i have ca. 100 queues...

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    Wow

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    edited November 2021
    Options

    I’ve just checked our config for this, it’s slightly different to yours

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    Maybe the adapter class is relevant?

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    yes.. u are right.. maybe this one... i will try

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    Otherwise I’m out of ideas…

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    Out of interest is the publish queue splitting a Spryker feature, or something custom you have done?

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    Spryker standard and some customizations

  • U01T075RRHD
    U01T075RRHD Posts: 118 🧑🏻‍🚀 - Cadet
    Options

    I you're running into memory issues with RabbitMq, have a look at lazy queues:

    https://www.rabbitmq.com/lazy-queues.html

  • U01T075RRHD
    U01T075RRHD Posts: 118 🧑🏻‍🚀 - Cadet
    Options

    They can reduce the memory footprint significantly

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    @U01T075RRHD the problem about memory issue is related to process consumed from spryker to publish the data in the spy_product_abstract_page_search ... Is related to a lot of data loaded and to propel gc too...

    what do u mean with rabbitMq memory issue? In RMQ Server Ram used?

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    @UKEP86J66 it works!!! Thanks! U saved my day 😄 ...

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    🥳 great to hear!

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    I just forgot to specify the adapter class.. in this case the worker was not able to retrieve the message and start the process

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    By the way we had issues with memory for product page search and found there are 2 columns for data in that table. One is for debugging purposes and can be left blank which will certainly reduce the propel memory.

  • UKEP86J66
    UKEP86J66 Posts: 208 🧑🏻‍🚀 - Cadet
    Options

    I asked about it in Spryker slack but I cannot find the link though 😞

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    I think that the structured_data field (this that u left blank) is thought for the refresh command..

  • U01T075RRHD
    U01T075RRHD Posts: 118 🧑🏻‍🚀 - Cadet
    Options

    @UL65CH0MC Yes, my suggestion targeted RabbitMq memory usage ... so I guess I got you all wrong 😂 . Happy you could work out a solution

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    But i can check if the memory usage is related to this field..

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    @U01T075RRHD thanks.. But i am interested how did u changed the queues to lazy via spryker... did u overwrite the queue:setup ?

  • U01T075RRHD
    U01T075RRHD Posts: 118 🧑🏻‍🚀 - Cadet
    Options

    You can provide it in the queue setup using RabbitMqOptionTransfer::setArguments()

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    @U01T075RRHD so ?

        /**
         * @param string $queueName
         * @param string $routingKey
         *
         * @return \Generated\Shared\Transfer\RabbitMqOptionTransfer
         */
        protected function createQueueOptionTransfer($queueName, $routingKey = '')
        {
            $queueOptionTransfer = parent::createQueueOptionTransfer($queueName, $routingKey);
    
            $queueOptionTransfer->setArguments(['queue-mode' => 'lazy']);
    
            return $queueOptionTransfer;
        }
    

    Or the arguments should be passed in another way?

  • U01T075RRHD
    U01T075RRHD Posts: 118 🧑🏻‍🚀 - Cadet
    Options

    It should be x-queue-mode I guess