Calling Developers!
We are reenergizing our code contribution process! Learn More

What are the Slack Archives?

It’s a history of our time together in the Slack Community! There’s a ton of knowledge in here, so feel free to search through the archives for a possible answer to your question.

Because this space is not active, you won’t be able to create a new post or comment here. If you have a question or want to start a discussion about something, head over to our categories and pick one to post in! You can always refer back to a post from Slack Archives if needed; just copy the link to use it as a reference..

Hi all, I am facing an issue with publish&synchronize out of nowhere where I need your help. Tri

Options
U01F7P3D9NH
U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet

Hi all, I am facing an issue with publish&synchronize out of nowhere where I need your help. Triggering all abstracts ends in an infinite loop. F.e. 2000 msg in event queue getting unacked, pushes msgs successfully to sync.storage and then event msgs getting back to ready. No error, no retry. Pushing smaller batches (10-20 abstracts) mostly getting processed. Tried to track them down to the faulty one, thought I found it, but trigger the individual item worked also. With 500k msgs, it is not a good workaround 🙂 Chunksizes are limited already. Any ideas?

Comments

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 🧑🏻‍🚀 - Cadet
    Options

    could be memory limit or OOM (out of memory) error if chunks are to big. when it's "only" memory limit, you can try to start more workers in parallel (see config: \Spryker\Shared\Queue\QueueConstants::QUEUE_ADAPTER_CONFIGURATION) instead of having too big chunk sizes. You can also try reducing event queue chunk size and increasing only publish and storage queues chunk sizes (especially writing to storage can usually be much bigger), but this depends a bit on your implementation how and if this is possible by configuration

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    product_abstract P+S has in Spryker a problem about a lot of memory allocation and cpu time... the problem is mainly related to the attributes...

    how many attributes do u have in your system imported?

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    edited December 2022
    Options

    I'll check php error logs in the container that are not pushed to cloudwatch yet and which you can't check on your own because of that.

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    Options

    There are none. 🤔

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    Options

    And last OOM killer involvement was on 1st of November.

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    Options
  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options

    @valerii.trots it is reproducable on local env, so you could tell me where to look. @UL65CH0MC it is a difference how many abstracts are pushed in queue? because with 10 abstracts it works fine

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    @U01F7P3D9NH, yes, absolutely

  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options

    1,6k attributes in the table shouldn´t break everything 🙂

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    on my project i optimized it and reduced the memory to standard ca. 120MB per process and execution time to a few seconds instead minutes and ca,2,5GB for every chunk...

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    1,6K is not lot, but can be the problem... There are some parts of codes where spryker make a lot of foreach wihtout sense and load a lot of not necessary data from db etc etc...

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    I have also created since 1 year a support ticket and attached the pyz fix, but till now no fix in core...

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    even if it is the same problem.. but i suppose yes, it is (i think)

  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options

    i´ll try it now with chunk limit of 5 but it looks kinda the same, storage queues are filled and processed but "main" message stays

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    what events are in the main message? can u please get a message and see the type/name?

  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options
    {"listenerClassName":"Spryker\\Zed\\ProductStorage\\Communication\\Plugin\\Event\\Listener\\ProductAbstractStoragePublishListener","transferClassName":"Generated\\Shared\\Transfer\\EventEntityTransfer","transferData":{"id":"6","event":null,"name":null,"foreign_keys":[],"modified_columns":[],"original_values":[],"additional_values":[]},"eventName":"Product.product_abstract.publish"}
    
  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    but it is a product storage.. and u said that the storage msgs are consumed properly... or not?

  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options

    I paused one queue so you can see that things are processed

  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options

    I purge event queue and the other queues getting empty

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    how it is configured your sync chunk?

  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options
    // ---------- Event
    $config[EventConstants::EVENT_CHUNK] = 5;
    
    // ---------- Synchronization
    $config[SynchronizationConstants::EXPORT_MESSAGE_CHUNK_SIZE] = 5;
    $config[SynchronizationConstants::DEFAULT_SYNC_STORAGE_MESSAGE_CHUNK_SIZE] = 5;
    $config[SynchronizationConstants::DEFAULT_SYNC_SEARCH_QUEUE_MESSAGE_CHUNK_SIZE] = 5;
    
  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet
    Options

    the problem that i explained before was more related to product_abstract_search and product_concrete_search publish events... but in your case it seems that you have probably timeouts by sync process and therefore u not get the ack status...

  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options

    super strange... how can 100k concrete with chunk 100 running smooth vs 10 abstracts and chunk 5 fail without and sign of high mem or cpu 😄

  • U01F7P3D9NH
    U01F7P3D9NH Posts: 60 🧑🏻‍🚀 - Cadet
    Options

    @U01BZ7Q3XRV & @UL65CH0MC Thanks guys for the help. Seems I got trolled by IDE or the Broker. During writing product abstract storage of high variant abstracts (around 5MB per row) it stoped in the middle (7 stores, 6 languages). IDE breakpoint just disappeared without any sign of error, also nothing in the log. Still dont know why, but only with chunksize of 1 the real reason appeared and rabbit container was killed by OOM. With

    "memory_limit": 20480m
    

    and nothing less it is working again and chunks are back to 100. @valerii.trots Could you unleash the EC2 beast for the weekend? 😄

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    Options

    Please create a support request and my colleagues will take care. Unfortunately I'm not as available as was previously due to position change and internal tasks I have.