Calling Developers!
We are reenergizing our code contribution process! Learn More

What are the Slack Archives?

It’s a history of our time together in the Slack Community! There’s a ton of knowledge in here, so feel free to search through the archives for a possible answer to your question.

Because this space is not active, you won’t be able to create a new post or comment here. If you have a question or want to start a discussion about something, head over to our categories and pick one to post in! You can always refer back to a post from Slack Archives if needed; just copy the link to use it as a reference..

Hello everyone, because we have so much data that would need to be pushed to the Storage-Redis, we

UQ958DA4U
UQ958DA4U Posts: 232 🧑🏻‍🚀 - Cadet

Hello everyone,

because we have so much data that would need to be pushed to the Storage-Redis, we opted to switch it out for a direct database connection as outlined here: https://documentation.spryker.com/docs/howto-replace-key-value-storage-with-database
Now we've got the problem that we need a reliable way to set this up with docker/sdk. Is there any way to pass new environment variables to the Yves container without manually patching the generator scripts?

Thanks in advance.

Comments

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet

    how much data do u have?

  • UQ958DA4U
    UQ958DA4U Posts: 232 🧑🏻‍🚀 - Cadet

    Enough that Redis is not a viable solution any longer. We are talking milions of products.

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet

    Ok. we have 26GB of data in redis and it works well..

    We have ca. 3Mio Product Abstract, 3Mio Product Concrete, 5Mio Product Images, 3Mio Price, 3Mio Product Sales Unit, 3Mio Product Quantity, a lot of Mio Product Stocks + Availability, 1000 Categories (only 1 Tree) etc etc etc.. etc...... and without Redis the System will be very slow...

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet

    if u have 100Mio products, maybe can (a little bit) understand your question.. even if u can scale up with Storage CTE and more worker parallel the P+S ...

  • UQ958DA4U
    UQ958DA4U Posts: 232 🧑🏻‍🚀 - Cadet

    We are simply at a point where Redis cannibalizes all our cluster resources for itself due to data duplication. 30GB would not be enough and we are growing.

  • UQ958DA4U
    UQ958DA4U Posts: 232 🧑🏻‍🚀 - Cadet

    Also, I don't see how a SELECT on an indexed table would be much slower than a KV access with a properly indexed and sized database. The idea of decoupling by default is nice, but we've clearly found a limit here. We were also advised by Spryker to consider this approach if Redis starts exploding.

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet

    ok, of course i don't know your data size and other aspects... I think that request to redis are faster than request to zed and forward to db

    Imagine u have Yves Server/Instance that for every glossary or storage request have to call the zed (because only zed can communicate with db) and then zed server/Instance have to make another request (in best case) to another server where db is running... i think that the result can be that the system is a little bit slower (maybe)...

    Other side u save up a lot of rmq messages, php process for syncing etc.. and your data are sure synced always...

  • UQ958DA4U
    UQ958DA4U Posts: 232 🧑🏻‍🚀 - Cadet

    The way the tutorial sets it up, the Storage driver is exchanged, so no Zed calls are made. The Yves frontend connects directly to the database. I agree that constant REST requests would be an overwhelming overhead.

  • UQ958DA4U
    UQ958DA4U Posts: 232 🧑🏻‍🚀 - Cadet

    also, we're still using the publish process but just not the sync process that pushes the data to Redis. So we're still working with denormalized data, it's just not read from Redis but from the _storage tables in the DB instead.

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet

    yes i see...
    i did not know that

    StorageDatabase
    

    have a self connectionProvider to communicate via client layer with db...

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet

    then would be an acceptable solution...

  • UQ958DA4U
    UQ958DA4U Posts: 232 🧑🏻‍🚀 - Cadet

    that's really all besides my original question unfortunately 😉
    if we only did SQL queries through Zed, we wouldn't have the issue mentioned in the post as that of course already gets the credentials passed in via environment by docker/sdk.

  • giovanni.piemontese
    giovanni.piemontese Technical Lead @ Löffelhardt Spryker Solution Partner Posts: 871 🧑🏻‍🚀 - Cadet

    I think u have include the database env located here docker/generator/src/templates/env/database (i.e. for zed here -> docker/generator/src/templates/env/application/zed.env.twig:25 ) in docker/generator/src/templates/env/application/yves.env.twig too... and then in docker/generator/src/templates/application/yves.yml.twig u have to extends the depends_on with database too..

  • sprymiker
    sprymiker Cloud Platform Architect Sprykee Posts: 781 🧑🏻‍🚀 - Cadet

    Locally it can just use the same database credentials from primary database.
    You can still separate dev and cloud config on PHP side to take different environment variables.

  • sprymiker
    sprymiker Cloud Platform Architect Sprykee Posts: 781 🧑🏻‍🚀 - Cadet

    Aha… So you problem is that Yves/Glue does not have database env variables at all.

    In this case I would suggest to use this functionality:
    https://github.com/spryker/docker-sdk/blob/master/docs/99-deploy.file.reference.v1.md#image-environment

    image:
      ...
      environment:
            STORAGE_DATABASE_HOST: database
            STORAGE_DATABASE_USER: spryker
            STORAGE_DATABASE_PASSWORD: secret
    

    But only for dev deploy.yml

  • UQ958DA4U
    UQ958DA4U Posts: 232 🧑🏻‍🚀 - Cadet

    thanks