Prisma implementation in production


#1

Hey Guys, I have been working with prisma for a while. And now I am planning on pushing my service in production and would like to know what is the best practice to maintain redundancy for my prisma server. I have been thinking of creating multiple server and putting it behind the load balancer but when I do that prisma deploy takes time to reflect the changes on all nodes. I wonder if there is any way to handle such scenarios. Any help would be appreciated. Thanks!


#2

Hi @mpandey

Please refer to this issue for a guide: https://github.com/prisma/prisma/issues/2850

There is also a nice blog post on it: https://techblog.commercetools.com/prisma-horizontal-scaling-a-practical-guide-3a05833d4fc3


#3

My team has been trying to setup a multi node prisma production server. We referred this link: https://github.com/prisma/prisma/issues/2850(given by you), tried to do exactly what is mentioned in the post. Even after successful setup, we are getting some random errors for schema deployment. The majority of the time it complained that it couldn’t reach the server, which was odd considering that we were watching the proxy logs which were returning 200 responses for POST requests to the /management endpoint of the primary node.
Other times it complained about the cluster$default project not existing, and those were accompanied by 200 responses for POST reqs to the /cluster endpoint. And few times, it did worked. Knowing the fact that, we are relatively new to Prisma and multi node setup, we could be missing something very minute which is causing all the trouble. It would be to get some help on this. Thanks!


#4

I think your issue is related to a recent bug that we fixed.

Please try again with 1.27.3


#5

Hey Harshit,
We tried to updating the version but still no luck. Can you help us in this?


#6

I tried setting up prisma server locally with prisma-prod (version 1.27.3 and 1.28.0) to reproduce the issue locally. But I am still getting following error Error: Could not connect to server at http://localhost:4466. Please check if your server is running. for prisma deploy. And the weird thing is I am not seeing any error in the prisma container logs.

Below are my configuration for prisma, rabbitmq with mysql database:

  • Docker compose file

I am also not able access the management api in the playground getting Project not found: 'management@default.

I am not sure where I am going wrong or is there any different setting which needs to be configured. Any help would be appreciated. Thanks!


#7

Hi @mpandey

If you are running prisma without kubernetes please do not use prisma prod image. It is not recommended for horizontally scaled environments.

Please use the default image from our docs, it will work quite well in production.


#8

Hi Harshit. I’m one of Madhav’s team mates and I have been able to succesfully reproduce this problem in a local kubernetes cluster (using Docker Desktop). I’ll include the reproduction steps as well as a link to a zip file that has the prisma config.

Reproduction steps

  1. Set up kubernetes cluster
  2. Download kubernetes-prisma.zip and extract files: https://drive.google.com/file/d/18wFzxvVg-B8E7tUmUC0FKGx2ZedIyiTR/view?usp=sharing
  3. From extracted directory, run:
    kubectl apply -f 1-prisma-deploy-postgres.yml
    kubectl apply -f 2-prisma-deploy-rabbitmq
    kubectl apply -f 3-prisma-deploy-prisma-mgmt
    kubectl apply -f 4-prisma-deploy-prisma.yml
    Notes: All the kubernetes config could be grouped into one file, but for easier (human) parsing, I’ve included 4 different files for each part of the system: postgres db, rabbitmq, prisma management, and prisma (non-management)
  4. Run :
    kubectl get pods --namespace=prisma-mw
  5. Verify all pods are up. Copy name of the prisma-mgmt pod.
  6. Run:
    kubectl port-forward 4467:4466 --namespace=prisma-mw
  7. From an prisma project with the prisma client/cli (v 1.28.1) installed, create a test.env file with these variables:
    PRISMA_ENDPOINT=http://localhost:4467/test/dev
    PRISMA_MANAGEMENT_API_SECRET=mysecret42
  8. From the same project mentioned, run:
    prisma deploy -e test.env
    9.Notice output. I see this (with debug enabled):

environment TypeError: Cannot read property ‘serverInfo’ of undefined
environment at Cluster. (C:\Users\tester\AppData\Roaming\nvm\v10.14.2\node_modules\prisma\node_modules\prisma-yml\src\Cluster.ts:227:19)
environment at step (C:\Users\tester\AppData\Roaming\nvm\v10.14.2\node_modules\prisma\node_modules\prisma-yml\dist\Cluster.js:32:23)
environment at Object.next (C:\Users\tester\AppData\Roaming\nvm\v10.14.2\node_modules\prisma\node_modules\prisma-yml\dist\Cluster.js:13:53)
environment at fulfilled (C:\Users\tester\AppData\Roaming\nvm\v10.14.2\node_modules\prisma\node_modules\prisma-yml\dist\Cluster.js:4:58)
environment at process._tickCallback (internal/process/next_tick.js:68:7) +0ms
Error: Could not connect to server at http://localhost:4467. Please check if your server is running.
at Deploy. (C:\Users\tester\AppData\Roaming\nvm\v10.14.2\node_modules\prisma\node_modules\prisma-cli-core\src\commands\deploy\deploy.ts:127:13)
at step (C:\Users\tester\AppData\Roaming\nvm\v10.14.2\node_modules\prisma\node_modules\prisma-cli-core\dist\commands\deploy\deploy.js:45:23)
at Object.next (C:\Users\tester\AppData\Roaming\nvm\v10.14.2\node_modules\prisma\node_modules\prisma-cli-core\dist\commands\deploy\deploy.js:26:53)
at fulfilled (C:\Users\tester\AppData\Roaming\nvm\v10.14.2\node_modules\prisma\node_modules\prisma-cli-core\dist\commands\deploy\deploy.js:17:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
Exiting with code: 1

Note 1: Steps 7 and 8 can be done in different ways using the prisma cli, but it all results in the same issue
Note 2: I can reproduce this outside of kubernetes (using docker-compose) and I see this same issue.

Please let us know what we can do. I feel like we are missing something here…or there could possibly be a bug.

Thanks!


#9

@mwest1

Can you please open this as an github issue so that this doesn’t get lost. I was unable to reproduce this in GCP but I will try a local reproduction soon.


#10

@pantharshit00
Yes. Will do. Thanks.


#11

Ugh. I just found something frustrating while diving into the prisma source code. :upside_down_face:
We were using the config property ‘managementApiEnabled’ , when it should have been:
‘enableManagementApi’.

I’m not sure where went wrong with getting the incorrect property name (there are some posts in github with that wrong name as well…maybe it was changed somewhat recently?), but changing that makes things seemingly work - at least I’m able to access the management api.


#12

Looks like this is the reason why I was unable to reproduce this