I use Oracle VirtuaBox because I’m a big fan of Oracle products, especially when they are good and free. However, you can use Hyper-V (then just skip this paragraph and use –driver=hyperv<\/em> instead of –driver=virtualbox<\/em>)<\/p>\n I have VirtualBox installed. You can install it for free https:\/\/www.virtualbox.org\/wiki\/Downloads<\/a><\/p>\n Virtualization must be enabled in the BIOS:<\/p>\n Even if this is the Hyper-V requirements, they are the same for VirtualBox and hypervisorlaunchtype<\/em> at off<\/em> allows VirtualBox.<\/p>\n I’ll use Chocolatey software manager to install Minikube, here is how to install it from a PowerShell window:<\/p>\n (run this on a PowerShell window as Administrator)<\/p>\n Minikube is a tool to run a single node Kubernetes on laptop:<\/p>\n (run this on a ‘DOS’ window as Administrator)<\/p>\n The minikube command will be used to interact with the VM (start, stop, tunnel…), and the kubectl to interact with kubernetes.<\/p>\n Helm is an open source package tool for kubernetes to install applications.<\/p>\n (run this on a ‘DOS’ window as Administrator)<\/p>\n I’ll use Helm to install the package I want to run. I work with databases, and I’ll install one for this demo. Even if Kubernetes is initially designed to scale stateless applications, it has evolved with stateful sets, and persistent storage. However, it may not be the right platform for a monolith database. Then, I’ll run YugabyteDB<\/strong>, an open-source cloud-native distributed database, which makes more sense in an environment that can scale by adding pods, and where replication maintains the availability when a node goes down, or is added.<\/p>\n The installation on a local cluster is documented: This creates a cluster with no replication (replication factor 1) which is ok for a local lab on a laptop.<\/p>\n I’ll do something a bit different, trying RF3 (the replication factor should be an odd number as there’s a quorum election for the leader). This means more pods, and I’ll limit the memory and CPU so that it can run on any laptop. I’m doing this on my Surface Pro which has 2 cores, 4 Intel threads, and 16GB RAM).<\/p>\n First add the Yugabyte Helm charts repository<\/p>\n Here is the current version:<\/p>\n You can download the chart if you want to have a look at it (helm repo update & helm fetch yugabytedb\/yugabyte)<\/p>\n Now I can start the Minikube node:<\/p>\n This creates and starts a VirtualBox VM. It seems that whatever the –cpu I pass, it creates a 2 vCPU VM which is not sufficient for a RF3 cluster (I need 3 yb-master and at least 3 yb-tserver).<\/p>\n I’ll update the settings with VBoxManage to set to 4 vCPU (the maximum threads on my Surface Pro) but capping them at 80% to leave some capacity to for host OS.<\/p>\n Of course you can do that with the VirtualBox GUI.<\/p>\n I create a namespace for my Yugabyte universe:<\/p>\n Checking them:<\/p>\n All that can also visible from the web console that is started with: I can start a YugabyteDB universe on a Kubernetes cluster with a simple Here it is with my settings:<\/p>\n I’ve defined storage.ephemeral=true here and that’s probably not what you want for a database in general. But, first, this is a lab on my laptop, running in a VirtualBox VM that I can snapshot. And I use a database that is replicated (replication factor 3) and then one node, out of 3, can be lost, storage included. And a new node added later, with no data loss, as the replicas will be populated from the other nodes.<\/p>\n If you want no replication, less servers, but persistent storage, just change the last 3 settings with replicas.master=1,replicas.tserver=1,storage.ephemeral=false<\/p>\n With low resource this may take a few seconds:<\/p>\n The stateful sets take care of the order (masters are started before tservers) and the network addresses (which must be known before they are created).<\/p>\n You can ssh to the VM to check what is running:<\/p>\n (The tty columns was not set correctly) but all is also accessible from the web console of kubernetes with `minikube dashboard` and choosing the right namespace.<\/p>\n When you want to cleanup all this you can:<\/p>\n the last command even deletes the VirtualBox VM.<\/p>\n But before cleaning it up I check that I can connect to my distributed database:<\/p>\n This runs the YSQLSH command line for YugabyteDB SQL API through kubernetes container exec.<\/p>\n But, as this SQL API is fully compatible with PostgreSQL I can just forward the YSQL port from one of the table servers:<\/p>\n With this I can connect with psql, DBeaver, or any PostgreSQL client as YugabyteDB is fully postgres-compatible:<\/p>\n You see the version, this YugabyteDB version 2.5.3 is compatible with PostgreSQL 11.2, and I have created a demo table sharded over 3 tablets, with 1000 rows.<\/p>\n I can check the tablets from the web console, exposed by the yb-master-ui service. Here are all exposed services:<\/p>\n There’s no external IP in Minikube but it is easy to tunnel the UI (port 7000 of the master):<\/p>\n Then I can access the YugabyteDB GUI on http:\/\/localhost:7000<\/a> and this is where the following screenshots come from.<\/p>\n The big advantage of Kubenetes is that a node can be killed, another created, scaling down and up the cluster. That’s possible with a stateless application, but also with a database, when it is distributed shared-nothing one. And it stays available thanks to a replication factor higher than one.<\/p>\n All 3 nodes are up: The yb-tserver-2 is considered temporarily unavailable, still with 3 tablets but no leaders: After a minute, the node is definitely considered as dead: Now I scale up to 3 nodes again:<\/p>\n The node is detected and rebalancing starts: As I defined my minikube VM with only 4 vCPUs I canot scale-up the yb-server to one additional pod. But, and please remember this is a lab, I can kill one of the yb-master to leave room for one additional pod:<\/p>\n The failure of one master is detected, but thanks to RF3 the system is still available: I used non-persistent storage for two reasons. First, Minikube has some limitations when compared with a real Kubernetes cluster, and I’m not sure how I can run a multi-node RF3 cluster with persistent volumes and easily scale it up and down. But the main reason is that this is a lab, running on VirtualBox, and I can save the state at this VBox level.<\/p>\n Here is how I stop the VirtualBox VM keeping the disc + memory state, and start it up again:<\/p>\n No data was loss here, my demo table is still there because, from the Kubernetes perspectives, the pods were not stopped, just paused by the hypervisor.<\/p>\n In summary: you can play, on a 2 cores laptop, with scaling up and down a distributed database running its nodes on kubernetes pods. Feel free to follow me and discuss on Twitter about this. Databases in microservices and stateless containers is hot topic today.<\/p>\n During the week-end, I played with #kubernetes<\/a> on my laptop and here is a blog post:https:\/\/t.co\/SIz5tUiKeM<\/a> – Not a "Hello World" example. A real RDBMS scaling on containers #minikube<\/a> #k8s<\/a> @virtualbox<\/a> @Windows<\/a> @Yugabyte<\/a><\/p>\nMicrosoft Windows [Version 10.0.19042.906]\n(c) Microsoft Corporation. All rights reserved.\n\nC:\\Users\\Franck> systeminfo\n\n...\nHyper-V Requirements: VM Monitor Mode Extensions: Yes\n Virtualization Enabled In Firmware: Yes\n Second Level Address Translation: Yes\n Data Execution Prevention Available: Yes\n\nC:\\WINDOWS\\system32>bcdedit\n\n...\nhypervisorlaunchtype Off\n<\/code><\/pre>\n <\/a> Install Chocolatey<\/h3>\n
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https:\/\/chocolatey.org\/install.ps1'))\n<\/code><\/pre>\nMicrosoft Windows [Version 10.0.19042.906]\n(c) Microsoft Corporation. All rights reserved.\n\nC:\\Users\\Franck>choco\nChocolatey v0.10.15\nPlease run 'choco -?' or 'choco <command> -?' for help menu.\n\nC:\\Users\\Franck>\n<\/code><\/pre>\n <\/a> Install Minikube<\/h3>\n
choco install minikube\n<\/code><\/pre>\n <\/a> Install Helm<\/h3>\n
choco install kubernetes-helm\n<\/code><\/pre>\n <\/a> Testing with a database: YugabyteDB<\/h3>\n
\nhttps:\/\/docs.yugabyte.com\/latest\/quick-start\/create-local-cluster\/kubernetes\/<\/a><\/p>\nhelm repo add yugabytedb https:\/\/charts.yugabyte.com\n<\/code><\/pre>\n\nC:\\Users\\Franck> helm repo update\nC:\\Users\\Franck> helm search repo yugabytedb\/yugabyte\n\nNAME CHART VERSION APP VERSION DESCRIPTION\nyugabytedb\/yugabyte 2.5.3 2.5.3.1-b10 YugabyteDB is the high-performance distributed ...\n<\/code><\/pre>\n <\/a> Start<\/h3>\n
minikube start --driver=virtualbox --cpus=4 --memory 8192 --disk-size 30g \n<\/code><\/pre>\nminikube stop\n\n\"C:\\Program Files\\Oracle\\VirtualBox\\VBoxManage\" modifyvm minikube --cpus 4 --cpuexecutioncap 80\n\nminikube start\n<\/code><\/pre>\n <\/a> Create the distributed DB<\/h3>\n
kubectl create namespace franck-yb\n<\/code><\/pre>\nC:\\Users\\Franck> kubectl get namespaces\n\nNAME STATUS AGE\ndefault Active 7m54s\nfranck-yb Active 24s\nkube-node-lease Active 7m55s\nkube-public Active 7m55s\nkube-system Active 7m55s\nkubernetes-dashboard Active 9s\n<\/code><\/pre>\n
\nminikube dashboard<\/code> as this opens a localhost port to display the kubernetes dashboard.<\/p>\nhelm install yugabyte yugabytedb\/yugabyte --namespace franck-yb<\/code> but the defaults are too large for my laptop:<\/p>\n\n
helm install yugabyte yugabytedb\/yugabyte --namespace franck-yb --set resource.master.requests.cpu=0.5,resource.master.requests.memory=1Gi,storage.master.count=1,storage.master.size=2Gi,resource.tserver.requests.cpu=0.5,resource.tserver.requests.memory=1Gi,storage.tserver.count=1,storage.tserver.size=2Gi,replicas.master=3,replicas.tserver=3,storage.ephemeral=true\n<\/code><\/pre>\nC:\\Users\\Franck> kubectl --namespace franck-yb get pods\n\nNAME READY STATUS RESTARTS AGE\nyb-master-0 0\/1 ContainerCreating 0 1m1s\nyb-master-1 0\/1 ContainerCreating 0 1m1s\nyb-master-2 0\/1 ContainerCreating 0 1m\nyb-tserver-0 0\/1 ContainerCreating 0 1m1s\nyb-tserver-1 0\/1 ContainerCreating 0 1m1s\nyb-tserver-2 0\/1 ContainerCreating 0 1m1s\n<\/code><\/pre>\nminikube ssh\nstty columns 120\n<\/code><\/pre>\nhelm delete yugabyte -n franck-yb\nminikube delete\n<\/code><\/pre>\nkubectl exec --namespace franck-yb -it yb-tserver-0 -- \/home\/yugabyte\/bin\/ysqlsh -h yb-tserver-0.yb-tservers.franck-yb\n<\/code><\/pre>\nkubectl --namespace franck-yb port-forward svc\/yb-tservers 5433:5433\n<\/code><\/pre>\nC:\\Users\\fpa> psql -h localhost -p 5433 -U yugabyte\n\npsql (12.6, server 11.2-YB-2.5.3.1-b0)\nType \"help\" for help.\n\nyugabyte=# create table demo ( k int primary key, v int ) split into 3 tablets;\nCREATE TABLE\nyugabyte=# insert into demo select generate_series,42 from generate_series(1,1000);\nINSERT 0 1000\nyugabyte=# select count(*) from demo;\n count\n------------\n 1000\n(1 row)\n<\/code><\/pre>\nC:\\Users\\Franck> kubectl --namespace franck-yb get services\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nyb-master-ui LoadBalancer 10.105.151.9 <pending> 7000:31447\/TCP 7m42s\nyb-masters ClusterIP None <none> 7000\/TCP,7100\/TCP 7m42s\nyb-tserver-service LoadBalancer 10.100.217.206 <pending> 6379:31842\/TCP,9042:32404\/TCP,5433:31626\/TCP 7m42s\nyb-tservers ClusterIP None <none> 9000\/TCP,12000\/TCP,11000\/TCP,13000\/TCP,9100\/TCP,6379\/TCP,9042\/TCP,5433\/TCP 7m42s\n<\/code><\/pre>\nkubectl --namespace franck-yb port-forward svc\/yb-master-ui 7000:7000\n<\/code><\/pre>\n <\/a> scale-down \/ scale-up<\/h3>\n
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-114151-all-nodes-up.jpg\")
<\/a>
\nMy table is hash-sharded into 3 tablets, with one leader on each node and followers on 2 other nodes:
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-114304-distributed-leaders.jpg\")
<\/a>
\nI scale down the tablet server stateful-set to 2 nodes:<\/p>\nC:\\Users\\Franck> kubectl scale -n franck-yb statefulset yb-tserver --replicas=2\nstatefulset.apps\/yb-tserver scaled\n<\/code><\/pre>\n
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-114410-node-down-new-leader-election.jpg\")
<\/a>
\nThe follower that was in yb-tserver-0 has been elected as the new leader for this tablet:
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-114520-temporarily-scaled-down.jpg\")
<\/a>
\nAt that point, my demo table is still fully available. And each of the remaining node have a full copy of the data.<\/p>\n
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-114626-node-is-dead.jpg\")
<\/a><\/p>\nC:\\Users\\Franck> kubectl scale -n franck-yb statefulset yb-tserver --replicas=3\nstatefulset.apps\/yb-tserver scaled\n<\/code><\/pre>\n
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-114713-scaled-up.jpg\")
<\/a>
\nThe tablet leaders distribution is quickly re-balanced to the 3 pods:
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-114844-back-to-3-nodes.jpg\")
<\/a>
\nWith 1000 rows this is very fast of course.
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-120420-rebalanced.jpg\")
<\/a><\/p>\nkubectl scale -n franck-yb statefulset yb-master --replicas=2\nkubectl scale -n franck-yb statefulset yb-tserver --replicas=4\n<\/code><\/pre>\n
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-124417.jpg\")
<\/a>
\nThe new node has taken over some of the followers:
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-124546.jpg\")
<\/a>
\nThere is no need to elect a leader in this new node:
\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-124800.jpg\")
<\/a>
\nOf course, if I create a new table it will be sharded with leaders on all nodes:<\/p>\nyugabyte=# create table demo2 as select * from demo;\nSELECT 1000\n<\/code><\/pre>\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2021-04-12-125828.jpg\")
<\/a>
\nThere are 32 tablets for this table because I didn’t specify the SPLIT INTO clause and the default is 8 per server (–ysql_num_shards_per_tserver=8<\/i>) here.<\/p>\n <\/a> ephemeral storage?<\/h3>\n
C:\\Users\\fpa> \"C:\\Program Files\\Oracle\\VirtualBox\\VBoxManage\" controlvm minikube savestate\n0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%\n\nC:\\Users\\fpa> \"C:\\Program Files\\Oracle\\VirtualBox\\VBoxManage\" startvm minikube --type headless\nWaiting for VM \"minikube\" to power on...\nVM \"minikube\" has been successfully started.\n<\/code><\/pre>\n\n