Create/Manage docker swarm cluster

Continuing the effort of previous post now I will show you how to create and manage a docker swarm cluster using docker-machine and docker-swarm

In docker swarm you create one or more managers and worker machines in the cluster .. the manager(s) take care of the orchestration of your deployed services (e.g., Creation/Replication/Assigning tasks to nodes/load balancing/service discovery ...).

Step 1: Create cluster machines using docker-machine

docker-machine create --driver virtualbox --virtualbox-memory "3000" master
docker-machine create --driver virtualbox --virtualbox-memory "3000" worker1
docker-machine create --driver virtualbox --virtualbox-memory "3000" worker2

In the above I've created 3 machines called (master, worker1, and worker2) with the same configuration and as I'll create the cluster on my local machine I've created them as virtual machines with docker engine installed on all of them. later we will initialize them as a swarm cluster.

Note: Docker machine can also allow you create these machines on digital ocean, AWS, and other third party cloud services .. they have other drivers .. please check docker-machine manual pages for more details.

You can also run commands on these machines without having to connect to them.

List all machines configured so far:

se7so@se7so:~$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS
master    *        virtualbox   Running   tcp://192.168.99.100:2376           v17.04.0-ce   
worker1   -        virtualbox   Running   tcp://192.168.99.102:2376           Unknown       Unable to query docker version: Get https://192.168.99.102:2376/v1.15/version: x509: certificate is valid for 192.168.99.101, not 192.168.99.102
worker2   -        virtualbox   Running   tcp://192.168.99.101:2376           Unknown       Unable to query docker version: Get https://192.168.99.101:2376/v1.15/version: x509: certificate is valid for 192.168.99.102, not 192.168.99.101

Show machine IP:

se7so@se7so:~$ docker-machine ip master
192.168.99.100
se7so@se7so:~$ docker-machine ip worker1
192.168.99.102
se7so@se7so:~$ docker-machine ip worker2
192.168.99.101    

Step 2: Set up environment to run commands on master

eval "$(docker-machine env master)

Step 3: Init the swarm

You have to run this command on a manager machine.

docker swarm init --advertise-addr MASTER_MACHINE_IP

Note: This command will give you as output 2 commands one of them in order to join a machine as worker and the other one if you need to join a machine as a manager to the cluster swarm.

Step 4: Configure the worker machines to join the swarm

The below swarm join command is always generated after init swarm command above .. this will include the swarm cluster token you see below.

In order to run on both workers I will have to SSH to each one run the command then exit.

docker-machine ssh worker1
docker swarm join \
    --token SWMTKN-1-5ymh4597gc11bqq2keldy951fmwsqr4z8wjjcp47v5m43sv8qp-cbml91zj5xfilfi0syw1dl2o4 \
    192.168.99.100:2377
exit

docker-machine ssh worker2
docker swarm join \
    --token SWMTKN-1-5ymh4597gc11bqq2keldy951fmwsqr4z8wjjcp47v5m43sv8qp-cbml91zj5xfilfi0syw1dl2o4 \
    192.168.99.100:2377
exit

Step 5: Display all cluster nodes configured

se7so@se7so:~$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
govkclp1jeoy56e5kzwksd0z2 *  master    Ready   Active        Leader
lrl9lydrq7om79pp3aty1pea3    worker2   Ready   Active        
rlrt6252a1rjkzqdbig4a70bp    worker1   Ready   Active

Notice the master is configured as leader and the * is because we are currently connected to it.

Now I've configured a cluster with 1 manager and 2 workers .. lets run our rest-service and grpc-service from our previous post..

 

Step 6: Create a network to make our services visible to each other

docker network create -d overlay my_network

Step 7: Create services and publish its ports

For simplicity I've pushed both images to my docker hub account rest-service and grpc-service and now I will pull/run them on the cluster.

I've pushed them using the following commands:

docker tag rest-service husseincoder/rest-service
docker push husseincoder/rest-service
docker tag grpc-service husseincoder/grpc-service
docker push husseincoder/grpc-service

docker service create -p 80:8080 --name rest-service --network my_network husseincoder/rest-service 
docker service create -p 5000:5000 -p 5001:5001 --name grpc-service --network my_network husseincoder/grpc-service

Here I've just deployed the services with 1 replicas .. however I could have used --replicas parameter to set the number of replicas of the service.

You can also see that I've exposed the ports and also specified the network to make sure they can access each other using the service name

 

Step 8: Display all services

se7so@se7so:~$ docker service ls
ID            NAME          MODE        REPLICAS  IMAGE
ierr7y08cidg  rest-service  replicated  1/1       husseincoder/rest-service:latest
mhc7ffeqv01j  grpc-service  replicated  1/1       husseincoder/grpc-service:latest

This will display the services and show you many of the replicas has been started and how many are still being prepared .. it should show something like this:

 

Step 9: Display tasks of a service

se7so@se7so:~$ docker service ps rest-service
ID            NAME            IMAGE                             NODE     DESIRED STATE  CURRENT STATE           ERROR  PORTS
nc8m3zwjwivx  rest-service.2  husseincoder/rest-service:latest  worker2  Running        Running 13 minutes ago

In my case the manager decided to run it on the worker2 node .. it could be different in your case ;).

 

Step 10: Scale a service

As I've mentioned before I could have set the number of replicas of a service at creation time .. now since I've already started it lets scale one of them to run 5 replicas.

se7so@se7so:~$ docker service scale rest-service=3
rest-service scaled to 3

se7so@se7so:~$ docker service ls
ID            NAME          MODE        REPLICAS  IMAGE
ierr7y08cidg  rest-service  replicated  3/3       husseincoder/rest-service:latest
mhc7ffeqv01j  grpc-service  replicated  1/1       husseincoder/grpc-service:latest

se7so@se7so:~$ docker service ps rest-service
ID            NAME            IMAGE                             NODE     DESIRED STATE  CURRENT STATE           ERROR  PORTS
ezb2p3qo0tp2  rest-service.1  husseincoder/rest-service:latest  worker1  Running        Running 35 seconds ago         
nc8m3zwjwivx  rest-service.2  husseincoder/rest-service:latest  worker2  Running        Running 16 minutes ago         
uye5mlquy6l2  rest-service.3  husseincoder/rest-service:latest  master   Running        Running 35 seconds ago

Of course you can either scale up or down.

 

Step 11: Let's try calling the service

You can actually specify any of the cluster machine IPs not necessarily the manager.





Now you can start playing with docker-machine and docker-swarm and manage your own cluster .. Please share and go to the next topic - Kubernetes basics with demo.