Docker + Microservices all in one

In the following topic(s) I will give a quick demo on how to use docker and docker-compose to streamline deployment of your application in dev/pre-prod environments with minimal effort.

This topic will continue for 2 to 3 articles till we reach to a point where we can CI/CD the whole application to production.

Keywords of the technologies used:

1. Spring Boot/REST

2. Google Protobuf/Grpc

3. Maven

4. Docker engine

5. Docker compose

6. Grafana + Graphite + StatsD

7. ElasticSearch + Logstash + Kibana (aka ELK)

The application basically consists of 2 microservices one of them (External) exposes 2 REST endpoints (JSON) and the other one (Internal) exposes 2 GRPC endpoints (ProtoBuf) .. the internal one is basically the backend and the external one is the client API ..

The application allows you to test complexity of a password by searching a dictionary of millions of passwords matches that share same prefix.

As part of the application set up there are few containers to monitor the application .. one of them has (Grafana+Graphite+stasD) images and the other one has (ElasticSearch + logstash + Kibana) for logs. (Not covered in this topic)

To get your hands dirty and try out the application yourself please clone the repository from here. (Feel free to contribute in case of issues or new features)

Lets quickly go through the configuration files and service implementation to give you an impression how the application works and services interact to each other.

protobuf-commons:

This module contains protocol buffer files definition .. if you are not familiar with protocol buffers please familiarize yourself with it here. To me its just enough to know that its the model classes definition for our backend service.

// model.proto           
syntax = "proto3";
package com.se7so.model;

option java_multiple_files = true;
option java_package = "com.se7so.model";
option java_outer_classname = "Model";

message FindPasswordsQuery {
    string query = 1;
}

message FindPasswordsResponse {
    int32 numOfMatches = 1;
    repeated string matches = 2;
}

message PasswordsServiceHealthStatus {
    string status = 1;
    int32 totalPasswordsLoaded = 2;
}

// services.proto

syntax = "proto3";
package com.se7so.services;

option java_multiple_files = true;
option java_package = "com.se7so.service";
option java_outer_classname = "Service";

import "google/protobuf/empty.proto";
import "model.proto";

service PasswordsService {
    rpc findPasswords (com.se7so.model.FindPasswordsQuery) returns (com.se7so.model.FindPasswordsResponse) {
    }
}

service PasswordsServiceHealthService {
    rpc getPasswordsServiceHealthStatus (google.protobuf.Empty) returns (com.se7so.model.PasswordsServiceHealthStatus) {
    }
}

grpc-service:

This module has the grpc endpoints implementation (PasswordsService and HealthService) that listen on ports 5000 and 5001.

The PasswordsService accepts FindPasswordsQuery and respond with a FindPasswordsResponse .. both are protobuf messages that are defined in the protobuf-commons .proto files.

The HealthStatusService doesn't have input but gives HealthServiceStatus response which contains information about the service (e.g., Running/Error and number of passwords loaded).

package com.se7so.grpc;

import com.se7so.dict.PasswordDictReader;
import com.se7so.model.FindPasswordsQuery;
import com.se7so.model.FindPasswordsResponse;
import com.se7so.service.PasswordsServiceGrpc;
import io.grpc.Server;
import io.grpc.ServerInterceptors;
import io.grpc.ServerServiceDefinition;
import io.grpc.stub.StreamObserver;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;

import java.util.List;

@RequiredArgsConstructor(onConstructor =  @__(@Autowired))
@Log4j2
public class PasswordsService extends PasswordsServiceGrpc.PasswordsServiceImplBase implements GrpcService {

    @Getter
    @Value("${password.service.grpc.port}")
    private int port;
    @Value("${password.service.max.results}")
    private int maxResults;

    @Setter
    @Getter
    private Server server;
    private final GrpcServerInterceptor interceptor;
    private final PasswordDictReader reader;

    @Override
    public void findPasswords(FindPasswordsQuery request, StreamObserver responseObserver) {
        String prefix = request.getQuery();

        List results = reader.getDict().findPrefixes(prefix);
        int totalMatches = results.size();

        if(totalMatches > maxResults) {
            results = results.subList(0, maxResults);
        }

        responseObserver.onNext(FindPasswordsResponse.newBuilder()
                .addAllMatches(results)
                .setNumOfMatches(totalMatches)
                .build());

        responseObserver.onCompleted();
    }

    @Override
    public ServerServiceDefinition getServiceDefinition() {
        return ServerInterceptors.intercept(bindService(), interceptor);
    }
}
 

package com.se7so.grpc;

import com.google.protobuf.Empty;
import com.se7so.dict.PasswordDictReader;
import com.se7so.dict.PasswordTrie;
import com.se7so.model.FindPasswordsQuery;
import com.se7so.model.FindPasswordsResponse;
import com.se7so.model.PasswordsServiceHealthStatus;
import com.se7so.service.PasswordsServiceGrpc;
import com.se7so.service.PasswordsServiceHealthServiceGrpc;
import io.grpc.Server;
import io.grpc.ServerInterceptors;
import io.grpc.ServerServiceDefinition;
import io.grpc.netty.NettyServerBuilder;
import io.grpc.stub.StreamObserver;
import lombok.Data;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;

import java.io.IOException;

@RequiredArgsConstructor(onConstructor =  @__(@Autowired))
@Log4j2
public class HealthStatusService extends PasswordsServiceHealthServiceGrpc.PasswordsServiceHealthServiceImplBase implements GrpcService {

    @Getter
    @Value("${health.status.grpc.port}")
    private int port;
    @Getter
    @Setter
    private Server server;
    private final GrpcServerInterceptor interceptor;
    private final PasswordDictReader passwordReader;

    @Override
    public void getPasswordsServiceHealthStatus(Empty request, StreamObserver responseObserver) {
        responseObserver.onNext(PasswordsServiceHealthStatus.newBuilder()
                .setStatus(passwordReader.getDict().size() == 0 ? "Error" : "Running")
                .setTotalPasswordsLoaded(passwordReader.getDict().size())
                .build());

        responseObserver.onCompleted();
    }

    @Override
    public ServerServiceDefinition getServiceDefinition() {
        return ServerInterceptors.intercept(bindService(), interceptor);
    }
}

The PasswordDictReader is basically our data store that loads passwords file and store it in memory as a Trie data structure .. it also has methods to search and get matches .. for simplicity you can check its implementation here.

rest-service:

In the rest service module there are REST endpoints that can be called easily from a browser to test the complexity of a password or to check the backend service health status.

Of course the rest-service communicate with the GRPC service by issuing a remote call to one of the services defined there .. and get back the response and map to DTO which is then given back as json to the client.

package com.se7so.rest;

import com.se7so.client.PasswordsServiceClient;
import com.se7so.model.PassServiceHealthDto;
import com.se7so.model.PasswordsResponseDto;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/")
public class RestServiceAPI {

    @Autowired
    private PasswordsServiceClient passwordsServiceClient;

    @RequestMapping(value = "/health", produces = "application/json")
    public PassServiceHealthDto getPasswordServiceHealthDto() {
        return passwordsServiceClient.getHealthStatus();
    }

    @RequestMapping(value = "/passwords", produces = "application/json")
    @ResponseBody
    public PasswordsResponseDto getPasswordsService(@RequestParam(value = "q") String query) {
        return passwordsServiceClient.findPasswordMatches(query);
    }
}
       
 

Docker:

In the grpc-service I use the openjdk:8 as the base image and copy the jar file of the grpc-service module to /home/app.jar also copying the dictionary of passwords to /home.

After copying all the needed files .. issuing a command to start the service.

# grpc-service

FROM openjdk:8

ADD target/grpc-service-1.0-SNAPSHOT.jar /home/app.jar
ADD rockyou.tar.gz /home

CMD java -jar /home/app.jar

The rest-service docker file is doing the same thing .. but no need for the dictionary file in this case ..

# rest-service

FROM openjdk:8

ADD target/rest-service-1.0-SNAPSHOT.jar /home/app.jar

CMD java -jar /home/app.jar

Docker Compose:

And here is the docker-compose file that defines how the services are defined and communicate together to be able to deploy all services and configurations using one command docker-compose up.

#docker-compose.yml

version: '2'
services:
  grpc-service:
    build: ./grpc-service/
    links:
     - graphite
    ports:
     - "5000:5000"
  rest-service:
    build: ./rest-service/
    links:
      - grpc-service
    ports:
      - "80:8080"
  graphite:
    image: "jlachowski/grafana-graphite-statsd"
    ports:
      - "2003:2003"
      - "8082:80"

The compose yml file contains definition of the three containers and exposes ports to the outside world .. also link the rest-service to the grpc-service to be able to call it at runtime.

Run:

Deploy the application using docker-compose:

cd /path/to/dockerized-microservices

mvn install # You will need this only once to build the application and generate binaries

docker-compose build # you will need this only once when you change the application

docker-compose up

Give it sometime to load services ...

 Go to http://localhost/health

Go to http://localhost/passwords?q=123
Go to http://localhost/passwords?q=mypass

Go to http://localhost:8082/dashboard/db/grpc-service-monitor

Clean up:

docker-compose stop # Stops the services without cleaning the containers

docker-compose down # Stops the services and clean up the created containers

Read the next topic in this series Create/Manage docker swarm cluster.

Lagrange multiplier to solve "Mushroom Scientists" problem

Yesterday I was trying to solve this problem on codeforces and found its solution very interesting .. to be honest I had to do alot of research to understand the editorial solution .. so I managed to write a clearer documentation for how to solve these kind of problems.

First of all the problem is that you are trying to maximize f(x,y,z) = x^a Y^b Z^c with the constraint function g(x,y,z) = X+Y+Z = S ..

First of all if (a,b,c) = (0,0,0) then the maximum you get for f(x,y,z) is 1 :) .. otherwise lets solve the problem of at least having one of them > 0.

We can use Lagrange multiplier to solve this problem by doing the following:

=> f(x,y,z) = ƛg(x,y,z) -> Lagrange multiplier

=> x^a . y^b . z^c = ƛ(x+y+z)

by taking derivative d/dx we get:

=> ax^(a-1) . y^b . z^c = ƛ(1+0+0) = ƛ -> (1)

by taking the derivative d/dy we get:

=> bx^a . y^(b-1) . z^c = ƛ(0+1+0) = ƛ -> (2)

by taking the derivative d/dz we get:

=> cx^a . y^b . z^(c-1) = ƛ(0+0+1) = ƛ -> (3)

This implies the following:

=>  ax^(a-1) . y^b . z^c = bx^a . y^(b-1) . z^c = cx^a . y^b . z^(c-1) = ƛ

=> (1) = (2) = (3)

By solving  (1) and (2) together we get:

=> ax^(a-1) . y^b . z^c = bx^a . y^(b-1) . z^c = ƛ

=> ax^(a-1) . y^b = bx^a . y^(b-1)

=> bx = ay

=> a/x = b/y -> (4)

By solving (2) and (3) we get:

=>  bx^a . y^(b-1) . z^c = cx^a . y^b . z^(c-1)

=> bz = cy

=> c/z = b/y -> (5)

From (4) and (5) we get:

a/x = b/y = c/z -> (6)

Now we can use this fact in g(x,y,z) we get:

=> x+y+z = S

=> x+bx/a+cx/a = S

=> ax+bx+cx = Sa

=> x = Sa/(a+b+c) -> (7)

Doing the same for y and z you get:

=> y = Sb/(a+b+c)

=> z = Sc/(a+b+c)

Now the problem is solved and here is my code:

Web Test Automation

Most of tech companies now realise that manual testing is not as effective as anticipated and wanted to automate the whole process of users going to their websites performing some actions, submitting a form, or triggering a search,... etc.

You have probably thought many times of automating some repetitive tasks that you do every day or sometimes multiple times per day. Some people automate making their morning coffee - Believe me :).

Testing is one of the repetitive tasks that developers tend to do very often everyday .. Why? because it's important to make sure that things are working as expected before pushing them to production.

In this post I will not talk about the theory of automation or the frameworks themselves as there are plenty of articles for this purpose .. This article is just to introduce you to the world of automation and point you towards the next step.

Links to read

 

Test Automation

Selenium Framework

Phantom JS
Selenium headless mode

Steps to run this example

• Create a maven project and copy these files
• Make sure you download chrome driver from here
• Edit the path to your chrome driver installation
• Run the Example.java class

Example

In this example I will use Selenium framework to test Amazon DE website and search for something there .. then print the search page title .. No assertions/checks will be done as it depends on your business and the purpose of your test.

Pom.xml that defines libraries used:



Cheers ;)