In the following topic(s) I will give a quick demo on how to use docker and docker-compose to streamline deployment of your application in dev/pre-prod environments with minimal effort.
This topic will continue for 2 to 3 articles till we reach to a point where we can CI/CD the whole application to production.
Keywords of the technologies used:
1. Spring Boot/REST
2. Google Protobuf/Grpc
3. Maven
4. Docker engine
5. Docker compose
6. Grafana + Graphite + StatsD
7. ElasticSearch + Logstash + Kibana (aka ELK)
The application basically consists of 2 microservices one of them (External) exposes 2 REST endpoints (JSON) and the other one (Internal) exposes 2 GRPC endpoints (ProtoBuf) .. the internal one is basically the backend and the external one is the client API ..
The application allows you to test complexity of a password by searching a dictionary of millions of passwords matches that share same prefix.
As part of the application set up there are few containers to monitor the application .. one of them has (Grafana+Graphite+stasD) images and the other one has (ElasticSearch + logstash + Kibana) for logs. (Not covered in this topic)
To get your hands dirty and try out the application yourself please clone the repository from here. (Feel free to contribute in case of issues or new features)
Lets quickly go through the configuration files and service implementation to give you an impression how the application works and services interact to each other.
protobuf-commons:
This module contains protocol buffer files definition .. if you are not familiar with protocol buffers please familiarize yourself with it here. To me its just enough to know that its the model classes definition for our backend service.
// model.proto syntax = "proto3"; package com.se7so.model; option java_multiple_files = true; option java_package = "com.se7so.model"; option java_outer_classname = "Model"; message FindPasswordsQuery { string query = 1; } message FindPasswordsResponse { int32 numOfMatches = 1; repeated string matches = 2; } message PasswordsServiceHealthStatus { string status = 1; int32 totalPasswordsLoaded = 2; }
// services.proto
syntax = "proto3";
package com.se7so.services;
option java_multiple_files = true;
option java_package = "com.se7so.service";
option java_outer_classname = "Service";
import "google/protobuf/empty.proto";
import "model.proto";
service PasswordsService {
rpc findPasswords (com.se7so.model.FindPasswordsQuery) returns (com.se7so.model.FindPasswordsResponse) {
}
}
service PasswordsServiceHealthService {
rpc getPasswordsServiceHealthStatus (google.protobuf.Empty) returns (com.se7so.model.PasswordsServiceHealthStatus) {
}
}
grpc-service:
This module has the grpc endpoints implementation (PasswordsService and HealthService) that listen on ports 5000 and 5001.
The PasswordsService accepts FindPasswordsQuery and respond with a FindPasswordsResponse .. both are protobuf messages that are defined in the protobuf-commons .proto files.
The HealthStatusService doesn't have input but gives HealthServiceStatus response which contains information about the service (e.g., Running/Error and number of passwords loaded).
package com.se7so.grpc; import com.se7so.dict.PasswordDictReader; import com.se7so.model.FindPasswordsQuery; import com.se7so.model.FindPasswordsResponse; import com.se7so.service.PasswordsServiceGrpc; import io.grpc.Server; import io.grpc.ServerInterceptors; import io.grpc.ServerServiceDefinition; import io.grpc.stub.StreamObserver; import lombok.Getter; import lombok.RequiredArgsConstructor; import lombok.Setter; import lombok.extern.log4j.Log4j2; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import java.util.List; @RequiredArgsConstructor(onConstructor = @__(@Autowired)) @Log4j2 public class PasswordsService extends PasswordsServiceGrpc.PasswordsServiceImplBase implements GrpcService { @Getter @Value("${password.service.grpc.port}") private int port; @Value("${password.service.max.results}") private int maxResults; @Setter @Getter private Server server; private final GrpcServerInterceptor interceptor; private final PasswordDictReader reader; @Override public void findPasswords(FindPasswordsQuery request, StreamObserver
responseObserver) { String prefix = request.getQuery(); List results = reader.getDict().findPrefixes(prefix); int totalMatches = results.size(); if(totalMatches > maxResults) { results = results.subList(0, maxResults); } responseObserver.onNext(FindPasswordsResponse.newBuilder() .addAllMatches(results) .setNumOfMatches(totalMatches) .build()); responseObserver.onCompleted(); } @Override public ServerServiceDefinition getServiceDefinition() { return ServerInterceptors.intercept(bindService(), interceptor); } }
package com.se7so.grpc;
import com.google.protobuf.Empty;
import com.se7so.dict.PasswordDictReader;
import com.se7so.dict.PasswordTrie;
import com.se7so.model.FindPasswordsQuery;
import com.se7so.model.FindPasswordsResponse;
import com.se7so.model.PasswordsServiceHealthStatus;
import com.se7so.service.PasswordsServiceGrpc;
import com.se7so.service.PasswordsServiceHealthServiceGrpc;
import io.grpc.Server;
import io.grpc.ServerInterceptors;
import io.grpc.ServerServiceDefinition;
import io.grpc.netty.NettyServerBuilder;
import io.grpc.stub.StreamObserver;
import lombok.Data;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import java.io.IOException;
@RequiredArgsConstructor(onConstructor = @__(@Autowired))
@Log4j2
public class HealthStatusService extends PasswordsServiceHealthServiceGrpc.PasswordsServiceHealthServiceImplBase implements GrpcService {
@Getter
@Value("${health.status.grpc.port}")
private int port;
@Getter
@Setter
private Server server;
private final GrpcServerInterceptor interceptor;
private final PasswordDictReader passwordReader;
@Override
public void getPasswordsServiceHealthStatus(Empty request, StreamObserver responseObserver) {
responseObserver.onNext(PasswordsServiceHealthStatus.newBuilder()
.setStatus(passwordReader.getDict().size() == 0 ? "Error" : "Running")
.setTotalPasswordsLoaded(passwordReader.getDict().size())
.build());
responseObserver.onCompleted();
}
@Override
public ServerServiceDefinition getServiceDefinition() {
return ServerInterceptors.intercept(bindService(), interceptor);
}
}
The PasswordDictReader is basically our data store that loads passwords file and store it in memory as a Trie data structure .. it also has methods to search and get matches .. for simplicity you can check its implementation here.
rest-service:
In the rest service module there are REST endpoints that can be called easily from a browser to test the complexity of a password or to check the backend service health status.
Of course the rest-service communicate with the GRPC service by issuing a remote call to one of the services defined there .. and get back the response and map to DTO which is then given back as json to the client.
package com.se7so.rest;
import com.se7so.client.PasswordsServiceClient;
import com.se7so.model.PassServiceHealthDto;
import com.se7so.model.PasswordsResponseDto;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/")
public class RestServiceAPI {
@Autowired
private PasswordsServiceClient passwordsServiceClient;
@RequestMapping(value = "/health", produces = "application/json")
public PassServiceHealthDto getPasswordServiceHealthDto() {
return passwordsServiceClient.getHealthStatus();
}
@RequestMapping(value = "/passwords", produces = "application/json")
@ResponseBody
public PasswordsResponseDto getPasswordsService(@RequestParam(value = "q") String query) {
return passwordsServiceClient.findPasswordMatches(query);
}
}
Docker:
In the grpc-service I use the openjdk:8 as the base image and copy the jar file of the grpc-service module to /home/app.jar also copying the dictionary of passwords to /home.
After copying all the needed files .. issuing a command to start the service.
# grpc-service
FROM openjdk:8
ADD target/grpc-service-1.0-SNAPSHOT.jar /home/app.jar
ADD rockyou.tar.gz /home
CMD java -jar /home/app.jar
The rest-service docker file is doing the same thing .. but no need for the dictionary file in this case ..
# rest-service
FROM openjdk:8
ADD target/rest-service-1.0-SNAPSHOT.jar /home/app.jar
CMD java -jar /home/app.jar
Docker Compose:
And here is the docker-compose file that defines how the services are defined and communicate together to be able to deploy all services and configurations using one command docker-compose up.
And here is the docker-compose file that defines how the services are defined and communicate together to be able to deploy all services and configurations using one command docker-compose up.
#docker-compose.yml
version: '2'
services:
grpc-service:
build: ./grpc-service/
links:
- graphite
ports:
- "5000:5000"
rest-service:
build: ./rest-service/
links:
- grpc-service
ports:
- "80:8080"
graphite:
image: "jlachowski/grafana-graphite-statsd"
ports:
- "2003:2003"
- "8082:80"
The compose yml file contains definition of the three containers and exposes ports to the outside world .. also link the rest-service to the grpc-service to be able to call it at runtime.
Run:
Deploy the application using docker-compose:
cd /path/to/dockerized-microservices
mvn install # You will need this only once to build the application and generate binaries
docker-compose build # you will need this only once when you change the application
docker-compose up
Give it sometime to load services ...
Go to http://localhost/health
Go to http://localhost/passwords?q=123
Go to http://localhost/passwords?q=mypass
Go to http://localhost:8082/dashboard/db/grpc-service-monitor
Clean up:
docker-compose stop # Stops the services without cleaning the containers
docker-compose down # Stops the services and clean up the created containers
Read the next topic in this series Create/Manage docker swarm cluster.
Appreciative for such brilliant blog yours...!
ReplyDeleteMobile App Development Company
Excellent work!! keep blogging !!
ReplyDeleteWell I truly enjoyed studying it. This article offered by you is very useful for proper planning.
ReplyDelete메이저사이트
경마
온라인경마
Bookmarked this website page, will come back for extra articles. You, my pal, ROCK! I found just the information I already searched all over the place and simply couldn’t come across. What an ideal web-site.
ReplyDeleteMajortotosaiteu
trueraingyeongmasite
truerainkajifalsesite
totopickpro2