Monday, December 24, 2018

Command line tools every (Java) developer should know

In this post I will go through some useful command line tools to debug and tune your service. I will show a sample output and description of the most important parts of the output, feel free to read the manual pages for more details about the tools and supported flags to tune their output.

vmstat


This system tool is very useful to view and monitor statistics of your system processes, memory, paging, block IO, disks, and CPU. Its useful to check IO/CPU utilization and spot any bottlenecks.

Example:

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 12583088 119480 1590740    0    0   406    72  470 1035 10  3 87  0  0
 0  0      0 12582832 119480 1590740    0    0     0     0  451 1562  1  0 98  0  0
 0  0      0 12579484 119480 1590732    0    0     0     0  408 1345  1  0 98  0  0

The above command is to show stats with 1 second delay.

Processes: Amount of runnable and un-interruptible processes
Memory: Free/cache memory and memory used as buffers and the amount of virtual memory used (swpd)
Swap: Swap in and out (from disk)
IO: Blocks in (READ) and out (WRITE) from disk
System: Number of interrupts per second and context switches
CPU: Time running user code vs system (kernel) code, idle time, and waiting for IO

 

iostat


This system tool is very useful to view and monitor IO statistics of your system.

Example:

$ iostat -xm 5
Linux 4.15.0-42-generic

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.27    0.02    1.59    0.15    0.00   92.97

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     5.29   20.82    6.42     0.43     0.11    40.69     0.02    1.18    0.25    4.20   0.14   0.39

The -x flag is to show extended statistics and -m is to view stats in megabytes per second. Of course some of the information above can also be viewed by the previous tool `vmstat`.

%user, %system, %iowait, %idle: The percentage of CPU spent running user code, system code, waiting for IO, and idle
rrqm/s, wrqm/s: The number of (read/write) requests merged per second that were queued to the device
r/s, w/s, rMB/s, wMB/s: Reads/Writes (after merge) per seconds, MBs per seconds (read/write)
await, r_await, w_await: Average time in ms for IO requests to the device to be served, average time in ms for read, average time in ms for write

nicstat


This tool isn't installed by default in linux systems but can be installed using package managers or downloaded from sourceforge. It basically does the same as previous tools but for network monitoring.

Example:

$ nicstat 5
    Time      Int   rKB/s   wKB/s   rPk/s   wPk/s    rAvs    wAvs %Util    Sat
20:10:52       lo    0.33    0.33    3.98    3.98   85.25   85.25  0.00   0.00
20:10:52   wlp2s0    6.18    1.85   12.23    9.03   517.6   210.0  0.00   0.00

Prints network statistics ever 5 seconds.

Int: Interface name
rKB/s, wKB/s: KBs Read(Received) Write(Sent)
rPk/s, wPk/s: Packets received/sent
rAvs, wAvs: Average packet size received/sent
%Util, Sat: Percentage utilization of the interface, Saturation (errors/second)

jcmd


This tool and the coming few ones come along with the JDK and can be used to send diagnostic commands to a running JVM.

Example:

$ jcmd 8880 VM.uptime # 8880 is the process ID, VM.uptime is to check uptime of the running java process

8880:
163.883 s

$ jcmd 8880 VM.system_properties # View system properties of a single java process

8880:
#Mon Dec 24 20:32:01 CET 2018
java.runtime.name=OpenJDK Runtime Environment
sun.boot.library.path=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
java.vm.version=25.191-b12
java.vm.vendor=Oracle Corporation
java.vendor.url=http\://java.oracle.com/
path.separator=\:
java.vm.name=OpenJDK 64-Bit Server VM
file.encoding.pkg=sun.io
user.country=US
..... # and more .....

$ jcmd 8880 VM.version

8880:
OpenJDK 64-Bit Server VM version 25.191-b12
JDK 8.0_191

$ jcmd 8880 VM.command_line # The command line of the process was started

8880:
VM Arguments:
java_command: HelloWorld -Dse7so=29 -client
java_class_path (initial): .
Launcher Type: SUN_STANDARD

$ jcmd 8880 VM.flags # Enable flags - try with [-all]

8880:
-XX:CICompilerCount=3 -XX:InitialHeapSize=262144000 -XX:MaxHeapSize=4175429632 -XX:MaxNewSize=1391460352 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=87031808 -XX:OldSize=175112192 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC

$ jcmd 8880 Thread.print # Thread dump - see also jstack tool below
# Very long output here that can be saved to a file for later analysis

$ jcmd 8880 GC.run #
8880:
Command executed successfully

$ jcmd 8880 GC.class_histogram # Show heap histogram ([C is character array, [B is byte array, [I is integer array)
8880:

 num     #instances         #bytes  class name
----------------------------------------------
   1:          1157        1614520  [C
   2:           131         216072  [I
   3:           474          53992  java.lang.Class
   4:          1145          27480  java.lang.String
   5:           531          26632  [Ljava.lang.Object;
   6:            20          25264  [B
   7:           378           9072  java.util.LinkedList$Node
   8:           211           6752  java.util.HashMap$Node
   9:           190           6080  java.util.LinkedList
  10:            75           5400  java.lang.reflect.Field
  11:           256           4096  java.lang.Integer
  12:            93           3720  java.lang.ref.SoftReference
  13:           111           3552  java.util.Hashtable$Entry
  14:             7           2632  java.lang.Thread
  15:            12           1984  [Ljava.util.HashMap$Node;
  16:            51           1896  [Ljava.lang.String;
  17:            38           1824  sun.util.locale.LocaleObjectCache$CacheEntry
  18:            47           1504  java.util.concurrent.ConcurrentHashMap$Node
  19:            23           1472  java.net.URL
  20:            14           1120  [Ljava.util.WeakHashMap$Entry;
  21:            14           1120  java.lang.reflect.Constructor
  22:             2           1064  [Ljava.lang.invoke.MethodHandle;
  23:             1           1040  [Ljava.lang.Integer;
  24:            26           1040  java.io.ObjectStreamField
  25:             6            992  [Ljava.util.Hashtable$Entry;
  26:            15            840  java.lang.Class$ReflectionData
  27:            21            840  java.lang.ref.Finalizer


jconsole


In fact this tool is a GUI tool but it worth mentioning as it provides very useful information to monitor and control java applications.

It views information about Memory, CPU, Threads, and Classes loaded.

jmap


Provides heap dumps and other information about JVM memory usage.

Example:

$ jmap -dump:live,format=b,file=/tmp/dump.hprof 9293
Dumping heap to /tmp/dump.hprof ...
Heap dump file created

Generates a dump from a live process, in binary format and saves it to /tmp/dump.hprof - you can then analyze it with any heap dump analyzer.

jstack


Useful to generate thread dumps with full stack traces.

Example:

$ stack -l > /tmp/myapp.tdump

You can then analyze the thread dump using any thread dump analyzer.

visualvm


This is also a GUI tool but very useful to monitor your java application. Its man page describes it as all-in-one java troubleshooting tool and in fact it is very nice and can be very useful along with the above commands. you can also use to analyze heap and thread dumps generated earlier using jmap and jstack.

jstat


Is one of the best tools to view very useful stats about GC, JIT compiler, and so many other things to monitor.

Example:

$ jstat -gcutil -t 9293 100ms
Timestamp         S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT   
         5452.8   0.00   0.00  77.88   0.86  96.16  84.57     10    0.056    10    0.295    0.351
         5453.0   0.00   0.00  77.88   0.86  96.16  84.57     10    0.056    10    0.295    0.351

The above shows GC information (the -t to show timestamps) every 100ms.

S0, S1: Survivor spaces 0 and 1 utilization as a percentage of the space's current capacity
E, O, M: Eden space, Old space, Metaspace utilization as percentage of the space's current capacity
CCS: Compressed class space utilization as a percentage
YGC, YGCT: Number of young generation GC events and young generation gc time
FGC, FGCT: Number of full GC events, and full GC time
GCT: Total garbage collection time

When the Eden space percentage goes up this triggers a young generation minor GC event which then will reduce its size and the S0, S1 will be swapped and of course Old generation increases a bit (some objects move from Y to O) - Full GC events are triggered when the Old generation percentage goes up  and as a result its size is reduced and fragmented.

$ jstat -class -t 9293 1s
Timestamp       Loaded  Bytes  Unloaded  Bytes     Time   
         5845.2   3602  7665.3       13    15.8       1.33
         5846.2   3602  7665.3       13    15.8       1.33
         5847.2   3602  7665.3       13    15.8       1.33
         5848.2   3602  7665.3       13    15.8       1.33

The same command but to view class loader statistics every 1 second.

Loaded: Number of classes loaded
Bytes: Number of bytes loaded
Unloaded: Number of classes unloaded
Bytes: Number of bytes unloaded
Time: Time spent performing class loading and unloading

$ jstat -printcompilation -t 9293 1s
Timestamp       Compiled  Size  Type Method
         6067.7     3744    684    1 java/util/ComparableTimSort sort
         6068.7     3744    684    1 java/util/ComparableTimSort sort
         6069.7     3744    684    1 java/util/ComparableTimSort sort

The same command but to view Java HotSpot VM compiler method statistics.

Compiled: Number of compilation tasks performed on the method
Size: Size of code byte code (not binary code)
Type: Compilation type of the most recently compiled method 
Method: Class name and method name identifying the most recently compiled method
$ jstat -compiler -t 9293 1s
Timestamp       Compiled Failed Invalid   Time   FailedType FailedMethod
         6378.9     3753      3       0    12.54          1 org/netbeans/lib/profiler/server/ProfilerRuntimeObjLiveness traceObjAlloc
         6380.0     3753      3       0    12.54          1 org/netbeans/lib/profiler/server/ProfilerRuntimeObjLiveness traceObjAlloc
         6381.0     3753      3       0    12.54          1 org/netbeans/lib/profiler/server/ProfilerRuntimeObjLiveness traceObjAlloc
         6382.0     3753      3       0    12.54          1 org/netbeans/lib/profiler/server/ProfilerRuntimeObjLiveness traceObjAlloc

Java HotSpot VM Just-in-Time compiler statistics.

Compiled: Number of compilation tasks performed
Failed: Number of failed compilation tasks
Invalid: Number of invalidated compilation tasks
Time: Time spent on compilation
FailedMethod: The most recent failed method to compile

The last two examples are a bit hard to understand if you don't really know the basics of JIT and how it works .. you can go ahead and read what happens behind the scene to realize the importance of both examples and how you can use them to diagnose possible performance issues in your services.

Those are the most useful tools from my point of view to monitor, diagnose, troubleshoot your services .. please comment with other tools you might see useful but not stated above.
Cheers

Saturday, October 20, 2018

The guide to be a better team member

Its been a while since I posted something, and even longer for a non-tech one. 

Today's post is a recipe for being a better team member or even better if you already are. The major thing here is to start applying all the following steps yourself before asking others to do them. 


Step 1: Feedback

 

Everyday we give or receive feedback in different formats (e.g., Code review, Design review, or even face to face feedback). I see most of the issues that arise in every team is because of feedback - either someone gives feedback in an inappropriate way or not being open for feedback.

My manager once told me "Any feedback is a good feedback", and another manager told me once "Take the feedback and filter what really makes sense to you but always respect and assume good intentions".

The idea behind this is to always assume good intentions keeping in mind culture difference and preferences of people based on culture, background, and experience.

In case you are the one who is giving feedback, always think about it and think what is the best way to give it either in private or if OK to be in public (e.g., Review sessions). Also keep in mind culture difference and other people's preferences and make sure you don't turn people down by giving feedback inappropriately, this way you will lose them or even worse lose their trust.

Step 2: Disagree and commit

 

Everyday you and other team members propose solutions for current team problems either technical or non-technical. You and your colleagues have less chance to agree on all of them.

Never ever take it personal or be frustrated that there are different opinions or views "I know its hard" .. but usually if the team agrees on something and commit to it, you MUST commit to it as well. This has great potential for someone in the team to learn something new.

Step 3: Choose a mentor for yourself


Mentoring is one of the best ways for someone to learn and receive great feedback that helps improving.

I personally usually ask one or two of my colleagues to observe me and give me feedback in private if they have any, from my personal experience this helped me to improve in different areas which I had good potential for improvement "and of course still have".

Step 4: Mentor someone

 


This is exactly the same as step number 3 but the other way around, where you will be the mentor/observer for another team memer(s).

From my experience this helps earning your colleagues' trust and appreciation for your honest feedback both positive and non-positive.

Step 5: Learn from your mistakes


This is of course the clearest one so far. Its also a great way to measure improvement. 

As an example if you receive code review feedback in an open way as potential to learn something new by constructive discussions with other team members, only this way you will learn and remember to not repeat the same mistake again and again.

Good luck

Sunday, April 1, 2018

AWS kubernetes + Mongodb cluster network configuration

In this tutorial I will quickly go through the steps I have done to set up two different clusters and properly configure both of them in a way that suits my need .. here are what I wanted to have in the end:

- Setting up a kubernetes cluster with one master and 2 nodes
- Setting up a mongodb cluster with one primary and 2 secondary instances
- Be able to connect from my application running on kubernetes to mongodb servers
- Mongodb server to not be publicly exposed .. so I wanted to keep it private for security reasons

Ready? Lets go ;)

1. Create kubernetes cluster using kops

Note: If your cluster is not created successfully .. may be you need to request some limits of instance types you configure for master and nodes of the cluster ;)

2. Create mongo-db cluster

I just went through this awesome guide which is using a basic AWS template to create all needed resources and configuration.

3. Connect both clusters together

Now at this step you can easily SSH to your bastian instance and connect to your mongodb  service and test your credentials and so on.

Lets try to connect from our publicly available bastian server:

se7so@se7so:~/Desktop/se7so-workspace$ ssh -i "mykeypair.pem" ubuntu@ec2-bastian-****
Last login: Sun Apr  1 17:15:28 2018 from *.*.*.*
ubuntu@linux-bastian:~$ ssh -i "mykeypair.pem" ec2-user@mongodb-primary
Last login: Sun Apr  1 17:15:28 2018 from *.*.*.*
[ec2-user@mongodb-primary ~]$ mongo
MongoDB shell version v3.4.14
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.14

But you may want to be able to directly connect to your DB from your application which will later run as a container on K8s POD.

But in this template the mongodb has only a private IP and not exposed publicly through a dns or elastic IP .. it also belongs to a diff VPC than our K8s cluster ...

You will need to configure two things in order to successfully connect to your mongodb instance from your K8s cluster.

3.1 Create a connection peering between both VPCs (Make sure you pick up the VPC ID that your mongodb instances belong to, and the one that your K8s instances belong to).

3.2 Once requested .. you will need to accept the request, right click accept connection peering.

3.3 Once you accept the request .. you will need then to adjust the route tables of both VPCs by adding a new route (Destination should be the CIDR of other VPC,  and target should be the ID of the connection peering request you created earlier).

At this step you configured both VPCs to be able to connect the IP ranges from the other one .. but that's not enough to connect to our mongodb cluster either through SSH or mongodb port(s).

3.4 Go to security groups of your mongodb primary instance and edit the inbound rules by allowing the security group of K8s cluster instances to connect on mongodb ports, you can also allow SSH.

Note: Outbound is accepting all traffic by default so no need to change anything.

4. Connect from K8s cluster

Now you can SSH to any of your K8s nodes or master and SSH to mongodb primary instance using its private IP.

root@k8-master:/# telnet 10.*.*.* 27017
Trying 10.*.*.*...
Connected to 10.*.*.*.
Escape character is '^]'.


5. Test connection from a running POD on K8s

Now create a kubernetes pod and SSH to it, to test your connection to mongodb instance.

root@nginx-7c87f569d-9mp8q:/# telnet 10.*.*.* 27017
Trying 10.*.*.*...

Oops it doesn't work :S

After many tries and alot of research I found some solutions to manually adjust iptables rules to route traffic to IP ranges of mongodb cluster to the host machine .. but I didn't like this solution as its not sustainable enough for me..

Now its time to use my kubernetes tricks :) .. what I came up with is the following ;)

5.1 Create a kubernetes service without a selector .. to point to the mongodb IP/PORT .. here are my configuration:

service.yaml
---
kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 27017

endpoints.yaml:
---
kind: Endpoints
apiVersion: v1
metadata:
  name: my-service
subsets:
  - addresses:
      - ip: 10.0.4.205
    ports:
      - port: 27017

Run:

se7so@se7so:~/Desktop/se7so-workspace/$ kubectl apply -f .
endpoints "my-service" created
service "my-service" created

SSH again to your K8s POD and run:

se7so@se7so:~/Desktop/se7so-workspace/$ kubectl exec -it nginx-7c87f569d-9mp8q bash
root@nginx-7c87f569d-9mp8q:/# telnet my-service.default.svc.cluster.local 80
Trying 100.71.103.125...
Connected to my-service.default.svc.cluster.local.
Escape character is '^]'.

Using mongo CLI:

Note: I'm redirecting 80 to 27017 ports so don't get confused .. not sure why I did it this way :P

root@nginx-7c87f569d-9mp8q:/# mongo --host my-service.default.svc.cluster.local --port 80
MongoDB shell version: 3.2.11
connecting to: my-service.default.svc.cluster.local:80/test
Welcome to the MongoDB shell.
s0:PRIMARY>
Note: you don't have to use `.default.svc.cluster.local` postfix, you can just type 'my-service'


Pingo it works and its more sustainable solution ;) .. I really hope this will help you saving some hours of research.