Containerized Software: Monitoring

Andere delen in de serie:

In het geval van monitoring willen we de docker.machine.com-host als de client gebruiken, voor de andere hosts springboot1.machine.com en springboot2.machine.com, dat wil zeggen we willen docker-opdrachten op de docker.machine.com-host uitvoeren, zodat ze worden uitgevoerd op de andere hosts (docker --tlsverify -H = springboot1.machine.com: 2376 ps of docker --tlsverify -H = springboot2.machine.com: 2376 ps).

Docker werkt standaard via een Unix-socket zonder netwerkverbinding. Het kan ook communiceren via een HTTP-socket. Als we Docker nodig hebben om op een veilige manier bereikbaar te zijn via het netwerk, kunnen we TLS inschakelen door de tlsverify-vlag op te geven en Docker’s tlscacert-vlag naar een vertrouwd CA-certificaat te wijzen. In de daemon-modus staat het alleen verbindingen toe van clients die zijn geverifieerd door een certificaat dat is ondertekend door die CA. In de clientmodus maakt het alleen verbinding met servers met een certificaat dat is ondertekend door die CA. Door OpenSSL te gebruiken kunnen we de nodige sleutels en certificaten genereren.

Om de privé- en openbare ondertekeningssleutels te genereren, kunnen we

# create the necessary directories
[root@docker ~]# mkdir -p /opt/docker/certs/{ca,client,server}
 
# generate private and public signing keys
# genrsa - generate an RSA private key
# -aes256: encrypt the private key with the defined cipher
# -out: the output filename
# numbits: the size of the private key to generate in bits. This must be the last option specified.
[root@docker ~]# openssl genrsa -aes256 -out /opt/docker/certs/ca/ca-key.pem 2048
Generating RSA private key, 2048 bit long modulus
...........................+++
...................+++
e is 65537 (0x10001)
Enter pass phrase for /opt/docker/certs/ca/ca-key.pem:
Verifying - Enter pass phrase for /opt/docker/certs/ca/ca-key.pem:
 
# req - PKCS#10 certificate request and certificate generating utility.
# -new: this option generates a new certificate request. It will prompt the user for the relevant field values.
# -x509: this option outputs a self signed certificate instead of a certificate request.
# -days n: when the -x509 option is being used this specifies the number of days to certify the certificate for.
# -key: specifies the file to read the private key from.
# -[digest]: this specifies the message digest to sign the request with (run 'openssl dgst -h output' to see a list of possible values).
# -out filename: specifies the output filename to write to.
# -subj arg: sets subject name for new request or supersedes the subject name when processing a request. The arg must be formatted as /type0=value0/type1=value1/type2=...
[root@docker ~]# openssl req -new -x509 -days 365 -key /opt/docker/certs/ca/ca-key.pem -sha256 -out /opt/docker/certs/ca/ca.pem -subj '/C=NL/ST=Middleware/L=Snippets/O=Middleware Snippets/OU=Blogging/CN=*.machine.com'
Enter pass phrase for /opt/docker/certs/ca/ca-key.pem:

Merk op dat we een wild-card hebben gebruikt in de algemene naam (CN = *.machine.com) zodat we de certificaten op alle hosts kunnen gebruiken.

Om de serversleutel en een certificaatondertekeningsverzoek te genereren, kunnen we het volgende gebruiken

# generate the server key and certificate signing request
# genrsa - generate an RSA private key
# -out: the output filename
# numbits: the size of the private key to generate in bits. This must be the last option specified.
[root@docker ~]# openssl genrsa -out /opt/docker/certs/server/server-key.pem 2048
Generating RSA private key, 2048 bit long modulus
...........................................................................................................................................+++
......................................................................................................................+++
e is 65537 (0x10001)
 
# req - PKCS#10 certificate request and certificate generating utility.
# -new: this option generates a new certificate request. It will prompt the user for the relevant field values.
# -key: specifies the file to read the private key from.
# -[digest]: this specifies the message digest to sign the request with (run 'openssl dgst -h output' to see a list of possible values).
# -out filename: specifies the output filename to write to.
# -subj arg: sets subject name for new request or supersedes the subject name when processing a request. The arg must be formatted as /type0=value0/type1=value1/type2=...
[root@docker ~]# openssl req -new -key /opt/docker/certs/server/server-key.pem -sha256 -out /opt/docker/certs/server/server.csr -subj '/C=NL/ST=Middleware/L=Snippets/O=Middleware Snippets/OU=Blogging/CN=*.machine.com'

Vervolgens moeten we het certificaatondertekeningsverzoek van onze CA ondertekenen

# we can also add additional subject identities if needed
# x509 - Certificate display and signing utility
# -req: by default a certificate is expected on input. With this option a certificate request is expected instead
# -days arg: specifies the number of days to make a certificate valid for.
# -[digest]: this specifies the message digest to sign the request with (run 'openssl dgst -h output' to see a list of possible values).
# -in filename: specifies the input filename to read a certificate from .
# -CA filename: specifies the CA certificate to be used for signing. When this option is present x509 behaves like a "mini CA". The input file is signed by this CA using this option: that is its issuer name is set to the subject name of the CA and it is digitally signed using the CAs private key.
# -CAkey filename: sets the CA private key to sign a certificate with. If this option is not specified then it is assumed that the CA private key is present in the CA certificate file.
# -CAcreateserial: with this option the CA serial number file is created if it does not exist: it will contain the serial number "02" and the certificate being signed will have the 1 as its serial number. Normally if the -CA option is specified and the serial number file does not exist it is an error.
# -out filename: specifies the output filename to write to.
# -extfile filename: file containing certificate extensions to use. If not specified then no extensions are added to the certificate.
[root@docker ~]# echo subjectAltName = IP:192.168.101.210,IP:192.168.101.220,IP:192.168.101.230,IP:127.0.0.1 > /opt/docker/certs/server/extfile.cnf
[root@docker ~]# openssl x509 -req -days 365 -sha256 -in /opt/docker/certs/server/server.csr -CA /opt/docker/certs/ca/ca.pem -CAkey /opt/docker/certs/ca/ca-key.pem -CAcreateserial -out /opt/docker/certs/server/server-cert.pem -extfile /opt/docker/certs/server/extfile.cnf
Signature ok
subject=/C=NL/ST=Middleware/L=Snippets/O=Middleware Snippets/OU=Blogging/CN=*.machine.com
Getting CA Private Key
Enter pass phrase for /opt/docker/certs/ca/ca-key.pem:
 
# sign the public key without additional subject identities
[root@docker ~]# openssl x509 -req -days 365 -sha256 -in /opt/docker/certs/server/server.csr -CA /opt/docker/certs/ca/ca.pem -CAkey /opt/docker/certs/ca/ca-key.pem -CAcreateserial -out /opt/docker/certs/server/server-cert.pem
Signature ok
subject=/C=NL/ST=Middleware/L=Snippets/O=Middleware Snippets/OU=Blogging/CN=*.machine.com
Getting CA Private Key
Enter pass phrase for /opt/docker/certs/ca/ca-key.pem:

Ten slotte genereren we de clientsleutel en het certificaat

# generate client key and certificate signing request
# genrsa - generate an RSA private key
# -out: the output filename
# numbits: the size of the private key to generate in bits. This must be the last option specified.
[root@docker ~]# openssl genrsa -out /opt/docker/certs/client/key.pem 2048
Generating RSA private key, 2048 bit long modulus
..+++
.................+++
e is 65537 (0x10001)
 
# req - PKCS#10 certificate request and certificate generating utility.
# -new: this option generates a new certificate request. It will prompt the user for the relevant field values.
# -key: specifies the file to read the private key from.
# -out filename: specifies the output filename to write to.
# -subj arg: sets subject name for new request or supersedes the subject name when processing a request. The arg must be formatted as /type0=value0/type1=value1/type2=..
[root@docker ~]# openssl req -new -key /opt/docker/certs/client/key.pem -out /opt/docker/certs/client/client.csr -subj '/CN=client'
 
# sign the client key
# x509 - Certificate display and signing utility
# -req: by default a certificate is expected on input. With this option a certificate request is expected instead
# -days arg: specifies the number of days to make a certificate valid for.
# -[digest]: this specifies the message digest to sign the request with (run 'openssl dgst -h output' to see a list of possible values).
# -in filename: specifies the input filename to read a certificate from .
# -CA filename: specifies the CA certificate to be used for signing. When this option is present x509 behaves like a "mini CA". The input file is signed by this CA using this option: that is its issuer name is set to the subject name of the CA and it is digitally signed using the CAs private key.
# -CAkey filename: sets the CA private key to sign a certificate with. If this option is not specified then it is assumed that the CA private key is present in the CA certificate file.
# -CAcreateserial: with this option the CA serial number file is created if it does not exist: it will contain the serial number "02" and the certificate being signed will have the 1 as its serial number. Normally if the -CA option is specified and the serial number file does not exist it is an error.
# -out filename: specifies the output filename to write to.
# -extfile filename: file containing certificate extensions to use. If not specified then no extensions are added to the certificate.
[root@docker ~]# echo extendedKeyUsage = clientAuth > /opt/docker/certs/client/extfile.cnf
[root@docker ~]# openssl x509 -req -days 365 -sha256 -in /opt/docker/certs/client/client.csr -CA /opt/docker/certs/ca/ca.pem -CAkey /opt/docker/certs/ca/ca-key.pem -CAcreateserial -out /opt/docker/certs/client/cert.pem -extfile /opt/docker/certs/client/extfile.cnf
Signature ok
subject=/CN=client
Getting CA Private Key
Enter pass phrase for /opt/docker/certs/ca/ca-key.pem:
 
# change the permissions of the private keys
[root@docker ~]# chmod 0400 /opt/docker/certs/ca/ca-key.pem /opt/docker/certs/client/key.pem /opt/docker/certs/server/server-key.pem
 
# change the permissions of the public keys
[root@docker ~]# chmod 0444 /opt/docker/certs/ca/ca.pem /opt/docker/certs/client/cert.pem /opt/docker/certs/server/server-cert.pem
 
# optionally the certificate signing requests and extention files can be removed
[root@docker ~]# rm -f /opt/docker/certs/client/client.csr /opt/docker/certs/client/extfile.cnf /opt/docker/certs/server/server.csr /opt/docker/certs/server/extfile.cnf
 
# copy the certificates to the other hosts
[root@docker certs]# scp -rp * root@springboot1.machine.com:/opt/docker/certs/
root@springboot1.machine.com's password:
ca-key.pem                                                                                                                                                     100% 1766     1.7KB/s   00:00
ca.pem                                                                                                                                                         100% 1383     1.4KB/s   00:00
ca.srl                                                                                                                                                         100%   17     0.0KB/s   00:00
key.pem                                                                                                                                                        100% 1679     1.6KB/s   00:00
cert.pem                                                                                                                                                       100% 1155     1.1KB/s   00:00
server-key.pem                                                                                                                                                 100% 1675     1.6KB/s   00:00
server-cert.pem                                                                                                                                                100% 1265     1.2KB/s   00:00
[root@docker certs]# scp -rp * root@springboot2.machine.com:/opt/docker/certs/
root@springboot2.machine.com's password:
ca-key.pem                                                                                                                                                     100% 1766     1.7KB/s   00:00
ca.pem                                                                                                                                                         100% 1383     1.4KB/s   00:00
ca.srl                                                                                                                                                         100%   17     0.0KB/s   00:00
key.pem                                                                                                                                                        100% 1679     1.6KB/s   00:00
cert.pem                                                                                                                                                       100% 1155     1.1KB/s   00:00
server-key.pem                                                                                                                                                 100% 1675     1.6KB/s   00:00
server-cert.pem                     

Om ervoor te zorgen dat de Docker-daemon alleen verbindingen accepteert van clients die een certificaat leveren dat door onze CA wordt vertrouwd, bewerken we het bestand /usr/lib/systemd/system/docker.service

[root@springboot2 system]# cat docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
 
[Service]
Type=notify
#ExecStart=/usr/bin/docker daemon -H fd://
ExecStart=/usr/bin/docker -d --tlsverify --tlscacert=/opt/docker/certs/ca/ca.pem --tlscert=/opt/docker/certs/server/server-cert.pem --tlskey=/opt/docker/certs/server/server-key.pem -H=0.0.0.0:2376
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
 
[Install]
WantedBy=multi-user.target
 
# Restart docker
[root@springboot2 system]# systemctl daemon-reload
[root@springboot2 system]# systemctl start docker.service
[root@springboot2 system]# systemctl status docker.service
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: active (running) since Wed 2015-08-19 14:36:56 CEST; 7s ago
     Docs: https://docs.docker.com
 Main PID: 11021 (docker)
   CGroup: /system.slice/docker.service
           -- 11021 /usr/bin/docker -d --tlsverify --tlscacert=/opt/docker/certs/ca/ca.pem --tlscert=/opt/docker/certs/server/server-cert.pem --tlskey=/opt/docker/certs/server/server-key.pem ...

Om te communiceren met de Docker-daemon moeten we nu het volgende gebruiken

[root@docker ~]# docker --tlsverify --tlscacert=/opt/docker/certs/ca/ca.pem --tlscert=/opt/docker/certs/client/cert.pem --tlskey=/opt/docker/certs/client/key.pem -H=springboot2.machine.com:2376 pull docker.machine.com:5000/springbootexample:latest
latest: Pulling from springbootexample
f1b10cd84249: Pull complete
c852f6d61e65: Pull complete
7322fbe74aa5: Pull complete
4cfba0adf483: Pull complete
6a5b64d0925e: Pull complete
e136fb7fd983: Pull complete
f027b1b29898: Pull complete
e8261eb40ec4: Pull complete
20ee6480f62e: Pull complete
369ff136c2b5: Pull complete
c200531e0027: Pull complete
012ae7d5f8b5: Pull complete
c8a563965cf0: Pull complete
Digest: sha256:0b50559aa6d94d3a6919c5efd77b2ce262760c1bc4a9db7cb0b4f539ee854e0e
Status: Downloaded newer image for docker.machine.com:5000/springbootexample:latest
 
[root@docker ~]# docker --tlsverify --tlscacert=/opt/docker/certs/ca/ca.pem --tlscert=/opt/docker/certs/client/cert.pem --tlskey=/opt/docker/certs/client/key.pem -H=springboot2.machine.com:2376 images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
docker.machine.com:5000/springbootexample   latest              c8a563965cf0        3 hours ago         528.4 MB
 
[root@docker ~]# docker --tlsverify --tlscacert=/opt/docker/certs/ca/ca.pem --tlscert=/opt/docker/certs/client/cert.pem --tlskey=/opt/docker/certs/client/key.pem -H=springboot2.machine.com:2376 run -d --name="springbootexample" --net="host" -v /home/temp:/home/temp -p 8080:8080 -p 9090:9090 docker.machine.com:5000/springbootexample
38a2ff01166a7c5c540c4fc6117946a535af9cf915a0c5bb7ae29b573429392e
 
[root@docker ~]# docker --tlsverify --tlscacert=/opt/docker/certs/ca/ca.pem --tlscert=/opt/docker/certs/client/cert.pem --tlskey=/opt/docker/certs/client/key.pem -H=springboot2.machine.com:2376 ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
38a2ff01166a        docker.machine.com:5000/springbootexample   "java -XX:+UnlockComm"   41 seconds ago      Up 39 seconds                           springbootexample
 
[root@docker ~]# docker --tlsverify --tlscacert=/opt/docker/certs/ca/ca.pem --tlscert=/opt/docker/certs/client/cert.pem --tlskey=/opt/docker/certs/client/key.pem -H=springboot1.machine.com:2376 start springbootexample
springbootexample
 
[root@docker ~]# docker --tlsverify --tlscacert=/opt/docker/certs/ca/ca.pem --tlscert=/opt/docker/certs/client/cert.pem --tlskey=/opt/docker/certs/client/key.pem -H=springboot1.machine.com:2376 ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
7193fa0ef4c8        docker.machine.com:5000/springbootexample   "java -XX:+UnlockComm"   3 hours ago         Up 19 seconds                           springbootexample

Om Docker-clientverbindingen standaard te beveiligen, kunnen we de clientcertificaatbestanden naar de .docker-directory in de homedirectory verplaatsen en de omgevingsvariabelen DOCKER_HOST en DOCKER_TLS_VERIFY instellen.

# create the .docker directory
[root@docker ~]# mkdir -p ~/.docker
 
# copy the certificates
[root@docker ~]# cp /opt/docker/certs/ca/ca.pem ~/.docker
[root@docker ~]# cp /opt/docker/certs/client/cert.pem ~/.docker
[root@docker ~]# cp /opt/docker/certs/client/key.pem ~/.docker
 
# check
[root@docker .docker]# ll
-r--r--r-- 1 root root 1383 Aug 19 15:02 ca.pem
-r--r--r-- 1 root root 1155 Aug 19 15:02 cert.pem
-r-------- 1 root root 1679 Aug 19 15:07 key.pem
 
# edit .bashrc
[root@docker ~]# cat .bashrc
# .bashrc
 
# User specific aliases and functions
 
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
 
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
 
export DOCKER_HOST=tcp://springboot1.machine.com:2376
export DOCKER_TLS_VERIFY=1
 
# check
[root@docker ~]# echo $DOCKER_HOST
tcp://springboot1.machine.com:2376
[root@docker ~]# echo $DOCKER_TLS_VERIFY
1
# this communicates only with springboot1.machine.com
[root@docker ~]# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
docker.machine.com:5000/springbootexample   latest              c8a563965cf0        4 hours ago         528.4 MB
 
# note that it is better to leave out the $DOCKER_HOST and $DOCKER_TLS_VERIFY environment variables and use
[root@docker ~]# docker --tlsverify -H=springboot1.machine.com:2376 ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
7193fa0ef4c8        docker.machine.com:5000/springbootexample   "java -XX:+UnlockComm"   3 hours ago         Up 15 minutes                           springbootexample
[root@docker ~]# docker --tlsverify -H=springboot2.machine.com:2376 ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
38a2ff01166a        docker.machine.com:5000/springbootexample   "java -XX:+UnlockComm"   23 minutes ago      Up 23 minutes                           springbootexample

Monitoring

Met de client-server configratie op zijn plek, zijn we nu makelijker in staat de individuele containers te monitoren, bijvoorbeeld om de logging te controlleren:

[root@docker ~]# docker --tlsverify -H=springboot1.machine.com:2376 logs springbootexample
 
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
...
INFO 1 --- [           main] Example                                  : Starting Example on springboot1.machine.com with PID 1 (/u01/springboot/jars/springboot.jar started by root in /)
...
INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
INFO 1 --- [           main] Example                                  : Started Example in 3.554 seconds (JVM running for 4.839)
INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring FrameworkServlet 'dispatcherServlet'
INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started
INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 9 ms

Merk op dat Spring Boot standaard alleen logt naar de console en niet naar logbestanden schrijft. Als we naast de console-uitvoer naar logbestanden willen schrijven, moeten we een logging.file– of logging.path-eigenschap instellen. Om flight recordings te maken kunnen we het volgende gebruiken:

# open a command prompt for springboot1.machine.com
[root@docker ~]# docker --tlsverify -H=springboot1.machine.com:2376 exec -ti springbootexample /bin/bash
 
# obtain the PID of the Spring Boot process
[root@springboot1 /]# jcmd
1 /u01/springboot/jars/springboot.jar
238 sun.tools.jcmd.JCmd
 
# start a Java Flight Recording
[root@springboot1 /]# jcmd 1 JFR.start name=springboot1 settings=default duration=600s filename=/home/temp/springboot1.jfr
1:
Started recording 1. The result will be written to: /home/temp/springboot1.jfr
 
# check the Java Flight Recording
[root@springboot1 /]# jcmd 1 JFR.check
1:
Recording: recording=1 name="springboot1" duration=10m filename="/home/temp/springboot1.jfr" compress=false (running)
 
# exit the command prompt for springboot1.machine.com
[root@springboot1 /]# exit
exit
 
# open a command prompt for springboot2.machine.com
[root@docker ~]# docker --tlsverify -H=springboot2.machine.com:2376 exec -ti springbootexample /bin/bash
 
# obtain the PID for the Spring Boot process
[root@springboot2 /]# jcmd
1 /u01/springboot/jars/springboot.jar
61 sun.tools.jcmd.JCmd
 
# start a Java Flight Recording
[root@springboot2 /]# jcmd 1 JFR.start name=springboot2 settings=default duration=600s filename=/home/temp/springboot2.jfr
1:
Started recording 1. The result will be written to: /home/temp/springboot2.jfr
 
# check the Java Flight Recording
[root@springboot2 /]# jcmd 1 JFR.check
1:
Recording: recording=1 name="springboot2" duration=10m filename="/home/temp/springboot2.jfr" compress=false (running)
 
# exit the command prompt for springboot2.machine.com
[root@springboot2 /]# exit
exit
 
# obtain the flight recordings
[root@docker ~]# scp root@springboot1.machine.com:/home/temp/springboot1.jfr /tmp/
root@springboot1.machine.com's password:
springboot1.jfr                                                                                                                                                100% 5339KB   5.2MB/s   00:00
[root@docker ~]# scp root@springboot2.machine.com:/home/temp/springboot2.jfr /tmp/
root@springboot2.machine.com's password:
springboot2.jfr

De flight recordings kunnen in Java Mission Control worden bekeken:

Oeps, we worden geblokkeerd door de garbage collector (maar slechts één keer, en slechts voor 88 milliseconden.).

Scroll to Top