summaryrefslogtreecommitdiff
path: root/docs/systems/monitor.rst
blob: 72439bb5ce507bfced97099dca04b947a58ef2f0 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
.. index::
   single: Systems; Monitor

=======
Monitor
=======

Purpose
=======

This system hosts an `Icinga`_ instance to centrally monitor the services in
the CAcert network (especially for security updates and certificate
expiry).

.. note::

   To access the system you need a client certificate where the first email
   address in the Subject Distinguished Name field is a cacert.org address.
   Subject Alternative Names are not checked.

   If you are the administrator of a service please ask the monitor admins to
   add your system to the monitoring configuration and add you as system
   contact to allow for notifications and tasks like service outage
   acknowledgement, adding notes, rescheduling checks or setting downtimes for
   your service.

.. _Icinga: https://www.icinga.org/

Application Links
-----------------

The Icinga classic frontend
   https://monitor.cacert.org/

Administration
==============

System Administration
---------------------

* Primary: :ref:`people_jandd`
* Secondary: None

Application Administration
--------------------------

+-------------+-----------------------+
| Application | Administrator(s)      |
+=============+=======================+
| Icinga      | :ref:`people_jandd`   |
+-------------+-----------------------+

Contact
-------

* monitor-admin@cacert.org

Additional People
-----------------

:ref:`people_jandd` and :ref:`people_mario` have :program:`sudo` access on that
machine too.

Basics
======

Physical Location
-----------------

This system is located in an :term:`LXC` container on physical machine
:doc:`infra02`.

Logical Location
----------------

:IP Internet: :ip:v4:`213.154.225.230`
:IP Intranet: :ip:v4:`172.16.2.18`
:IP Internal: :ip:v4:`10.0.0.18`
:IPv6:        :ip:v6:`2001:7b8:616:162:2::18`
:MAC address: :mac:`00:ff:73:b3:17:43` (eth0)

.. seealso::

   See :doc:`../network`

.. index::
   single: Monitoring; Monitor

Monitoring
----------

:internal checks: :monitor:`monitor.infra.cacert.org`

DNS
---

.. index::
   single: DNS records; Monitor

=================== ======== =========================
Name                Type     Content
=================== ======== =========================
monitor.cacert.org. IN CNAME infrastructure.cacert.org
=================== ======== =========================

.. seealso::

   See :wiki:`SystemAdministration/Procedures/DNSChanges`

Operating System
----------------

.. index::
   single: Debian GNU/Linux; Stretch
   single: Debian GNU/Linux; 9.4

* Debian GNU/Linux 9.4

Applicable Documentation
------------------------

This is it :-)

.. seealso::

   :ref:`Setup package update monitoring for a new container
   <setup_apt_checking>`

Services
========

Listening services
------------------

+----------+---------+---------+-----------------------------+
| Port     | Service | Origin  | Purpose                     |
+==========+=========+=========+=============================+
| 22/tcp   | ssh     | ANY     | admin console access        |
+----------+---------+---------+-----------------------------+
| 25/tcp   | smtp    | local   | mail delivery to local MTA  |
+----------+---------+---------+-----------------------------+
| 80/tcp   | http    | ANY     | Icinga classic web frontend |
+----------+---------+---------+-----------------------------+
| 443/tcp  | https   | ANY     | Icinga classic web frontend |
+----------+---------+---------+-----------------------------+
| 5666/tcp | nrpe    | monitor | remote monitoring service   |
+----------+---------+---------+-----------------------------+
| 5432/tcp | pgsql   | local   | PostgreSQL database for IDO |
+----------+---------+---------+-----------------------------+

.. note::

   The ssh port is reachable via NAT on infrastructure.cacert.org:11822


Running services
----------------

.. index::
   single: apache httpd
   single: cron
   single: icinga
   single: ido2db
   single: nrpe
   single: openssh
   single: postfix
   single: postgresql
   single: puppet agent
   single: rsyslog

+--------------------+--------------------+----------------------------------------+
| Service            | Usage              | Start mechanism                        |
+====================+====================+========================================+
| Apache httpd       | Webserver for      | init script                            |
|                    | Icinga classic     | :file:`/etc/init.d/apache2`            |
+--------------------+--------------------+----------------------------------------+
| cron               | job scheduler      | init script :file:`/etc/init.d/cron`   |
+--------------------+--------------------+----------------------------------------+
| Icinga             | Icinga monitoring  | init script                            |
|                    | daemon             | :file:`/etc/init.d/icinga`             |
+--------------------+--------------------+----------------------------------------+
| IDO2DB             | IDO database       | init script                            |
|                    | writer daemon      | :file:`/etc/init.d/ido2db`             |
+--------------------+--------------------+----------------------------------------+
| Nagios NRPE server | remote monitoring  | init script                            |
|                    | service  by        | :file:`/etc/init.d/nagios-nrpe-server` |
|                    | this system itself |                                        |
+--------------------+--------------------+----------------------------------------+
| openssh server     | ssh daemon for     | init script :file:`/etc/init.d/ssh`    |
|                    | remote             |                                        |
|                    | administration     |                                        |
+--------------------+--------------------+----------------------------------------+
| Postfix            | SMTP server for    | init script                            |
|                    | local mail         | :file:`/etc/init.d/postfix`            |
|                    | submission         |                                        |
+--------------------+--------------------+----------------------------------------+
| PostgreSQL         | PostgreSQL         | init script                            |
|                    | database server    | :file:`/etc/init.d/postgresql`         |
|                    | for IDO            |                                        |
+--------------------+--------------------+----------------------------------------+
| Puppet agent       | configuration      | init script                            |
|                    | management agent   | :file:`/etc/init.d/puppet`             |
+--------------------+--------------------+----------------------------------------+
| rsyslog            | syslog daemon      | init script                            |
|                    |                    | :file:`/etc/init.d/syslog`             |
+--------------------+--------------------+----------------------------------------+

Databases
---------

+------------+--------+-----------------+
| RDBMS      | Name   | Used for        |
+============+========+=================+
| PostgreSQL | icinga | Icinga IDO data |
+------------+--------+-----------------+

Connected Systems
-----------------

None

Outbound network connections
----------------------------

* :doc:`infra02` as resolving nameserver
* :doc:`emailout` as SMTP relay
* :doc:`puppet` (tcp/8140) as Puppet master
* :doc:`proxyout` as HTTP proxy for APT
* crl.cacert.org (rsync) for getting CRLs
* all :ip:v4range:`10.0.0.0/24` and :ip:v4range:`172.16.2.0/24` systems for
  monitoring their services

.. todo:: add IPv6 ranges when they are monitored

Security
========

.. sshkeys::
   :RSA:     SHA256:8iOQQGmuqi4OrF2Qkqt9665w8G7Dwl6U9J8bFfYz7V0 MD5:df:98:f5:ea:05:c1:47:52:97:58:8f:42:55:d6:d9:b6
   :DSA:     SHA256:Sh/3OWrodFWc8ZbVTV1/aJDbpt5ztGrwSSWLECTNrOI MD5:07:2b:10:b1:6d:79:35:0f:83:aa:fc:ba:d6:2f:51:dc
   :ECDSA:   SHA256:GWvYqhQUt9INh/7VRVu6Z2YORoy/YzgBxNBmX+ZvMsk MD5:48:46:b1:5a:4e:05:64:8a:c3:76:33:77:20:91:14:70
   :ED25519: SHA256:L5roC867bvxDJ0ckbhIQOt2A9Nh1RQBVuIJFWwrPLG0 MD5:10:94:56:09:5b:a2:28:ab:11:e0:0f:6e:e4:0c:38:bb


Non-distribution packages and modifications
-------------------------------------------

The Puppet agent package and a few dependencies are installed from the official
Puppet APT repository because the versions in Debian are too old to use modern
Puppet features.

Risk assessments on critical packages
-------------------------------------

Icinga and the classic frontend are a bit aged but have a good security track
record.

Apache httpd has a good reputation and is a low risk package.

NRPE is flawed and should be replaced. The risk is somewhat mitigated by
firewalling on :doc:`the infrastructure host <infra02>`.

The system uses third party packages with a good security track record and
regular updates. The attack surface is small due to the tightly restricted
access to the system. The puppet agent is not exposed for access from outside
the system.

Critical Configuration items
============================

The system configuration is managed via Puppet profiles. There should be no
configuration items outside of the Puppet repository.

.. todo:: move configuration of :doc:`monitor` to Puppet code

Keys and X.509 certificates
---------------------------

.. sslcert:: monitor.cacert.org
   :altnames:   DNS:monitor.cacert.org, DNS:monitor.intra.cacert.org
   :certfile:   /etc/ssl/certs/monitor.c.o.pem
   :keyfile:    /etc/ssl/private/monitor.c.o.priv
   :serial:     1381FF
   :expiration: Mar 16 11:41:06 2020 GMT
   :sha1fp:     64:34:16:0D:2C:1B:38:5D:61:38:17:6E:D5:1B:90:B9:CF:DC:A9:75
   :issuer:     CA Cert Signing Authority

* :file:`/etc/ssl/certs/cacert.allcerts.pem` CAcert.org Class 1 and Class 3 CA
  certificates (allowed CA certificates for client certificates and the
  certificate chain for the server certificate)
* :file:`/var/local/ssl/crls/`

.. seealso::

   * :wiki:`SystemAdministration/CertificateList`

CRL fetch job
-------------

The script :file:`/etc/cron.hourly/update-crls` is used to fetch CRLs once per
hour.

Apache httpd configuration
--------------------------

The HTTP and HTTPS VirtualHost configuration is defined in
:file:`/etc/apache2/sites-available/icinga-nossl` and
:file:`/etc/apache2/sites-available/icinga` the HTTP VirtualHost redirects to
the HTTPS VirtualHost.

Icinga configuration
--------------------

The Icinga configuration is stored in the :file:`/etc/icinga/` directory.
Database configuration for IDO is stored in :file:`ido2db.cfg`. The Icinga
classic frontend configuration is stored in :file:`cgi.cfg`. Host and service
configurations are defined in the :file:`objects/` subdirectory.

Tasks
=====

Changes
=======

Planned
-------

System Future
-------------

* No plans

Additional documentation
========================

.. seealso::

   * :wiki:`PostfixConfiguration`

References
----------

Wiki page for this system
   :wiki:`SystemAdministration/Systems/Monitor`