summaryrefslogtreecommitdiff
path: root/docs/systems/monitor.rst
blob: fb5472a54cd60d29d66e7c53a3616a12af13f19a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
.. index::
   single: Systems; Monitor

=======
Monitor
=======

Purpose
=======

This system hosts an `Icinga`_ instance to centrally monitor the services in
the CAcert network (especially for security updates and certificate
expiry).

.. note::

   To access the system you need a client certificate where the first email
   address in the Subject Distinguished Name field is a cacert.org address.
   Subject Alternative Names are not checked.

   If you are the administrator of a service please ask the monitor admins to
   add your system to the monitoring configuration and add you as system
   contact to allow for notifications and tasks like service outage
   acknowledgement, adding notes, rescheduling checks or setting downtimes for
   your service.

.. _Icinga: https://www.icinga.org/

Application Links
-----------------

The Icinga classic frontend
   https://monitor.cacert.org/

Administration
==============

System Administration
---------------------

* Primary: :ref:`people_martin`
* Secondary: :ref:`people_neo`

Application Administration
--------------------------

+-------------+-----------------------+
| Application | Administrator(s)      |
+=============+=======================+
| Icinga      | :ref:`people_martin`, |
|             | :ref:`people_neo`,    |
|             | :ref:`people_jandd`   |
+-------------+-----------------------+

Contact
-------

* monitor-admin@cacert.org

Additional People
-----------------

:ref:`people_jandd` and :ref:`people_mario` have :program:`sudo` access on that
machine too.

Basics
======

Physical Location
-----------------

This system is located in an :term:`LXC` container on physical machine
:doc:`infra02`.

Logical Location
----------------

:IP Internet: :ip:v4:`213.154.225.230`
:IP Intranet: :ip:v4:`172.16.2.18`
:IP Internal: :ip:v4:`10.0.0.18`
:MAC address: :mac:`10.0.0.18` (eth0)

.. seealso::

   See :doc:`../network`

DNS
---

.. index::
   single: DNS records; Monitor

=================== ======== =========================
Name                Type     Content
=================== ======== =========================
monitor.cacert.org. IN CNAME infrastructure.cacert.org
=================== ======== =========================

.. seealso::

   See :wiki:`SystemAdministration/Procedures/DNSChanges`

Operating System
----------------

.. index::
   single: Debian GNU/Linux; Wheezy
   single: Debian GNU/Linux; 7.10

* Debian GNU/Linux 7.10

Applicable Documentation
------------------------

This is it :-)

.. seealso::

   :ref:`Setup package update monitoring for a new container
   <setup_apt_checking>`

Services
========

Listening services
------------------

+----------+---------+---------+-----------------------------+
| Port     | Service | Origin  | Purpose                     |
+==========+=========+=========+=============================+
| 22/tcp   | ssh     | ANY     | admin console access        |
+----------+---------+---------+-----------------------------+
| 25/tcp   | smtp    | local   | mail delivery to local MTA  |
+----------+---------+---------+-----------------------------+
| 80/tcp   | http    | ANY     | Icinga classic web frontend |
+----------+---------+---------+-----------------------------+
| 443/tcp  | https   | ANY     | Icinga classic web frontend |
+----------+---------+---------+-----------------------------+
| 5666/tcp | nrpe    | monitor | remote monitoring service   |
+----------+---------+---------+-----------------------------+
| 5432/tcp | pgsql   | local   | PostgreSQL database for IDO |
+----------+---------+---------+-----------------------------+

.. note::

   The ssh port is reachable via NAT on infrastructure.cacert.org:11822


Running services
----------------

.. index::
   single: Apache
   single: Icinga
   single: IDO2DB
   single: Postfix
   single: PostgreSQL
   single: cron
   single: nrpe
   single: openssh

+--------------------+--------------------+----------------------------------------+
| Service            | Usage              | Start mechanism                        |
+====================+====================+========================================+
| openssh server     | ssh daemon for     | init script :file:`/etc/init.d/ssh`    |
|                    | remote             |                                        |
|                    | administration     |                                        |
+--------------------+--------------------+----------------------------------------+
| Apache httpd       | Webserver for      | init script                            |
|                    | Icinga classic     | :file:`/etc/init.d/apache2`            |
+--------------------+--------------------+----------------------------------------+
| cron               | job scheduler      | init script :file:`/etc/init.d/cron`   |
+--------------------+--------------------+----------------------------------------+
| rsyslog            | syslog daemon      | init script                            |
|                    |                    | :file:`/etc/init.d/syslog`             |
+--------------------+--------------------+----------------------------------------+
| Icinga             | Icinga monitoring  | init script                            |
|                    | daemon             | :file:`/etc/init.d/icinga`             |
+--------------------+--------------------+----------------------------------------+
| IDO2DB             | IDO database       | init script                            |
|                    | writer daemon      | :file:`/etc/init.d/ido2db`             |
+--------------------+--------------------+----------------------------------------+
| PostgreSQL         | PostgreSQL         | init script                            |
|                    | database server    | :file:`/etc/init.d/postgresql`         |
|                    | for IDO            |                                        |
+--------------------+--------------------+----------------------------------------+
| Postfix            | SMTP server for    | init script                            |
|                    | local mail         | :file:`/etc/init.d/postfix`            |
|                    | submission         |                                        |
+--------------------+--------------------+----------------------------------------+
| Nagios NRPE server | remote monitoring  | init script                            |
|                    | service  by        | :file:`/etc/init.d/nagios-nrpe-server` |
|                    | this system itself |                                        |
+--------------------+--------------------+----------------------------------------+

Databases
---------

+------------+--------+-----------------+
| RDBMS      | Name   | Used for        |
+============+========+=================+
| PostgreSQL | icinga | Icinga IDO data |
+------------+--------+-----------------+

Connected Systems
-----------------

None

Outbound network connections
----------------------------

* DNS (53) resolving nameservers 172.16.2.2 and 172.16.2.3
* :doc:`emailout` as SMTP relay
* ftp.nl.debian.org as Debian mirror
* security.debian.org for Debian security updates
* crl.cacert.org (rsync) for getting CRLs
* all :ip:v4range:`10.0.0.0/24` and :ip:v4range:`172.16.2.0/24` systems for
  monitoring their services

.. todo:: add IPv6 ranges when they are monitored

Security
========

.. sshkeys::
   :RSA:   df:98:f5:ea:05:c1:47:52:97:58:8f:42:55:d6:d9:b6
   :DSA:   07:2b:10:b1:6d:79:35:0f:83:aa:fc:ba:d6:2f:51:dc
   :ECDSA: 48:46:b1:5a:4e:05:64:8a:c3:76:33:77:20:91:14:70

Non-distribution packages and modifications
-------------------------------------------

* None

Risk assessments on critical packages
-------------------------------------

Icinga and the classic frontend are a bit aged but have a good security track
record.

Apache httpd has a good reputation and is a low risk package.

NRPE is flawed and should be replaced. The risk is somewhat mitigated by
firewalling on :doc:`the infrastructure host <infra02>`.

Critical Configuration items
============================

Keys and X.509 certificates
---------------------------

* :file:`/etc/ssl/certs/monitor.c.o.pem` server certificate
* :file:`/etc/ssl/private/monitor.c.o.priv` server key
* :file:`/etc/ssl/certs/cacert.allcerts.pem` CAcert.org Class 1 and Class 3 CA
  certificates (allowed CA certificates for client certificates and the
  certificate chain for the server certificate)
* :file:`/var/local/ssl/crls/`

.. seealso::

   * :wiki:`SystemAdministration/CertificateList`

CRL fetch job
-------------

The script :file:`/etc/cron.hourly/update-crls` is used to fetch CRLs once per
hour.

Apache httpd configuration
--------------------------

The HTTP and HTTPS VirtualHost configuration is defined in
:file:`/etc/apache2/sites-available/icinga-nossl` and
:file:`/etc/apache2/sites-available/icinga` the HTTP VirtualHost redirects to
the HTTPS VirtualHost.

Icinga configuration
--------------------

The Icinga configuration is stored in the :file:`/etc/icinga/` directory.
Database configuration for IDO is stored in :file:`ido2db.cfg`. The Icinga
classic frontend configuration is stored in :file:`cgi.cfg`. Host and service
configurations are defined in the :file:`objects/` subdirectory.

Tasks
=====

Planned
-------

.. todo:: upgrade to Debian Jessie
.. todo:: switch to Icinga2 and Icingaweb2

Changes
=======

System Future
-------------

* No plans

Additional documentation
========================

.. seealso::

   * :wiki:`PostfixConfiguration`

References
----------

Wiki page for this system
   :wiki:`SystemAdministration/Systems/Monitor`