Improve system documentation
[cacert-infradocs.git] / docs / systems / monitor.rst
1 .. index::
2 single: Systems; Monitor
3
4 =======
5 Monitor
6 =======
7
8 Purpose
9 =======
10
11 This system hosts an `Icinga`_ instance to centrally monitor the services in
12 the CAcert network (especially for security updates and certificate
13 expiry).
14
15 .. note::
16
17 To access the system you need a client certificate where the first email
18 address in the Subject Distinguished Name field is a cacert.org address.
19 Subject Alternative Names are not checked.
20
21 If you are the administrator of a service please ask the monitor admins to
22 add your system to the monitoring configuration and add you as system
23 contact to allow for notifications and tasks like service outage
24 acknowledgement, adding notes, rescheduling checks or setting downtimes for
25 your service.
26
27 .. _Icinga: https://www.icinga.org/
28
29 Application Links
30 -----------------
31
32 The Icinga classic frontend
33 https://monitor.cacert.org/
34
35 Administration
36 ==============
37
38 System Administration
39 ---------------------
40
41 * Primary: :ref:`people_jandd`
42 * Secondary: None
43
44 Application Administration
45 --------------------------
46
47 +-------------+-----------------------+
48 | Application | Administrator(s) |
49 +=============+=======================+
50 | Icinga | :ref:`people_jandd` |
51 +-------------+-----------------------+
52
53 Contact
54 -------
55
56 * monitor-admin@cacert.org
57
58 Additional People
59 -----------------
60
61 :ref:`people_jandd` and :ref:`people_mario` have :program:`sudo` access on that
62 machine too.
63
64 Basics
65 ======
66
67 Physical Location
68 -----------------
69
70 This system is located in an :term:`LXC` container on physical machine
71 :doc:`infra02`.
72
73 Logical Location
74 ----------------
75
76 :IP Internet: :ip:v4:`213.154.225.230`
77 :IP Intranet: :ip:v4:`172.16.2.18`
78 :IP Internal: :ip:v4:`10.0.0.18`
79 :IPv6: :ip:v6:`2001:7b8:616:162:2::18`
80 :MAC address: :mac:`00:ff:73:b3:17:43` (eth0)
81
82 .. seealso::
83
84 See :doc:`../network`
85
86 .. index::
87 single: Monitoring; Monitor
88
89 Monitoring
90 ----------
91
92 :internal checks: :monitor:`monitor.infra.cacert.org`
93
94 DNS
95 ---
96
97 .. index::
98 single: DNS records; Monitor
99
100 =================== ======== =========================
101 Name Type Content
102 =================== ======== =========================
103 monitor.cacert.org. IN CNAME infrastructure.cacert.org
104 =================== ======== =========================
105
106 .. seealso::
107
108 See :wiki:`SystemAdministration/Procedures/DNSChanges`
109
110 Operating System
111 ----------------
112
113 .. index::
114 single: Debian GNU/Linux; Stretch
115 single: Debian GNU/Linux; 9.4
116
117 * Debian GNU/Linux 9.4
118
119 Applicable Documentation
120 ------------------------
121
122 This is it :-)
123
124 .. seealso::
125
126 :ref:`Setup package update monitoring for a new container
127 <setup_apt_checking>`
128
129 Services
130 ========
131
132 Listening services
133 ------------------
134
135 +----------+---------+---------+-----------------------------+
136 | Port | Service | Origin | Purpose |
137 +==========+=========+=========+=============================+
138 | 22/tcp | ssh | ANY | admin console access |
139 +----------+---------+---------+-----------------------------+
140 | 25/tcp | smtp | local | mail delivery to local MTA |
141 +----------+---------+---------+-----------------------------+
142 | 80/tcp | http | ANY | Icinga classic web frontend |
143 +----------+---------+---------+-----------------------------+
144 | 443/tcp | https | ANY | Icinga classic web frontend |
145 +----------+---------+---------+-----------------------------+
146 | 5666/tcp | nrpe | monitor | remote monitoring service |
147 +----------+---------+---------+-----------------------------+
148 | 5432/tcp | pgsql | local | PostgreSQL database for IDO |
149 +----------+---------+---------+-----------------------------+
150
151 .. note::
152
153 The ssh port is reachable via NAT on infrastructure.cacert.org:11822
154
155
156 Running services
157 ----------------
158
159 .. index::
160 single: apache httpd
161 single: cron
162 single: icinga
163 single: ido2db
164 single: nrpe
165 single: openssh
166 single: postfix
167 single: postgresql
168 single: puppet agent
169 single: rsyslog
170
171 +--------------------+--------------------+----------------------------------------+
172 | Service | Usage | Start mechanism |
173 +====================+====================+========================================+
174 | Apache httpd | Webserver for | init script |
175 | | Icinga classic | :file:`/etc/init.d/apache2` |
176 +--------------------+--------------------+----------------------------------------+
177 | cron | job scheduler | init script :file:`/etc/init.d/cron` |
178 +--------------------+--------------------+----------------------------------------+
179 | Icinga | Icinga monitoring | init script |
180 | | daemon | :file:`/etc/init.d/icinga` |
181 +--------------------+--------------------+----------------------------------------+
182 | IDO2DB | IDO database | init script |
183 | | writer daemon | :file:`/etc/init.d/ido2db` |
184 +--------------------+--------------------+----------------------------------------+
185 | Nagios NRPE server | remote monitoring | init script |
186 | | service by | :file:`/etc/init.d/nagios-nrpe-server` |
187 | | this system itself | |
188 +--------------------+--------------------+----------------------------------------+
189 | openssh server | ssh daemon for | init script :file:`/etc/init.d/ssh` |
190 | | remote | |
191 | | administration | |
192 +--------------------+--------------------+----------------------------------------+
193 | Postfix | SMTP server for | init script |
194 | | local mail | :file:`/etc/init.d/postfix` |
195 | | submission | |
196 +--------------------+--------------------+----------------------------------------+
197 | PostgreSQL | PostgreSQL | init script |
198 | | database server | :file:`/etc/init.d/postgresql` |
199 | | for IDO | |
200 +--------------------+--------------------+----------------------------------------+
201 | Puppet agent | configuration | init script |
202 | | management agent | :file:`/etc/init.d/puppet` |
203 +--------------------+--------------------+----------------------------------------+
204 | rsyslog | syslog daemon | init script |
205 | | | :file:`/etc/init.d/syslog` |
206 +--------------------+--------------------+----------------------------------------+
207
208 Databases
209 ---------
210
211 +------------+--------+-----------------+
212 | RDBMS | Name | Used for |
213 +============+========+=================+
214 | PostgreSQL | icinga | Icinga IDO data |
215 +------------+--------+-----------------+
216
217 Connected Systems
218 -----------------
219
220 None
221
222 Outbound network connections
223 ----------------------------
224
225 * :doc:`infra02` as resolving nameserver
226 * :doc:`emailout` as SMTP relay
227 * :doc:`puppet` (tcp/8140) as Puppet master
228 * :doc:`proxyout` as HTTP proxy for APT
229 * crl.cacert.org (rsync) for getting CRLs
230 * all :ip:v4range:`10.0.0.0/24` and :ip:v4range:`172.16.2.0/24` systems for
231 monitoring their services
232
233 .. todo:: add IPv6 ranges when they are monitored
234
235 Security
236 ========
237
238 .. sshkeys::
239 :RSA: SHA256:8iOQQGmuqi4OrF2Qkqt9665w8G7Dwl6U9J8bFfYz7V0 MD5:df:98:f5:ea:05:c1:47:52:97:58:8f:42:55:d6:d9:b6
240 :DSA: SHA256:Sh/3OWrodFWc8ZbVTV1/aJDbpt5ztGrwSSWLECTNrOI MD5:07:2b:10:b1:6d:79:35:0f:83:aa:fc:ba:d6:2f:51:dc
241 :ECDSA: SHA256:GWvYqhQUt9INh/7VRVu6Z2YORoy/YzgBxNBmX+ZvMsk MD5:48:46:b1:5a:4e:05:64:8a:c3:76:33:77:20:91:14:70
242 :ED25519: SHA256:L5roC867bvxDJ0ckbhIQOt2A9Nh1RQBVuIJFWwrPLG0 MD5:10:94:56:09:5b:a2:28:ab:11:e0:0f:6e:e4:0c:38:bb
243
244
245 Non-distribution packages and modifications
246 -------------------------------------------
247
248 The Puppet agent package and a few dependencies are installed from the official
249 Puppet APT repository because the versions in Debian are too old to use modern
250 Puppet features.
251
252 Risk assessments on critical packages
253 -------------------------------------
254
255 Icinga and the classic frontend are a bit aged but have a good security track
256 record.
257
258 Apache httpd has a good reputation and is a low risk package.
259
260 NRPE is flawed and should be replaced. The risk is somewhat mitigated by
261 firewalling on :doc:`the infrastructure host <infra02>`.
262
263 The system uses third party packages with a good security track record and
264 regular updates. The attack surface is small due to the tightly restricted
265 access to the system. The puppet agent is not exposed for access from outside
266 the system.
267
268 Critical Configuration items
269 ============================
270
271 The system configuration is managed via Puppet profiles. There should be no
272 configuration items outside of the Puppet repository.
273
274 .. todo:: move configuration of :doc:`monitor` to Puppet code
275
276 Keys and X.509 certificates
277 ---------------------------
278
279 .. sslcert:: monitor.cacert.org
280 :altnames: DNS:monitor.cacert.org, DNS:monitor.intra.cacert.org
281 :certfile: /etc/ssl/certs/monitor.c.o.pem
282 :keyfile: /etc/ssl/private/monitor.c.o.priv
283 :serial: 1381FF
284 :expiration: Mar 16 11:41:06 2020 GMT
285 :sha1fp: 64:34:16:0D:2C:1B:38:5D:61:38:17:6E:D5:1B:90:B9:CF:DC:A9:75
286 :issuer: CA Cert Signing Authority
287
288 * :file:`/etc/ssl/certs/cacert.allcerts.pem` CAcert.org Class 1 and Class 3 CA
289 certificates (allowed CA certificates for client certificates and the
290 certificate chain for the server certificate)
291 * :file:`/var/local/ssl/crls/`
292
293 .. seealso::
294
295 * :wiki:`SystemAdministration/CertificateList`
296
297 CRL fetch job
298 -------------
299
300 The script :file:`/etc/cron.hourly/update-crls` is used to fetch CRLs once per
301 hour.
302
303 Apache httpd configuration
304 --------------------------
305
306 The HTTP and HTTPS VirtualHost configuration is defined in
307 :file:`/etc/apache2/sites-available/icinga-nossl` and
308 :file:`/etc/apache2/sites-available/icinga` the HTTP VirtualHost redirects to
309 the HTTPS VirtualHost.
310
311 Icinga configuration
312 --------------------
313
314 The Icinga configuration is stored in the :file:`/etc/icinga/` directory.
315 Database configuration for IDO is stored in :file:`ido2db.cfg`. The Icinga
316 classic frontend configuration is stored in :file:`cgi.cfg`. Host and service
317 configurations are defined in the :file:`objects/` subdirectory.
318
319 Tasks
320 =====
321
322 Changes
323 =======
324
325 Planned
326 -------
327
328 System Future
329 -------------
330
331 * No plans
332
333 Additional documentation
334 ========================
335
336 .. seealso::
337
338 * :wiki:`PostfixConfiguration`
339
340 References
341 ----------
342
343 Wiki page for this system
344 :wiki:`SystemAdministration/Systems/Monitor`