20b89a218e6c95291390190c3216e728f7de2d83
[cacert-infradocs.git] / docs / systems / monitor.rst
1 .. index::
2 single: Systems; Monitor
3
4 =======
5 Monitor
6 =======
7
8 Purpose
9 =======
10
11 This system hosts an `Icinga`_ instance to centrally monitor the services in
12 the CAcert network (especially for security updates and certificate
13 expiry).
14
15 .. note::
16
17 To access the system you need a client certificate where the first email
18 address in the Subject Distinguished Name field is a cacert.org address.
19 Subject Alternative Names are not checked.
20
21 If you are the administrator of a service please ask the monitor admins to
22 add your system to the monitoring configuration and add you as system
23 contact to allow for notifications and tasks like service outage
24 acknowledgement, adding notes, rescheduling checks or setting downtimes for
25 your service.
26
27 .. _Icinga: https://www.icinga.org/
28
29 Application Links
30 -----------------
31
32 The Icinga classic frontend
33 https://monitor.cacert.org/
34
35 Administration
36 ==============
37
38 System Administration
39 ---------------------
40
41 * Primary: :ref:`people_jandd`
42 * Secondary: None
43
44 Application Administration
45 --------------------------
46
47 +-------------+-----------------------+
48 | Application | Administrator(s) |
49 +=============+=======================+
50 | Icinga | :ref:`people_jandd` |
51 +-------------+-----------------------+
52
53 Contact
54 -------
55
56 * monitor-admin@cacert.org
57
58 Additional People
59 -----------------
60
61 :ref:`people_jandd` and :ref:`people_mario` have :program:`sudo` access on that
62 machine too.
63
64 Basics
65 ======
66
67 Physical Location
68 -----------------
69
70 This system is located in an :term:`LXC` container on physical machine
71 :doc:`infra02`.
72
73 Logical Location
74 ----------------
75
76 :IP Internet: :ip:v4:`213.154.225.230`
77 :IP Intranet: :ip:v4:`172.16.2.18`
78 :IP Internal: :ip:v4:`10.0.0.18`
79 :IPv6: :ip:v6:`2001:7b8:616:162:2::18`
80 :MAC address: :mac:`00:ff:73:b3:17:43` (eth0)
81
82 .. seealso::
83
84 See :doc:`../network`
85
86 DNS
87 ---
88
89 .. index::
90 single: DNS records; Monitor
91
92 =================== ======== =========================
93 Name Type Content
94 =================== ======== =========================
95 monitor.cacert.org. IN CNAME infrastructure.cacert.org
96 =================== ======== =========================
97
98 .. seealso::
99
100 See :wiki:`SystemAdministration/Procedures/DNSChanges`
101
102 Operating System
103 ----------------
104
105 .. index::
106 single: Debian GNU/Linux; Stretch
107 single: Debian GNU/Linux; 9.4
108
109 * Debian GNU/Linux 9.4
110
111 Applicable Documentation
112 ------------------------
113
114 This is it :-)
115
116 .. seealso::
117
118 :ref:`Setup package update monitoring for a new container
119 <setup_apt_checking>`
120
121 Services
122 ========
123
124 Listening services
125 ------------------
126
127 +----------+---------+---------+-----------------------------+
128 | Port | Service | Origin | Purpose |
129 +==========+=========+=========+=============================+
130 | 22/tcp | ssh | ANY | admin console access |
131 +----------+---------+---------+-----------------------------+
132 | 25/tcp | smtp | local | mail delivery to local MTA |
133 +----------+---------+---------+-----------------------------+
134 | 80/tcp | http | ANY | Icinga classic web frontend |
135 +----------+---------+---------+-----------------------------+
136 | 443/tcp | https | ANY | Icinga classic web frontend |
137 +----------+---------+---------+-----------------------------+
138 | 5666/tcp | nrpe | monitor | remote monitoring service |
139 +----------+---------+---------+-----------------------------+
140 | 5432/tcp | pgsql | local | PostgreSQL database for IDO |
141 +----------+---------+---------+-----------------------------+
142
143 .. note::
144
145 The ssh port is reachable via NAT on infrastructure.cacert.org:11822
146
147
148 Running services
149 ----------------
150
151 .. index::
152 single: apache httpd
153 single: cron
154 single: icinga
155 single: ido2db
156 single: nrpe
157 single: openssh
158 single: postfix
159 single: postgresql
160 single: puppet agent
161 single: rsyslog
162
163 +--------------------+--------------------+----------------------------------------+
164 | Service | Usage | Start mechanism |
165 +====================+====================+========================================+
166 | Apache httpd | Webserver for | init script |
167 | | Icinga classic | :file:`/etc/init.d/apache2` |
168 +--------------------+--------------------+----------------------------------------+
169 | cron | job scheduler | init script :file:`/etc/init.d/cron` |
170 +--------------------+--------------------+----------------------------------------+
171 | Icinga | Icinga monitoring | init script |
172 | | daemon | :file:`/etc/init.d/icinga` |
173 +--------------------+--------------------+----------------------------------------+
174 | IDO2DB | IDO database | init script |
175 | | writer daemon | :file:`/etc/init.d/ido2db` |
176 +--------------------+--------------------+----------------------------------------+
177 | Nagios NRPE server | remote monitoring | init script |
178 | | service by | :file:`/etc/init.d/nagios-nrpe-server` |
179 | | this system itself | |
180 +--------------------+--------------------+----------------------------------------+
181 | openssh server | ssh daemon for | init script :file:`/etc/init.d/ssh` |
182 | | remote | |
183 | | administration | |
184 +--------------------+--------------------+----------------------------------------+
185 | Postfix | SMTP server for | init script |
186 | | local mail | :file:`/etc/init.d/postfix` |
187 | | submission | |
188 +--------------------+--------------------+----------------------------------------+
189 | PostgreSQL | PostgreSQL | init script |
190 | | database server | :file:`/etc/init.d/postgresql` |
191 | | for IDO | |
192 +--------------------+--------------------+----------------------------------------+
193 | Puppet agent | configuration | init script |
194 | | management agent | :file:`/etc/init.d/puppet` |
195 +--------------------+--------------------+----------------------------------------+
196 | rsyslog | syslog daemon | init script |
197 | | | :file:`/etc/init.d/syslog` |
198 +--------------------+--------------------+----------------------------------------+
199
200 Databases
201 ---------
202
203 +------------+--------+-----------------+
204 | RDBMS | Name | Used for |
205 +============+========+=================+
206 | PostgreSQL | icinga | Icinga IDO data |
207 +------------+--------+-----------------+
208
209 Connected Systems
210 -----------------
211
212 None
213
214 Outbound network connections
215 ----------------------------
216
217 * :doc:`infra02` as resolving nameserver
218 * :doc:`emailout` as SMTP relay
219 * :doc:`puppet` (tcp/8140) as Puppet master
220 * :doc:`proxyout` as HTTP proxy for APT
221 * crl.cacert.org (rsync) for getting CRLs
222 * all :ip:v4range:`10.0.0.0/24` and :ip:v4range:`172.16.2.0/24` systems for
223 monitoring their services
224
225 .. todo:: add IPv6 ranges when they are monitored
226
227 Security
228 ========
229
230 .. sshkeys::
231 :RSA: SHA256:8iOQQGmuqi4OrF2Qkqt9665w8G7Dwl6U9J8bFfYz7V0 MD5:df:98:f5:ea:05:c1:47:52:97:58:8f:42:55:d6:d9:b6
232 :DSA: SHA256:Sh/3OWrodFWc8ZbVTV1/aJDbpt5ztGrwSSWLECTNrOI MD5:07:2b:10:b1:6d:79:35:0f:83:aa:fc:ba:d6:2f:51:dc
233 :ECDSA: SHA256:GWvYqhQUt9INh/7VRVu6Z2YORoy/YzgBxNBmX+ZvMsk MD5:48:46:b1:5a:4e:05:64:8a:c3:76:33:77:20:91:14:70
234 :ED25519: SHA256:L5roC867bvxDJ0ckbhIQOt2A9Nh1RQBVuIJFWwrPLG0 MD5:10:94:56:09:5b:a2:28:ab:11:e0:0f:6e:e4:0c:38:bb
235
236
237 Non-distribution packages and modifications
238 -------------------------------------------
239
240 The Puppet agent package and a few dependencies are installed from the official
241 Puppet APT repository because the versions in Debian are too old to use modern
242 Puppet features.
243
244 Risk assessments on critical packages
245 -------------------------------------
246
247 Icinga and the classic frontend are a bit aged but have a good security track
248 record.
249
250 Apache httpd has a good reputation and is a low risk package.
251
252 NRPE is flawed and should be replaced. The risk is somewhat mitigated by
253 firewalling on :doc:`the infrastructure host <infra02>`.
254
255 The system uses third party packages with a good security track record and
256 regular updates. The attack surface is small due to the tightly restricted
257 access to the system. The puppet agent is not exposed for access from outside
258 the system.
259
260 Critical Configuration items
261 ============================
262
263 The system configuration is managed via Puppet profiles. There should be no
264 configuration items outside of the Puppet repository.
265
266 .. todo:: move configuration of :doc:`monitor` to Puppet code
267
268 Keys and X.509 certificates
269 ---------------------------
270
271 .. sslcert:: monitor.cacert.org
272 :altnames: DNS:monitor.cacert.org, DNS:monitor.intra.cacert.org
273 :certfile: /etc/ssl/certs/monitor.c.o.pem
274 :keyfile: /etc/ssl/private/monitor.c.o.priv
275 :serial: 1381FF
276 :expiration: Mar 16 11:41:06 2020 GMT
277 :sha1fp: 64:34:16:0D:2C:1B:38:5D:61:38:17:6E:D5:1B:90:B9:CF:DC:A9:75
278 :issuer: CA Cert Signing Authority
279
280 * :file:`/etc/ssl/certs/cacert.allcerts.pem` CAcert.org Class 1 and Class 3 CA
281 certificates (allowed CA certificates for client certificates and the
282 certificate chain for the server certificate)
283 * :file:`/var/local/ssl/crls/`
284
285 .. seealso::
286
287 * :wiki:`SystemAdministration/CertificateList`
288
289 CRL fetch job
290 -------------
291
292 The script :file:`/etc/cron.hourly/update-crls` is used to fetch CRLs once per
293 hour.
294
295 Apache httpd configuration
296 --------------------------
297
298 The HTTP and HTTPS VirtualHost configuration is defined in
299 :file:`/etc/apache2/sites-available/icinga-nossl` and
300 :file:`/etc/apache2/sites-available/icinga` the HTTP VirtualHost redirects to
301 the HTTPS VirtualHost.
302
303 Icinga configuration
304 --------------------
305
306 The Icinga configuration is stored in the :file:`/etc/icinga/` directory.
307 Database configuration for IDO is stored in :file:`ido2db.cfg`. The Icinga
308 classic frontend configuration is stored in :file:`cgi.cfg`. Host and service
309 configurations are defined in the :file:`objects/` subdirectory.
310
311 Tasks
312 =====
313
314 Planned
315 -------
316
317 .. todo:: switch to Icinga2 and Icingaweb2
318
319 Changes
320 =======
321
322 System Future
323 -------------
324
325 * No plans
326
327 Additional documentation
328 ========================
329
330 .. seealso::
331
332 * :wiki:`PostfixConfiguration`
333
334 References
335 ----------
336
337 Wiki page for this system
338 :wiki:`SystemAdministration/Systems/Monitor`