2599c610b831d7cf2ed97e1898213b0a8be3b438
[cacert-infradocs.git] / docs / systems / monitor.rst
1 .. index::
2 single: Systems; Monitor
3
4 =======
5 Monitor
6 =======
7
8 Purpose
9 =======
10
11 This system hosts an `Icinga`_ instance to centrally monitor the services in
12 the CAcert network (especially for security updates and certificate
13 expiry).
14
15 .. note::
16
17 To access the system you need a client certificate where the first email
18 address in the Subject Distinguished Name field is a cacert.org address.
19 Subject Alternative Names are not checked.
20
21 If you are the administrator of a service please ask the monitor admins to
22 add your system to the monitoring configuration and add you as system
23 contact to allow for notifications and tasks like service outage
24 acknowledgement, adding notes, rescheduling checks or setting downtimes for
25 your service.
26
27 .. _Icinga: https://www.icinga.org/
28
29 Application Links
30 -----------------
31
32 The Icinga classic frontend
33 https://monitor.cacert.org/
34
35 Administration
36 ==============
37
38 System Administration
39 ---------------------
40
41 * Primary: :ref:`people_jandd`
42 * Secondary: None
43
44 Application Administration
45 --------------------------
46
47 +-------------+-----------------------+
48 | Application | Administrator(s) |
49 +=============+=======================+
50 | Icinga | :ref:`people_jandd` |
51 +-------------+-----------------------+
52
53 Contact
54 -------
55
56 * monitor-admin@cacert.org
57
58 Additional People
59 -----------------
60
61 :ref:`people_jandd` and :ref:`people_mario` have :program:`sudo` access on that
62 machine too.
63
64 Basics
65 ======
66
67 Physical Location
68 -----------------
69
70 This system is located in an :term:`LXC` container on physical machine
71 :doc:`infra02`.
72
73 Logical Location
74 ----------------
75
76 :IP Internet: :ip:v4:`213.154.225.230`
77 :IP Intranet: :ip:v4:`172.16.2.18`
78 :IP Internal: :ip:v4:`10.0.0.18`
79 :MAC address: :mac:`00:ff:73:b3:17:43` (eth0)
80
81 .. seealso::
82
83 See :doc:`../network`
84
85 DNS
86 ---
87
88 .. index::
89 single: DNS records; Monitor
90
91 =================== ======== =========================
92 Name Type Content
93 =================== ======== =========================
94 monitor.cacert.org. IN CNAME infrastructure.cacert.org
95 =================== ======== =========================
96
97 .. seealso::
98
99 See :wiki:`SystemAdministration/Procedures/DNSChanges`
100
101 Operating System
102 ----------------
103
104 .. index::
105 single: Debian GNU/Linux; Stretch
106 single: Debian GNU/Linux; 9.3
107
108 * Debian GNU/Linux 9.3
109
110 Applicable Documentation
111 ------------------------
112
113 This is it :-)
114
115 .. seealso::
116
117 :ref:`Setup package update monitoring for a new container
118 <setup_apt_checking>`
119
120 Services
121 ========
122
123 Listening services
124 ------------------
125
126 +----------+---------+---------+-----------------------------+
127 | Port | Service | Origin | Purpose |
128 +==========+=========+=========+=============================+
129 | 22/tcp | ssh | ANY | admin console access |
130 +----------+---------+---------+-----------------------------+
131 | 25/tcp | smtp | local | mail delivery to local MTA |
132 +----------+---------+---------+-----------------------------+
133 | 80/tcp | http | ANY | Icinga classic web frontend |
134 +----------+---------+---------+-----------------------------+
135 | 443/tcp | https | ANY | Icinga classic web frontend |
136 +----------+---------+---------+-----------------------------+
137 | 5666/tcp | nrpe | monitor | remote monitoring service |
138 +----------+---------+---------+-----------------------------+
139 | 5432/tcp | pgsql | local | PostgreSQL database for IDO |
140 +----------+---------+---------+-----------------------------+
141
142 .. note::
143
144 The ssh port is reachable via NAT on infrastructure.cacert.org:11822
145
146
147 Running services
148 ----------------
149
150 .. index::
151 single: Apache
152 single: Icinga
153 single: IDO2DB
154 single: Postfix
155 single: PostgreSQL
156 single: cron
157 single: nrpe
158 single: openssh
159
160 +--------------------+--------------------+----------------------------------------+
161 | Service | Usage | Start mechanism |
162 +====================+====================+========================================+
163 | openssh server | ssh daemon for | init script :file:`/etc/init.d/ssh` |
164 | | remote | |
165 | | administration | |
166 +--------------------+--------------------+----------------------------------------+
167 | Apache httpd | Webserver for | init script |
168 | | Icinga classic | :file:`/etc/init.d/apache2` |
169 +--------------------+--------------------+----------------------------------------+
170 | cron | job scheduler | init script :file:`/etc/init.d/cron` |
171 +--------------------+--------------------+----------------------------------------+
172 | rsyslog | syslog daemon | init script |
173 | | | :file:`/etc/init.d/syslog` |
174 +--------------------+--------------------+----------------------------------------+
175 | Icinga | Icinga monitoring | init script |
176 | | daemon | :file:`/etc/init.d/icinga` |
177 +--------------------+--------------------+----------------------------------------+
178 | IDO2DB | IDO database | init script |
179 | | writer daemon | :file:`/etc/init.d/ido2db` |
180 +--------------------+--------------------+----------------------------------------+
181 | PostgreSQL | PostgreSQL | init script |
182 | | database server | :file:`/etc/init.d/postgresql` |
183 | | for IDO | |
184 +--------------------+--------------------+----------------------------------------+
185 | Postfix | SMTP server for | init script |
186 | | local mail | :file:`/etc/init.d/postfix` |
187 | | submission | |
188 +--------------------+--------------------+----------------------------------------+
189 | Nagios NRPE server | remote monitoring | init script |
190 | | service by | :file:`/etc/init.d/nagios-nrpe-server` |
191 | | this system itself | |
192 +--------------------+--------------------+----------------------------------------+
193
194 Databases
195 ---------
196
197 +------------+--------+-----------------+
198 | RDBMS | Name | Used for |
199 +============+========+=================+
200 | PostgreSQL | icinga | Icinga IDO data |
201 +------------+--------+-----------------+
202
203 Connected Systems
204 -----------------
205
206 None
207
208 Outbound network connections
209 ----------------------------
210
211 * DNS (53) resolving nameservers 172.16.2.2 and 172.16.2.3
212 * :doc:`emailout` as SMTP relay
213 * :doc:`proxyout` as HTTP proxy for APT
214 * crl.cacert.org (rsync) for getting CRLs
215 * all :ip:v4range:`10.0.0.0/24` and :ip:v4range:`172.16.2.0/24` systems for
216 monitoring their services
217
218 .. todo:: add IPv6 ranges when they are monitored
219
220 Security
221 ========
222
223 .. sshkeys::
224 :RSA: SHA256:8iOQQGmuqi4OrF2Qkqt9665w8G7Dwl6U9J8bFfYz7V0 MD5:df:98:f5:ea:05:c1:47:52:97:58:8f:42:55:d6:d9:b6
225 :DSA: SHA256:Sh/3OWrodFWc8ZbVTV1/aJDbpt5ztGrwSSWLECTNrOI MD5:07:2b:10:b1:6d:79:35:0f:83:aa:fc:ba:d6:2f:51:dc
226 :ECDSA: SHA256:GWvYqhQUt9INh/7VRVu6Z2YORoy/YzgBxNBmX+ZvMsk MD5:48:46:b1:5a:4e:05:64:8a:c3:76:33:77:20:91:14:70
227 :ED25519: SHA256:L5roC867bvxDJ0ckbhIQOt2A9Nh1RQBVuIJFWwrPLG0 MD5:10:94:56:09:5b:a2:28:ab:11:e0:0f:6e:e4:0c:38:bb
228
229
230 Non-distribution packages and modifications
231 -------------------------------------------
232
233 * None
234
235 Risk assessments on critical packages
236 -------------------------------------
237
238 Icinga and the classic frontend are a bit aged but have a good security track
239 record.
240
241 Apache httpd has a good reputation and is a low risk package.
242
243 NRPE is flawed and should be replaced. The risk is somewhat mitigated by
244 firewalling on :doc:`the infrastructure host <infra02>`.
245
246 Critical Configuration items
247 ============================
248
249 Keys and X.509 certificates
250 ---------------------------
251
252 * :file:`/etc/ssl/certs/monitor.c.o.pem` server certificate
253 * :file:`/etc/ssl/private/monitor.c.o.priv` server key
254 * :file:`/etc/ssl/certs/cacert.allcerts.pem` CAcert.org Class 1 and Class 3 CA
255 certificates (allowed CA certificates for client certificates and the
256 certificate chain for the server certificate)
257 * :file:`/var/local/ssl/crls/`
258
259 .. seealso::
260
261 * :wiki:`SystemAdministration/CertificateList`
262
263 CRL fetch job
264 -------------
265
266 The script :file:`/etc/cron.hourly/update-crls` is used to fetch CRLs once per
267 hour.
268
269 Apache httpd configuration
270 --------------------------
271
272 The HTTP and HTTPS VirtualHost configuration is defined in
273 :file:`/etc/apache2/sites-available/icinga-nossl` and
274 :file:`/etc/apache2/sites-available/icinga` the HTTP VirtualHost redirects to
275 the HTTPS VirtualHost.
276
277 Icinga configuration
278 --------------------
279
280 The Icinga configuration is stored in the :file:`/etc/icinga/` directory.
281 Database configuration for IDO is stored in :file:`ido2db.cfg`. The Icinga
282 classic frontend configuration is stored in :file:`cgi.cfg`. Host and service
283 configurations are defined in the :file:`objects/` subdirectory.
284
285 Tasks
286 =====
287
288 Planned
289 -------
290
291 .. todo:: switch to Icinga2 and Icingaweb2
292
293 Changes
294 =======
295
296 System Future
297 -------------
298
299 * No plans
300
301 Additional documentation
302 ========================
303
304 .. seealso::
305
306 * :wiki:`PostfixConfiguration`
307
308 References
309 ----------
310
311 Wiki page for this system
312 :wiki:`SystemAdministration/Systems/Monitor`