Service status and logs¶

Still under construction :)¶

One of the main tools a systems administrator has, is reading the logs of the various services and components that are running in a system. It is also the main entry-point for debugging and understanding whether a service is running as configured or not. As a consequence, knowing where to locate logs and how to efficiently extract information from them is of utmost importance for both general IT workflows and this course.

Services¶

In general, a service is a piece of software that shares (serves) content or resources. In most Linux operating systems, a service runs in the background as a daemon, managed by some type of init service - a service to handle other services. For most modern Linux systems, the init service is called systemd (system-daemon) and the main interface to control and manage systemd is called systemctl.

Trivia

In Unix-like systems, background processes are called daemons and by extension any service that runs as a daemon usually has the letter d appended to the process name. By way of example, the OpenSSH service daemon usually runs under the name sshd As another naming convention, the suffix ctl denotes a type of controller, hence the name systemctl as the controller for the system daemon systemd.

Is the service running? (go catch it then)¶

While installation and configuration are usually the first steps to setting up a service, we also need to ensure that the underlying software runs correctly. In most cases, this management is left to systemd

While brief in output, one of the first queries we can run is asking the system controller whether the service is active. For the examples we will be using the Apache web service httpd

[root@myserver ~]# systemctl is-active httpd
inactive
[root@myserver ~]# systemctl start httpd
[root@myserver ~]# systemctl is-active httpd
active

From the example output, we can also see that systemctl is capable of controlling the state of the service. By using the subcommand start, we are able to start the process in question. systemctl has a plethora of subcommands available, below are listed some of the most common ones

#For controlling the state of the service
start
stop
restart

#For checking the status of the service
is-active
is-enabled
status

Status?¶

systemctl status <service name> is probably one of the first commands a sysadmin runs to check what is happening with a particular service. Let's examine the output a bit more carefully

[root@myserver ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
     Active: active (running) since Wed 2024-12-11 12:47:27 EST; 4min 57s ago
       Docs: man:httpd.service(8)
   Main PID: 54760 (httpd)
     Status: "Total requests: 0; Idle/Busy workers 100/0;Requests/sec: 0; Bytes served/sec:   0 B/sec"
      Tasks: 177 (limit: 100207)
     Memory: 40.5M
        CPU: 517ms
     CGroup: /system.slice/httpd.service
             ├─54760 /usr/sbin/httpd -DFOREGROUND
             ├─54761 /usr/sbin/httpd -DFOREGROUND
             ├─54762 /usr/sbin/httpd -DFOREGROUND
             ├─54763 /usr/sbin/httpd -DFOREGROUND
             └─54764 /usr/sbin/httpd -DFOREGROUND

Dec 11 12:47:27 myserver.openstacklocal systemd[1]: Starting The Apache HTTP Server...
Dec 11 12:47:27 myserver.openstacklocal httpd[54760]: Server configured, listening on: port 80
Dec 11 12:47:27 myserver.openstacklocal systemd[1]: Started The Apache HTTP Server.

Take a good look at lines 2 and 3 - the first giving information whether the service is loaded, the path of the service file and finally whether the service is enabled. In this context, enabled means that the service is automatically started during operating system startup. Current output show the service as disabled, meaning it will not autostart after a reboot

The third line in the output conveys the state of the process, whether it is active or not. Currently we are able to see that indeed, the service is active and has already been running for 4min 57s. This basically replaces the query of is-active, and if running in a colored terminal will usually color the output green/red depending on the state.

Further down below, we are also able to see a truncated output of the logs the service writes. For a freshly started and unconfigured service, we only see preliminary logs that it is listening

Dec 11 12:47:27 myserver.openstacklocal systemd[1]: Starting The Apache HTTP Server...
Dec 11 12:47:27 myserver.openstacklocal httpd[54760]: Server configured, listening on: port 80
Dec 11 12:47:27 myserver.openstacklocal systemd[1]: Started The Apache HTTP Server.

Service is broken (getting late, WIP below)¶

After introducing a mistake into our service config, we can try restart it

[root@myserver ~]# systemctl restart httpd
Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xeu httpd.service" for details.

[root@myserver ~]# systemctl status httpd
× httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2024-12-11 13:19:08 EST; 4s ago
   Duration: 31min 39.567s
       Docs: man:httpd.service(8)
    Process: 54964 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
   Main PID: 54964 (code=exited, status=1/FAILURE)
        CPU: 54ms

Dec 11 13:19:08 myserver.openstacklocal systemd[1]: Starting The Apache HTTP Server...
Dec 11 13:19:08 myserver.openstacklocal httpd[54964]: httpd: Syntax error on line 133 of /etc/httpd/conf/httpd.conf: </Directory> directive missing closing '>'
Dec 11 13:19:08 myserver.openstacklocal systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Dec 11 13:19:08 myserver.openstacklocal systemd[1]: httpd.service: Failed with result 'exit-code'.
Dec 11 13:19:08 myserver.openstacklocal systemd[1]: Failed to start The Apache HTTP Server.