Table of content

• man pages
• Systemctl
• Journalctl
• Hostnamectl
• Systemd escape
• Systemd tmpfiles
• Systemd analyze
• In unit files
  • Overwrite unit files
    • Create Overwrite
    • Revert Overwrite
    • Finding Overwirtes
    • Cating Unit with overwrite
• Systemd Path
  • Watch methode
  • Path Unit File
  • Enabling units
• Transient timer units
• Systemd Networkd
  • VLAN
    • VLAN Preperation
    • VLAN Confirugation
      • VLAN NetDev
      • VLAN Physical Interface
      • VLAN Interface
• Enable debug log
• Errors
  • Service is not stopping due to systemd tty ask password agent

man pages

As we all know, systemd comes with 2 or 3 man pages ;) , to go through all them will take a while if you look for something specific like an attritbure to limit the cpu ussage for a service. If you know the attribute name already, then it gets easier, because then you can execute this and just search for your arrtibute:

$ man systemd.directives

This man page contains all (or at least nearly all) attributes which you could need and points to the man page which contains it.

Systemctl

Journalctl

Hostnamectl

Systemd escape

One of the most amasing helper tools from systemd, the systemd-escape!

It helps you in escaping one or multiple strings so that systemd understands them correctly and it can also converts them back.

If you ask your self now, why this is amazing, well, with that one, you can easeliyly get the right units from the string, or e.g. if you want to get the path from the mount unit super fast.

Copied from man systemd-escape

• To escape a single string:

  $ systemd-escape 'Hallöchien, Popöchien'
  Hall\xc3\xb6chien\x2c\x20Pop\xc3\xb6chien
  

• To undo escaping on a single string:

  $ systemd-escape -u 'Hall\xc3\xb6chien\x2c\x20Pop\xc3\xb6chien'
  Hallöchien, Popöchien
  

• To generate the mount unit for a path:

  $ systemd-escape -p --suffix=mount "/tmp//waldi/foobar/"
  tmp-waldi-foobar.mount
  

• To generate instance names of three strings:

  $ systemd-escape --template=systemd-nspawn@.service 'My Container 1' 'containerb' 'container/III'
  systemd-nspawn@My\x20Container\x201.service systemd-nspawn@containerb.service systemd-nspawn@container-III.service
  

• To extract the instance part of an instantiated unit:

  $ systemd-escape -u --instance 'systemd-nspawn@My\x20Container\x201.service'
  My Container 1
  

• To extract the instance part of an instance of a particular template:

  $ systemd-escape -u --template=systemd-nspawn@.service 'systemd-nspawn@My\x20Container\x201.service'
  My Container 1
  

Systemd tmpfiles

It is possible to combine --create, --clean, and --remove in one invocation (in which case removal and cleanup are executed before creation of new files). For example, during boot the following command line is executed to ensure that all temporary and volatile directories are removed and created according to the configuration file:

$ systemd-tmpfiles --remove --create

Systemd resolve

Resolve domain names, IPV4 and IPv6 addresses, DNS resource records, and services

$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (ens2)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.100.3
                      8.8.8.8
          DNS Domain: maas

Systemd analyze

systemd-analyze may be used to determine system boot-up performance statistics and retrieve other state and tracing information from the system and service manager, and to verify the correctness of unit files. It is also used to access special functions useful for advanced system manager debugging.

In unit files

Overwrite unit files

systemd allows you to create so called overwrites for units. These are files placed in a direcotry which has the same name as the unit + .d at the end and allow you to overwrite in there attributes form the original unit, as well as new attribtes can be added with these. This is ery helpful, if you want to stick to the original unit file, but e.g. just want to change some small things and without maintaining always the full service file.

Create Overwrite

To create an overwrite, you can use the command systemctl edit <unit>. This will open your beloved editor (hopefully not another one). Depending on the systemd verison you have, the content will look different. In older systemd versions, you just got a blank file opened. In newer version, you will also the content from the original unit.

And after you are done with your changes and have reloaded systemd, you are ready to use it.

Revert Overwrite

To revert a overwirte file, you guessed it right, use systemctl revert <unit>, reload systemd and it is gone like nothing has ever happened.

Finding Overwirtes

You mask askyour self now, how do you know for wich server you have create an overwrite. Don’t worry, you don’t need to start now a find command and filter thourgh till you got what you want. Systemd offers you the command systemd-delta. This will display you all your overwirtes on the system.

Cating Unit with overwrite

A very usefull feature of systemctl cat is that it also displays changes coming from the overwirtes.

Systemd Path

Path units allow you to trigger a service when an event happens in the filesystem, say, when a file gets deleted or a directory accessed.

A systemd path unit takes the extension .path, and it monitors a file or directory. A .path unit calls another unit (usually a .service unit with the same name) when something happens to the monitored file or directory. For example, if you have a picchanged.path unit to monitor the snapshot from your webcam, you will also have a picchanged.service that will execute a script when the snapshot is overwritten.

Watch methode

Path units contain a new section, [Path], with few more directives. First, you have the what-to-watch-for directives:

• PathExists= monitors whether the file or directory exists. If it does, the associated unit gets triggered. PathExistsGlob= works in a similar fashion, but lets you use globbing, like when you use ls *.jpg to search for all the JPEG images in a directory. This lets you check, for example, whether a file with a certain extension exists.
• PathChanged= watches a file or directory and activates the configured unit whenever it changes. It is not activated on every write to the watched file but only when a monitored file open for for writing is changed and then closed. The associated unit is executed when the file is closed.
• PathModified=, on the other hand, does activate the unit when anything is changed in the file you are monitoring, even before you close the file.
• DirectoryNotEmpty= does what it says on the box, that is, it activates the associated unit if the monitored directory contains files or subdirectories.

Then, we have Unit= that tells the .path which .service unit to activate, in case you want to give it a different name to that of your .path unit; MakeDirectory= can be true or false (or 0 or 1, or yes or no) and creates the directory you want to monitor before monitoring starts. Obviously, using MakeDirectory= in combination with PathExists= does not make sense. However, MakeDirectory= can be used in combination with DirectoryMode=, which you use to set the the mode (permissions) of the new directory. If you don’t use DirectoryMode=, the default permissions for the new directory are 0755.

Path Unit File

All these directives are very useful, but you will be just looking for changes made to one single file/directory, so your .path unit is very simple (/etc/systemd/system/dev_dir.path):

[Unit]
Description=Monitor the file/dir for changes

[Path]
PathModified=/dev

[Install]
WantedBy=multi-user.target

As you haven’t included a Unit= directive in your .path, the unit systemd expects a matching dev_dir.service unit which it will trigger when /dev gets modified:

[Unit]
Description=Executes script when a file has changed.

[Service]
Type=simple
ExecStart=/bin/chmod 755 /dev

[Install]
WantedBy=multi-user.target

Enabling units

After you have generated the files, you can just run the enable and the start and it will monitor the destinations

$ systemctl enable dev_dir.{path,service}
$ systemctl start dev_dir.path

Transient timer units

One can use systemd-run to create transient .timer units. That is, one can set a command to run at a specified time without having a service file. For example the following command touches a file after 30 seconds:

$ systemd-run --on-active=30 /bin/touch /tmp/foo

One can also specify a pre-existing service file that does not have a timer file. For example, the following starts the systemd unit named someunit.service after 12.5 hours have elapsed:

$ systemd-run --on-active="12h 30m" --unit someunit.service

These transient timer/service units can not be find in systemctl list-timers. You have to use systemctl list-units to find it.

See man systemd-run for more information and examples.

Systemd Networkd

VLAN

VLAN Preperation

Before starting to configure the VLAN(s), you should have a look at the following points:

• 8021q: To create VLAN(s) ontop of an existing network interface you need to ensure first, that the 8021q module is loaded in your kernal. To verify that the module is arleady loaded us the below command:

  $ lsmod | grep 8021q
  

  If you don’t get any result on that, then module is not loaded and you would need to do that.

  To load it just now for this runtime, use the command:

  $ modprobe 8021q
  

  To persistent this, add the module 8021q beneath /etc/modules-load.d inside a new file e.g. vlan.conf.

• VLAN on Devices: Next what you need to ensure is that your Networkdevice where your client is atached to, is able to work with VLAN(s) and that the correct VLAN Number(s) are enabled on that port.

• Emergency Way back: Make sure that you have an additional network interface avaiable and working, or that you are able to get a termianl session via a physical screen or what ever.

After everything is prepared, you can start with the configuration.

VLAN Confirugation

VLAN NetDev

First of all, you will need one or more VLAN configuration file(S), depends on how many VLAN(s) you need and your running network setup.

VLAN(s) get specified in .netdev files beneath /etc/systemd/network

Each VLAN gets its own file like 00-<VLANNAME>.netdev, for example /etc/systemd/network/00-veth0.42.netdev

These files will look something lime this:

[NetDev]
Name=<VLANNAME>
Kind=vlan

[VLAN]
Id=<VLANID>

Sample:

[NetDev]
Name=veth0.42
Kind=vlan

[VLAN]
Id=42

The same was done for the VLANs 21 and 84 on eth0, as well as 13 and 37 on the eth1 interface

VLAN Physical Interface

Now it depends on your setup at home and how you want to continue.

Either you can use the interface still as a normal interface with an attached IP, or you don’t use the low level network interface any more.

Both of the methods start in the same way.

You need to create a .network file beneath /etc/systemd/network like 10-<NICNAME>.network`` ( Sample: /etc/sytstemd/network/10-eth0.network`

Also the beginning of the file is the same and looks like this:

[Match]
Name=<NICNAME>
Type=ether

[Network]
Description=<Discription as you like>
VLAN=<VLANNAME1>
VLAN=<VLANNAME2>
VLAN=<VLANNAME..X>

Now you need to know what to do, either using it or not

Continue using the low level interface

In this case, you can jsut add the Address, Gateway and DNS attribute beneath the VLAN attribute(s) and and ensure that the VLAN ID is available through your Networkdeivce as the untaged default VLAN.

[Match]
Name=<NICNAME>
Type=ether

[Network]
Description=<Discription as you like>
VLAN=<VLANNAME1>
VLAN=<VLANNAME2>
VLAN=<VLANNAME..X>

Address=<IP-CIDR>
Gateway=<Gateway-IP>
DNS=<DNS-IP>

Sample

[Match]
Name=eth0
Type=ether

[Network]
Description=Main interface incl. VLAN(s)
VLAN=veth0.21
VLAN=veth0.42
VLAN=veth0.84

Address=10.21.42.84/24
Gateway=10.21.42.1
DNS=10.21.42.2

Stop using the low level interface

In this case, you will need to disable all the autoconfiguration which is turned on by default like and ensure that there is no [Address] block as well as no Address/Gateway/DNS attribute for this interface available.

[Match]
Name=<NICNAME>
Type=ether

[Network]
Description=<Discription as you like>
VLAN=<VLANNAME1>
VLAN=<VLANNAME2>
VLAN=<VLANNAME..X>

LinkLocalAddressing=no
LLDP=no
EmitLLDP=no
IPv6AcceptRA=no
IPv6SendRA=no

Sample

[Match]
Name=eth1
Type=ether

[Network]
Description=Backup phsical VLAN(s) only
VLAN=veth1.13
VLAN=veth1.37

LinkLocalAddressing=no
LLDP=no
EmitLLDP=no
IPv6AcceptRA=no
IPv6SendRA=no

VLAN Interface

Next step is to confirugre the VLAN it self, which is again done via .network file beneath /etc/systemd/network like /etc/systemd/network/20-<VLANNAME>.network (e.g. /etc/systemd/network/20-veth0.42)

This is the same as we are used to configure our interface for networkd, except that we use as Type vlan.

[Match]
Name=<VLANNAME>
Type=vlan

[Network]
Description=<VLAN interface desctiption>

[Address]
Address=<IP-CIDR>
Gateway=<Gateway-IP>
DNS=<DNS-IP>

After you did that for each new VLAN, you are ready to restart your network

$ systemctl restart systemd-networkd

Now there are several ways to display what you have done, pick one of them and be amused about the beautiful nature of VLANs:

• ip a
• networkctl list
• cat /proc/net/vlan/config

Enable debug log

To enable the debug log for systemd (internal) services you would have to add the SYSTEMD_LOG_LEVEL variable to the unit file in the [Service] section like this:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

An easy way to get it in is to use systemctl edit systemd-<unitname> which will create an overwrite file.

After you have added it, dont forget to reload and restart the service.

$ systemctl daemon-reload
$ systemctl restart systemd-<unitname>

When you are done with your analysis, just revert the changes again.

$ systemctl revert systemd-<unitname>
$ systemclt restart systemd-<unitname>

And you are back to the original state.

Errors

Service is not stopping due to systemd tty ask password agent

If you are expirincing that a service is not stoping without any reason, it might be that the systemd internal states got out of sync. To get it back from this state, just perform a systemd daemon-reload, but lets first have a look if it is the root cause:

We in our sample, we want to stop our database service:

$ systemctl stop postgresql.service

but the shell does not come back, so lets open another one on the same host and see what is going on there

The logs of postgres only show that the db is going down, but nothing more, same is shown in the systemd log.

Lets get the pid and trace what is going on:

$ ps afxj | grep stop
 7377 11445 11445  7376 pts/0    11445 S+   0   0:00   systemctl stop postgresql.service

Lets use the pid from above as a parameter to get all childs with pstree

$ pstree -p 11445
systemctl(11445)───systemd-tty-ask(11446)

So only one child process which is the systemd-tty-ask-password-agent and comparing that with the list of jobs from systemd:

$ systemctl list-jobs
      JOB UNIT               TYPE STATE
116037393 postgresql.service stop running

1 jobs listed.

You could see that others are passing through and finishing but that one stays.

Also an strace -fp 11446 does not result in anything, it just waits.

Lets give it a try and run the daemon-reload

$ systemctl daemon-reload
$ systemctl list-jobs
      JOB UNIT                        TYPE  STATE
116070360 apt-daily.service           start running
116069958 update_ssh_keystore.service start running

2 jobs listed.

And it finished the command, but it looks like it just died :D

systemctl status postgresql.service
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/etc/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2023-04-07 09:22:03 UTC; 14s ago

Now you should have your shell back and can start the service again.