Skip to content

/ devguide

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add info for systemd based distros #583

Open
wants to merge 1 commit into
base: master
from
Open

Add info for systemd based distros #583

wants to merge 1 commit into from
+60 −3

Conversation

@stratakis
Copy link

@stratakis stratakis commented May 15, 2020

Provide examples for managing the buildbot-worker
service through systemd unit files for systemd based
distributions.
@stratakis
Add info for systemd based distros
Loading status checks…
be6a64f 
Provide examples for managing the buildbot-worker
service through systemd unit files for systemd based
distributions.
@Mariatta
Copy link
Member

@Mariatta Mariatta commented May 15, 2020

Note I have just removed the Netlify integration, since we're using readthedocs preview build now.
@tiran
tiran requested changes May 15, 2020
View changes
buildworker.rst
Wants=network.target
After=network.target
Comment on lines +123 to +124

This comment has been minimized.

Sign in to view
@tiran

tiran May 15, 2020
Member

The builder should wait for network online and start after network is online and all local file systems are available:

Wants=network-online.target
After=network-online.target local-fs.target

This comment has been minimized.

Sign in to view
@stratakis

stratakis May 15, 2020
Author

Will amend that
buildworker.rst
Wants=network.target
After=network.target
Comment on lines +144 to +145

This comment has been minimized.

Sign in to view
@tiran

tiran May 15, 2020
Member 

Wants=network-online.target
After=network-online.target local-fs.target

This comment has been minimized.

Sign in to view
@stratakis

stratakis May 15, 2020
Author

Will amend that
buildworker.rst
where the enforcing mode is enabled, usually related to the twistd.pid file.
If the service fails to start, you should check the output of
``systemctl status buildbot-worker.service`` as well as the
``/var/log/audit/audit.log`` file for potential issues and remedies.

This comment has been minimized.

Sign in to view
@tiran

tiran May 15, 2020
Member

ausearch -m AVC

This comment has been minimized.

Sign in to view
@stratakis

stratakis May 15, 2020
Author

I usually use sealert -a /var/log/audit/audit.log which provides messages with the warning and how to fix them, but I thought it could be too specific for the devguide.

This comment has been minimized.

Sign in to view
@tiran

tiran May 19, 2020
Member

Is this really going to be an issue? Unless you define a policy the buildbot daemon should run in unconfined_t and not be affected by SELinux policy restrictions.

This comment has been minimized.

Sign in to view
@stratakis

stratakis May 19, 2020
Author

It was always affected on all the buildbots I've set up (Fedora and RHEL). Basically selinux prevents systemd's open and read access for the twistd.pid file.

This comment has been minimized.

Sign in to view
@stratakis

stratakis May 21, 2020
Author

So after following the instructions on the buildbot worker page to set it up and creating the systemd unit files, when trying to start the service, initially this error message is show (date/time is trimmed out):

`
Following twistd.log until startup finished..
AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost setroubleshoot[1489084]: SELinux is preventing systemd from read access on the file twistd.pid. For complete SELinux messages run: sealert -l 57d3c874-be75-46f6-82c9-3680ac54002c

localhost python3[1489084]: SELinux is preventing systemd from read access on the file twistd.pid.#12#012***** Plugin catchall (100. confidence) suggests **************************#12#012
If you believe that systemd should be allowed read access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012

localhost setroubleshoot[1489084]: SELinux is preventing systemd from read access on the file twistd.pid. For complete SELinux messages run: sealert -l 57d3c874-be75-46f6-82c9-3680ac54002c

localhost python3[1489084]: SELinux is preventing systemd from read access on the file twistd.pid.#12#012***** Plugin catchall (100. confidence) suggests **************************#12#012If you believe that systemd should be allowed read access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012
`
After adjusting the policy, another error comes up:

`
Following twistd.log until startup finished..

localhost audit[1]: AVC avc: denied { open } for pid=1 comm="systemd" path="/home/buildbot/buildarea/twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost audit[1]: AVC avc: denied { open } for pid=1 comm="systemd" path="/home/buildbot/buildarea/twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost setroubleshoot[8714]: SELinux is preventing systemd from open access on the file /home/buildbot/buildarea/twistd.pid. For complete SELinux messages run: sealert -l 3d54c639-fea4-4a18-be47-c5fe1d57a02a

localhost python3[8714]: SELinux is preventing systemd from open access on the file /home/buildbot/buildarea/twistd.pid.#012#012***** Plugin catchall (100. confidence) suggests **************************#12#012If you believe that systemd should be allowed open access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012`

After allowing that as well, the service starts normally.
buildworker.rst

|

* Alternatively you can create a cronjob. Add the following line to ``/etc/crontab``::

This comment has been minimized.

Sign in to view
@tiran

tiran May 15, 2020
Member

systemd timer would also work

This comment has been minimized.

Sign in to view
@stratakis

stratakis May 15, 2020 •
edited
Author

I haven't dealt or used before systemd timers. By doing a quick check, are you proposing to basically add an example of the cron job line to a systemd unit as a timer?

This comment has been minimized.

Sign in to view
@tiran

tiran May 19, 2020
Member

Please ignore my comment. A systemd timer requires a systemd service definition any way.
@tiran
Copy link
Member

@tiran tiran commented May 21, 2020

I see. SELinux and systemd both treat /home as a privileged and restricted area. System services are generally not allowed to access /home tos prevents daemons from stealing ssh private keys, personal emails, or your cat pictures.

I know of four options to deal with this problem

  • move buildbot out of /home. For example deploy buildbot code to /opt/buildbot and use systemd RuntimeDirectory, StateDirectory, and LogsDirectory (/run, /var/log, ... see man systemd.exec)
  • run buildbot as systemd user service. You have to put the service file to /home/buildbot/.config/systemd/user/buildbot.service, enable lingering with loginctl enable-linger buildbot, and start the service from a buildbot login shell (not su/sudo!) as systemctl --user enable --now buildbot.service.
  • create a custom SELinux policy, types, and file contexts for buildbot.
  • make the init_t SELinux type permissive semanage permissive -a init_t
@stratakis
Copy link
Author

@stratakis stratakis commented Jun 9, 2020

I see. SELinux and systemd both treat /home as a privileged and restricted area. System services are generally not allowed to access /home tos prevents daemons from stealing ssh private keys, personal emails, or your cat pictures.

I know of four options to deal with this problem

* move buildbot out of `/home`. For example deploy buildbot code to `/opt/buildbot` and use systemd `RuntimeDirectory`, `StateDirectory`, and `LogsDirectory` (/run, /var/log, ... see man systemd.exec)

That would go against the current guidelines of setting up a buildbot worker (or requiring a bigger overhaul) so I wouldn't go with that option.

* run buildbot as systemd user service. You have to put the service file to `/home/buildbot/.config/systemd/user/buildbot.service`, enable lingering with `loginctl enable-linger buildbot`, and start the service from a buildbot login shell (not su/sudo!) as `systemctl --user enable --now buildbot.service`.

This solution I liked the most, however the systemctl --user option is not available in RHEL7.

* create a custom SELinux policy, types, and file contexts for buildbot.

Maybe that would be the best way then. Any pointers on how to work with that?

* make the init_t SELinux type permissive `semanage permissive -a init_t`

Not sure I would like to change init_t
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tiran tiran

Assignees
No one assigned
Labels
CLA signed needs: PR update
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
@stratakis @Mariatta @tiran @willingc @the-knights-who-say-ni
You can’t perform that action at this time.