Big SELinux troubleshooting chart

Milos Malik

mmalik (at) redhat (dot) com

Red Hat Czech

BaseOS QE Securityhttp://bit.ly/bSELtchart

Why?

● High number of reported bugs against selinux-policy

● Some of them are easy to solve● SELinux troubleshooting is not as difficult as

you think● SELinux documentation maybe lacks some

practical steps● Scope = targeted policy● Audience = sysadmin, devel, qe, gss

Benefits

● ability to solve easy issues on your own● improved troubleshooting skills● the chart guides you through the

troubleshooting process● time-tested workflow

Workshop structure

● parts of the chart● whole chart● your questions● let's apply the chart on some bugs● feedback

High-level overview

● Identification → Analysis → Conservative solution → Radical solution → Workaround needed → Problem solved

Problem identification

● ausearch

● audit daemon● dmesg

● /var/log/messages or journal● sealert

● setroubleshoot applet

Analysis

● Is context of the process wrong? (source)● Is context of the object wrong? (target)● Is a type definition missing?● Is a rule missing?● Is there another discrepancy?● Why it happened?● Do we see the root cause or a consequence?

Conservative solutions● Follow the best practises● Enabling / disabling of booleans● Change of network port definitions● Change of file context definitions● Making a domain permissive

Radical solutions● Adding rules for existing policy types● Defining new types● Adding rules for new types● Additional policy modules

Workaround needed

● Constraints and overrides● Where? Kernel, application or selinux-policy● Developer expertise / magic is needed● How to find a workaround?

How to walk through the chart?

● Starting point● Minimum is 2 iterations (scenario executed in

enforcing and in permissive mode)● Step by step approach (which does not switch

whole system to permissive mode) may need more iterations

● Get to the end point

Questions?

Does it work?

● Let's apply the chart on a prepared list of bugs:– BZ#1261309

– BZ#1115601

– BZ#1101028

– BZ#1296238

● Does the audience propose some bugs?

Opportunities

● Creative web presentation designer / implementor wanted

● Feedback is appreciated● The chart will evolve based on new kinds of

bugs or available tools

Detailed slides follow

● Homework study for the audience● Each node of the chart is identified by a unique

number and described

1. is SELinux enabled?

● sestatus

● getenforce

● Enforcing or permissive mode means that SELinux is enabled

2. this chart does not help you

● The issue you encountered is not caused by SELinux

● You can enable SELinux either in /etc/selinux/config file or on kernel command line

3. run your scenario

● You may repeat this step several times, because some issues are difficult to investigate

4. does it work as expected?

● All expected functions of the scenario are present

5. what does ausearch say?

● ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today

● -ts start-time● -te end-time

6. is audit daemon running?

● service auditd status

7. start the audit daemon

● service auditd start

● to know the complete information about an SELinux denial we need to see it in the form of an audit message

● if you add “-w /etc/shadow -p w” to /etc/audit/rules.d/audit.rules file and restart the daemon then audit messages will contain full paths

8. do you see SELinux denials in /var/log/messages or in

journal?

● if audit daemon is not running, but syslog daemon or systemd is running, then SELinux denials are not written into /var/log/audit/audit.log

9. do you see SELinux denials in dmesg output?

● If none of following daemons is running then you can find SELinux denials in the output of dmesg:– audit daemon

– syslog daemon

– systemd

10. remove dontaudit rules

● semodule -DB

● dontaudit rule means that a particular access is denied but there is no record of the SELinux denial

11. is audit2allow available?

● very useful tool which is able to:– provide hints for solution

– summarize existing SELinux denials

– generate a local policy module

12. install policycoreutils-python package

● yum -y install policycoreutils-python

13. is selinux-policy documentation available?

● rpm -q selinux-policy-doc

● rpm -q selinux-policy-devel

14. install selinux-policy documentation

● yum -y install selinux-policy-devel

● yum -y install selinux-policy-doc

15. does audit2allow mention a boolean?

● Is there any SELinux boolean in the output of audit2allow?

16. check the boolean documentation

● semanage boolean -l

● man -K boolean-name

● HTML pages in /usr/share/doc/selinux-policy-*

17. does the documentation describe (part of) your scenario?

● yes - you found the right boolean● no - try another boolean● sometimes is necessary to use more booleans

at the same time

18. customize the boolean

● semanage boolean -m –-on ...

● semanage boolean -m –-off ...

19. does audit2allow mention a network port?

● usually has a suffix _port_t

20. customize port definitions

● semanage port -a -t … -p … portnumber● semanage port -d -t … -p … portnumber● semanage port -l

● which port types exist?– seinfo -t | grep _port_t | sort

21. does audit2why mention a constraint violation?

● useful tool which belongs to the same package as audit2allow

22. report the problemto CP or RHBZ

● CP → access.redhat.com● RHBZ → bugzilla.redhat.com

23. return dontaudit rules if you removed them before

● semodule -B

● one of clean-up tasks

24. return to enforcing modeif you left it before

● setenforce 1

● run for each domain you switched to permissive:– semanage permissive -d ..._t

● another clean-up task

25. does audit2allow mention filesystem objects?

● character device → tclass=chr_file● block device → tclass=blk_file● regular file → tclass=file● directory → tclass=dir● symbolic link → tclass=lnk_file● socket → tclass=sock_file● pipe → tclass=fifo_file

26. are these objects mislabeled?

● each AVC contains a tcontext=... part● matchpathcon -n /path/to/object● objects are mislabeled when these 2

values differ● if matchpathcon returns <<none>> then

use sesearch -T to find out which context did the object have when it was created

27. is reboot acceptable?

● the need for reboot is rare but– if there is a lot of mislabeled objects on the

filesystem, it maybe better to relabel them all during reboot

– if the system does not boot because of SELinux denials then a complete relabel is the only solution

28. run restorecon

● following command corrects labels:– restorecon -Rv /path/to/somewhere

● following command just shows which labels are wrong:– restorecon -Rvn /path/to/somewhere

29. find a better context and customize file context patterns

● use sesearch -A and specify 3 of 4 parameters (-s, -t, -c, -p)

● use semanage fcontext -a -t … -f … pattern

30. does audit2allow mention missing rules?

● no additional hints, just lines which start with allow ...

31. is local policy module acceptable?

● local policy module needs to be maintained

32. create local policy module

● the local policy module will contain macros: ausearch … | audit2allow -R -M mypolicy

● the local policy module will not contain macros: ausearch … | audit2allow -M mypolicy

33. insert the local policy module

● semodule -i mypolicy.pp

● filename and policy module name are 2 different things

● list of currently loaded policy modules: semodule -l

34. make the domain permissive● semanage permissive -a ..._t

● you can switch a chosen type to permissive and leave the rest of types in enforcing mode

● the use of setenforce 0 is advised only in desperate cases and temporarily

35. are there additional SELinux denials?

● use ausearch again and check the time when the SELinux denials appeared

● new SELinux denials may appear even if your scenario works fine → some accesses are really redundant

36. collect additional SELinux denials

● ausearch … > attachment-xyz.txt

● for further inspection by the selinux-policy developers

37. run fixfiles onboot

● the command makes sure that whole filesystem will be relabel during next reboot

38. reboot

● relabeling of filesystems may take several minutes depending on the number of stored objects

● enforcing=0 kernel parameter makes sure that SELinux starts in permissive mode, the mode configured in /etc/selinux/config is ignored

39. collect all constraint violations

● ausearch … | audit2why > attachment-xyz.txt

● for further inspection by the selinux-policy developers

Security vs. usability● SELinux policy should not prevent programs

from doing what is expected from them● programs should only access objects which are

necessary to fulfil their purpose● sometimes the expected and actual behavior

differ significantly● extremes: SELinux policy is either too strict or

too benevolent

Enforcing vs. permissive● the same scenario may involve different code

paths in enforcing and in permissive mode● that's why the sets of SELinux denials triggered

in these modes are usually different● not all SELinux denials triggered in permissive

mode must be fixed in policy● to collect all SELinux denials in enforcing mode

you usually need more than 1 iteration

Searching for the root cause● some of “solved” problems appear again● without knowing the root cause we are solving

consequences● complex interactions among various programs

make the activity more difficult● find a reproducer via

– inspection of relevant log files

– increased logging / enabled debugging messages– auditctl / ausearch

– policy module with special auditallow rules– strace / ltrace / gdb

Local policy is not almighty

● New network port types cannot be defined● New classes and permissions cannot be

defined either● You can disable / remove a policy module as

whole (everything it defines), but you cannot disable / remove a specifically chosen rule

● You cannot make the constraints less strict

Be careful when● enabling too powerful booleans

– *_all_rw

– *_full_access

– nis_enabled

– daemons_enable_cluster_mode

● using general contexts instead of specific ones● your local policy module is too benevolent● your file-systems are mislabeled● allowing permissions like dac_override, dac_read_search, sys_admin, setuid