Backing up permissions on directory

Before doing anything in Linux, it is also smart to have a rollback plan. Making blanket, recursive permission changes on a directory would certainly fall into this category!

Lets say you found a directory on your system where the file permissions were all 777, so you want to secure them a bit by changing the permissions over to 644. Something like:

[[email protected] ~]# find /var/www/vhosts/domain.com -type f -perm 0777 -print -exec chmod 644 {} \;

The paranoid among us will want to ensure we can revert things back to the way they were before. Thankfully there are two commands that can be used to either backup or restore permissions on a directory recursively: getfacl and setfacl

To backup all the permissions and ownerships within a given directory such as /var/www/vhosts/domain.com, do the following:

[[email protected] ~]# cd /var/www/vhosts/domain.com
[[email protected] ~]# getfacl -R . > permissions_backup

Now lets say you ran the find command, changed everything over to 644, then realized you broke your application cause it needed some files to be 664 or something, so you just want to roll back so you can investigate what happened.

You can roll back the permissions by running:

[[email protected] ~]# cd /var/www/vhosts/domain.com
[[email protected] ~]# setfacl --restore=permissions_backup

Backup entire servers permissions

If you wanted to backup the entire server’s permissions, you can do that by:

[[email protected] ~]# getfacl -R --absolute-names / > server_permissions_backup

And the restoration process remains the same:

[[email protected] ~]# setfacl --restore=server_permissions_backup

Find command examples

This is just a quick reference page for using find to do basic things.

Find is a pretty powerful tool that accepts a bunch of options for narrowing down your search. Some basic examples of stuff you can do are below.

Find a specific file type and extension older than 300 days and remove them

This will find files:
– Older than 300 days
– Is a file
– Match *.jpg
– Will not go into sub directories

This also works for those pesky directories that have millions of files.

First, always confirm the command will work before blindly removing files:

[[email protected] ~]# cd /path/to/directory
[[email protected] ~]# find . -maxdepth 1 -type f -name '*.jpg' -mtime +300 | xargs ls -al

Once you verified that the files displayed are the ones you want removed, remove them by running:

[[email protected] ~]# cd /path/to/directory
[[email protected] ~]# find . -maxdepth 1 -type f -name '*.jpg' -mtime +300 | xargs rm -f

Find files with 777 permissions

This will find all files that have 777 permissions:

[[email protected] ~]# cd /path/to/directory
[[email protected] ~]# find . -type f -perm 0777 -print

This will find all files that do NOT have 777 permissions

[[email protected] ~]# cd /path/to/directory
[[email protected] ~]# find / -type f ! -perm 777

Find Files with 777 Permissions and change to 644

Use caution with this, this is generally not smart to run blindly as it will go into subdirectories unless you set maxdepth.

[[email protected] ~]# cd /path/to/directory
[[email protected] ~]# find . -type f -perm 0777 -print -exec chmod 644 {} \;

Find Directories with 777 Permissions and change to 755

Use caution with this, this is generally not smart to run blindly as it will go into subdirectories unless you set maxdepth.

[[email protected] ~]# cd /path/to/directory
[[email protected] ~]# find . -type d -perm 777 -print -exec chmod 755 {} \;

Find empty directories

[[email protected] ~]# cd /path/to/directory
[[email protected] ~]# find /tmp -type d -empty

Find all hidden files within a directory

[[email protected] ~]# find /path/to/directory -type f -name ".*"

Find files owned by user or group

[[email protected] ~]# cd /path/to/directory
[[email protected] ~]# find /var/www -user apache
[[email protected] ~]# find /var/www -group apache

Find files that were modified in the last 30 days

[[email protected] ~]# find / -mtime 30

Find files that were modified in the last hour

[[email protected] ~]# find / -mmin -60

Find files that were changed within the last hour
Note, this one is specified in minutes only!

[[email protected] ~]# find / -cmin -60

Find files that were accessed in the last 5 days

[[email protected] ~]# find / -atime 5

Find files that were accessed within the last hour
Note, this one is specified in minutes only!

[[email protected] ~]# find / -amin -60

Count files per directory with find
This one is useful when you need to find the top 10 directories that contain the most amount of files.

[[email protected] ~]# vim count-files-per-directory.sh
#!/bin/bash

if [ $# -ne 1 ];then
  echo "Usage: `basename $0` DIRECTORY"
  exit 1
fi

echo "Please wait..."

find "[email protected]" -type d -print0 2>/dev/null | while IFS= read -r -d '' file; do 
    echo -e `ls -A "$file" 2>/dev/null | wc -l` "files in:\t $file"
done | sort -nr | head | awk '{print NR".", "\t", $0}'

exit 0

Now run it against the / directory:

[[email protected] ~]# bash count-files-per-directory.sh /
Please wait...
1. 	 768 files in:	 /usr/share/man/man1
2. 	 631 files in:	 /usr/lib64/python2.6
3. 	 575 files in:	 /usr/share/locale
4. 	 566 files in:	 /usr/share/vim/vim74/syntax
5. 	 496 files in:	 /usr/bin
6. 	 487 files in:	 /usr/share/man/man8
7. 	 393 files in:	 /usr/share/perl5/unicore/lib/gc_sc
8. 	 380 files in:	 /usr/include/linux
9. 	 354 files in:	 /usr/lib64/python2.6/encodings
10. 	 334 files in:	 /usr/share/man/man3

Or if you only need to run the search in a specific directory:

[[email protected] ~]# bash count-files-per-directory.sh /usr/share/man
Please wait...
1. 	 768 files in:	 /usr/share/man/man1
2. 	 487 files in:	 /usr/share/man/man8
3. 	 334 files in:	 /usr/share/man/man3
4. 	 124 files in:	 /usr/share/man/man5
5. 	 49 files in:	 /usr/share/man
6. 	 35 files in:	 /usr/share/man/ru/man8
7. 	 31 files in:	 /usr/share/man/man7
8. 	 27 files in:	 /usr/share/man/fr/man8
9. 	 25 files in:	 /usr/share/man/de/man8
10. 	 22 files in:	 /usr/share/man/ja/man8

Rolling back yum transactions

Ever had the system update a package, which winds up breaking the most random things? How can you roll back? How can you prevent that same buggy package from updating itself again the next time the system checks for updates, yet still get newer versions of that package when its released?

I ran across something like this recently. The symptom was that PHPMyAdmin was no longer working on this LAMP server. In short, it was found that an Apache update was to blame, which was found in this bug report: https://bz.apache.org/bugzilla/show_bug.cgi?id=61202

So how can the update to Apache be rolled back? First, try to confirm that Apache was indeed updated recently:

[[email protected] ~]# tail /var/log/yum.log
Jul 08 04:23:49 Updated: httpd24u-filesystem-2.4.26-1.ius.centos6.noarch
Jul 08 04:23:49 Updated: httpd24u-tools-2.4.26-1.ius.centos6.x86_64
Jul 08 04:23:50 Updated: httpd24u-2.4.26-1.ius.centos6.x86_64
Jul 08 04:23:50 Updated: 1:httpd24u-mod_ssl-2.4.26-1.ius.centos6.x86_64

Now find the transaction ID within yum by running:

[[email protected] ~]# yum history
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
   220 | root               | 2017-07-08 04:23 | Update         |    4

View the details of this transaction by running:

[[email protected] ~]# yum history info 220
...
Transaction performed with:
    Installed     rpm-4.8.0-55.el6.x86_64                       @centos6-x86_64
    Installed     yum-3.2.29-81.el6.centos.noarch               @centos6-x86_64
    Installed     yum-metadata-parser-1.1.2-16.el6.x86_64       @anaconda-CentOS-201410241409.x86_64/6.6
    Installed     yum-plugin-fastestmirror-1.1.30-40.el6.noarch @centos6-x86_64
    Installed     yum-rhn-plugin-2.4.6-1.el6.noarch             @spacewalk
Packages Altered:
    Updated httpd24u-2.4.25-4.ius.centos6.x86_64            @rackspace-centos6-x86_64-ius
    Update           2.4.26-1.ius.centos6.x86_64            @rackspace-centos6-x86_64-ius
    Updated httpd24u-filesystem-2.4.25-4.ius.centos6.noarch @rackspace-centos6-x86_64-ius
    Update                      2.4.26-1.ius.centos6.noarch @rackspace-centos6-x86_64-ius
    Updated httpd24u-mod_ssl-1:2.4.25-4.ius.centos6.x86_64  @rackspace-centos6-x86_64-ius
    Update                   1:2.4.26-1.ius.centos6.x86_64  @rackspace-centos6-x86_64-ius
    Updated httpd24u-tools-2.4.25-4.ius.centos6.x86_64      @rackspace-centos6-x86_64-ius
    Update                 2.4.26-1.ius.centos6.x86_64      @rackspace-centos6-x86_64-ius
history info
...

To roll back the updates, getting us back to Apache 2.4.25 in this case, simple undo the transaction by running:

[[email protected] ~]# yum history undo 220

Then confirm Apache is back to the previous version 2.4.25:

[[email protected] ~]# rpm -qa |grep -i httpd24u
httpd24u-filesystem-2.4.25-4.ius.centos6.noarch
httpd24u-2.4.25-4.ius.centos6.x86_64
httpd24u-mod_ssl-2.4.25-4.ius.centos6.x86_64
httpd24u-tools-2.4.25-4.ius.centos6.x86_64

Next, restart Apache so the changes take place:

[[email protected] ~]# service httpd restart

Finally, exclude the buggy packages from ever being installed again. In this example, Apache 2.4.26 will never be installed, however any newer versions released after that will install/update normally.

[[email protected] ~]# yum install yum-plugin-versionlock
[[email protected] ~]# yum versionlock add! httpd24u-mod_ssl-2.4.26-1.ius.centos6.x86_64 httpd24u-2.4.26-1.ius.centos6.x86_64 httpd24u-tools-2.4.26-1.ius.centos6.x86_64 httpd24u-filesystem-2.4.26-1.ius.centos6.noarch

Setting default kernel in grub2

With newer systems like CentOS 7 and Ubuntu 14.04 and 16.04 using Grub2, you can no longer simply update a single file to have your kernel boot off an older or newer kernel. There are a series of steps that must be followed. The examples below will show how to boot off an older kernel for their respective operating systems.

Please note, the instructions in this article will lock your kernel on whichever one you selected. Even if your system receives automatic kernel updates, those new kernels will have to be manually enabled within grub if you want to use them.

CentOS 7

First, check to see which kernel is currently running:

[[email protected] ~]# uname -r
3.10.0-514.16.1.el7.x86_64

That shows us we’re running 3.10.0-514.16.1, however I need to be running 3.10.0-327.36.3. So to use this specific named kernel, first changed the GRUB_DEFAULT to ‘saved’ in /etc/default/grub by:

[[email protected]~]# cp /etc/default/grub /etc/default/grub.bak
[[email protected]~]# vim /etc/default/grub
...
GRUB_DEFAULT=saved
...

Now create a backup of the grub config for recovery purposes if needed, then rebuild grub:

[[email protected] ~]# cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.bak
[[email protected] ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

Determine what the full kernel name is you want to use. Get the listing by running:

[[email protected]~]# grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2
CentOS Linux (3.10.0-514.16.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-514.2.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.22.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.3.1.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c11d017c89ca4e8685ae3d9791d472ca) 7 (Core)

I want to use the 3.10.0-327.36.3 kernel. Setting that to be the default kernel is simple. Set the desired kernel by running:

[[email protected] ~]# grub2-set-default "CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)"

Now verify that the change got applied in the configs by running:

[[email protected] ~]# grub2-editenv list
saved_entry=CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)

Reboot the system so it boots off the older kernel:

[email protected]:~# reboot

Finally, once the system comes back online, verify the desired kernel is running by:

[[email protected] ~]# uname -r
3.10.0-327.36.3.el7.x86_64

If the system rebooted, and dropped you into a grub shell with an error, you can boot up off of the backup grub.cfg file that was created by:

grub2> configfile (hd0,1)/boot/grub2/grub.cfg.bak

Ubuntu 14.04 and Ubuntu 16.04

First, check to see which kernel is currently running:

[email protected]:~# uname -r
4.4.0-48-generic

That shows us we’re running 4.4.0-48, however I need to be running 4.4.0-47. So to use this specific named kernel, first changed the GRUB_DEFAULT to ‘saved’ in /etc/default/grub by:

[email protected]:~# cp /etc/default/grub /etc/default/grub.bak
[email protected]:~# vim /etc/default/grub
...
GRUB_DEFAULT=saved
...

Now create a backup of the grub config for recovery purposes if needed, then rebuild grub:

[email protected]:~# cp /boot/grub/grub.cfg /boot/grub/grub.cfg.bak
[email protected]:~# update-grub

Determine what the full kernel name is you want to use. Get the listing by running:

[email protected]:~# egrep "^[[:space:]]?(submenu|menuentry)" /boot/grub/grub.cfg | cut -d "'" -f2
Ubuntu
Advanced options for Ubuntu
Ubuntu, with Linux 4.4.0-78-generic
Ubuntu, with Linux 4.4.0-78-generic (recovery mode)
Ubuntu, with Linux 4.4.0-47-generic
Ubuntu, with Linux 4.4.0-47-generic (recovery mode)

I want to use the 4.4.0-47-generic kernel. Setting that to be the default kernel is simple. However, you MUST prepend ‘Advanced options for Ubuntu’ to the kernel name as shown below since Ubuntu makes use of sub menus in the kernel listing. So set the desired kernel by running:

[email protected]:~# grub-set-default 'Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-47-generic'

Now verify that the change got applied in the configs by running:

[email protected]:~# grub-editenv list
saved_entry=Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-47-generic

Reboot the system so it boots off the older kernel:

[email protected]:~# reboot

Finally, once the system comes back online, verify the desired kernel is running by:

[email protected]:~# uname -r
4.4.0-47-generic

If the system rebooted, and dropped you into a grub shell with an error, you can boot up off of the backup grub.cfg file that was created by:

grub2> configfile (hd0,1)/boot/grub2/grub.cfg.bak

Setting up the old-releases repo for ubuntu

This is a guide on enabling the old-releases repos for the Ubuntu project. When Ubuntu marks a release EOL, the mainline repos are moved to an alternate location where they are preserved for historical purposes. It should be noted that when a system reaches EOL, the repos for it are no longer maintained.

It goes without saying that enabling the old-releases repos for EOL systems should only be used as a last resort. Continuing to run a system that has reached the end of life status is dangerous as it no longer receives security patches and bug fixes. There are also no promises that things will continue to work.

Mitigating security and reliability issues can be resolved all together by updating the system to a supported version of the operating system. If there is a compelling reason to enable the old-releases repos so packages can be installed, then proceed below.

Ubuntu 10.04 EOL Repos

Ubuntu 10.04 LTS went EOL on 4/2015. The procedure for setting up the system to use the old-releases repos are below.

First, create a backup of the /etc/apt/sources.list by:

[[email protected] ~]# cp /etc/apt/sources.list /etc/apt/sources.list.bak

Now update /etc/apt/sources.list to point to the old-releases repos accordingly. Keep in mind that there may be repos specified in here for Nginx, Varnish, Docker, etc. So be sure to only update the items needed by Ubuntu. The end result should look something like this:

[[email protected] ~]# vim /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.

deb http://old-releases.ubuntu.com/ubuntu/ lucid main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ lucid universe
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid universe
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates universe
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://old-releases.ubuntu.com/ubuntu/ lucid multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates multiverse

## Uncomment the following two lines to add software from the 'backports'
## repository.
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
# deb http://old-releases.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
# deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://old-releases.ubuntu.com/ubuntu lucid partner
# deb-src http://old-releases.ubuntu.com/ubuntu lucid partner

deb http://old-releases.ubuntu.com/ubuntu lucid-security main restricted
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security main restricted
deb http://old-releases.ubuntu.com/ubuntu lucid-security universe
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security universe
deb http://old-releases.ubuntu.com/ubuntu lucid-security multiverse
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security multiverse

Now refresh the package index from their sources by running:

[[email protected] ~]# apt-get update

Address any 404’s accordingly as that means the URL may be incorrect or may not longer exist.

Ubuntu 12.04 EOL Repos

Ubuntu 12.04 LTS is going EOL on 4/2017. The procedure for setting up the system to use the old-releases repos are below.

First, create a backup of the /etc/apt/sources.list by:

[[email protected] ~]# cp /etc/apt/sources.list /etc/apt/sources.list.bak

Now update /etc/apt/sources.list to point to the old-releases repos accordingly. Keep in mind that there may be repos specified in here for Nginx, Varnish, Docker, etc. So be sure to only update the items needed by Ubuntu. The end result should look something like this:

[[email protected] ~]# vim /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://old-releases.ubuntu.com/ubuntu/ precise main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ precise main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise universe
deb-src http://old-releases.ubuntu.com/ubuntu/ precise universe
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates universe
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise multiverse
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse

deb http://old-releases.ubuntu.com/ubuntu precise-security main restricted
deb-src http://old-releases.ubuntu.com/ubuntu precise-security main restricted
deb http://old-releases.ubuntu.com/ubuntu precise-security universe
deb-src http://old-releases.ubuntu.com/ubuntu precise-security universe
deb http://old-releases.ubuntu.com/ubuntu precise-security multiverse
deb-src http://old-releases.ubuntu.com/ubuntu precise-security multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://old-releases.ubuntu.com/ubuntu precise partner
# deb-src http://old-releases.ubuntu.com/ubuntu precise partner

## Uncomment the following two lines to add software from Ubuntu's
## 'extras' repository.
## This software is not part of Ubuntu, but is offered by third-party
## developers who want to ship their latest software.
# deb http://old-releases.ubuntu.com/ubuntu precise main
# deb-src http://old-releases.ubuntu.com/ubuntu precise main

Now refresh the package index from their sources by running:

[[email protected] ~]# apt-get update

Address any 404’s accordingly as that means the URL may be incorrect or may not longer exist.

Using hpasmcli for HP servers

HP comes with their server utility scripts called hpssacli and hpacucli. These tools allow you to view and modify your hardware configuration on the server. The hpacucli is the older implementation of the toolkit, but the syntax is pretty similar.

HP tools information

To show the firmware version, run:

[[email protected] ~]# hpasmcli -s "show server"

If you want to see extended information, run:

[[email protected] ~]# hpssacli controller all show config detail

General information

To view information regarding the server model, cpu, type, memory, etc, run:

[[email protected] ~]# hpasmcli -s "show server"

Hardware Health

If you want to view the health of the system and chassis components, run:

[[email protected] ~]# hpasmcli -s "show server"

The chassis can also only return specific components, such as:

[[email protected] ~]# hpasmcli -s "show powersupply"
[[email protected] ~]# hpasmcli -s "show dimm"
[[email protected] ~]# hpasmcli -s "show fans"
[[email protected] ~]# hpasmcli -s "show temp"

Storage health

To view the physical and virtual disks on the server:

[[email protected] ~]# hpssacli controller all show
[[email protected] ~]# hpssacli controller slot=3 physicaldrive all show
[[email protected] ~]# hpssacli controller slot=3 logicaldrive all show

On older HP servers, you can view the physical and virtual disks on the server by:

[[email protected] ~]# hpacucli controller slot=1 physicaldrive all show
[[email protected] ~]# hpacucli controller slot=1 logicaldrive all show

To see the storage battery status:

[[email protected] ~]# hpssacli controller all show status

Hardware logs

To display the hardware logs:

[[email protected] ~]# hpasmcli -s "show iml"

If you need to clear the hardware logs:

[[email protected] ~]# hpasmcli -s "clear iml"

CPU actions

To see if hyperthreading is enabled on the CPUs:

[[email protected] ~]# hpasmcli -s "show ht"

If you wanted to change the hyperthreading settings:

# Enable
[[email protected] ~]# hpasmcli -s "enable ht"

# Disable
[[email protected] ~]# hpasmcli -s "disable ht"

Using omreport and omconfig for Dell servers

Dell comes with their server utility scripts called omreport and omconfig. These tools allow you to view and modify your hardware configuration on the server.

Dell tools information

To see what version of the tools your running:

[[email protected] ~]# omreport about details=true

To see if there are updates available for the firmware:

[[email protected] ~]# omreport system version

To see what commands are available using omreport:

[[email protected] ~]# omreport system -?

General information

To view information regarding the server model, cpu type, memory, service tags, etc, run:

[[email protected] ~]# omreport system summary

Hardware Health

If you want to view the health of the system and chassis components, run:

[[email protected] ~]# omreport system

To only get the health information for the chassis:

[[email protected] ~]# omreport chassis

The chassis can also only return specific components, such as:

[[email protected] ~]# omreport chassis fans
[[email protected] ~]# omreport chassis memory
[[email protected] ~]# omreport chassis nics
[[email protected] ~]# omreport chassis processors
[[email protected] ~]# omreport chassis temps
[[email protected] ~]# omreport chassis batteries
[[email protected] ~]# omreport chassis pwrsupplies

Storage health

As a quick note, if the commands below report there are no controllers listed, check to be sure that the software is actually running by:

[[email protected] ~]# /opt/dell/srvadmin/sbin/srvadmin-services.sh status
dell_rbu (module) is stopped
ipmi driver is running
dsm_sa_datamgrd is stopped
dsm_sa_eventmgrd is stopped
dsm_sa_snmpd is stopped
dsm_om_shrsvcd is stopped
dsm_om_connsvcd is stopped
[[email protected] ~]# /opt/dell/srvadmin/sbin/srvadmin-services.sh restart

To view the physical and virtual disks on the server:

[[email protected] ~]# omreport storage pdisk controller=0
[[email protected] ~]# omreport storage vdisk controller=0
[[email protected] ~]# omreport storage pdisk controller=0 vdisk=0

If you just wanted a quick listing of the relevant disk information to see the state of the drives, run:

[[email protected] ~]# omreport storage pdisk controller=0 | grep -iE "^id|^status|name|state|Failure Predicted"
ID                              : 0:0:0
Status                          : Ok
Name                            : Physical Disk 0:0:0
State                           : Online
Failure Predicted               : No
ID                              : 0:0:1
Status                          : Ok
Name                            : Physical Disk 0:0:1
State                           : Online
Failure Predicted               : No

To see if there are any empty drive bays on the server:

[[email protected] ~]# omreport storage controller controller=0 info=pdslotreport | grep 'Empty Slots'

To see the storage battery status:

[[email protected] ~]# omreport storage battery controller=0

Hardware Logs

To display the hardware logs, run:

[[email protected] ~]# omreport system esmlog

If you need to view the alert logs:

[[email protected] ~]# omreport system alertlog

And if you needed to view the messages from the POST:

[[email protected] ~]# omreport system postlog

If you find you need to clear the logs, that can be performed by:

[[email protected] ~]# omconfig system esmlog action=clear
[[email protected] ~]# omconfig system alertlog action=clear
[[email protected] ~]# omconfig system postlog action=clear

CPU actions

To see if hyperthreading is enabled on the CPUs:

[[email protected] ~]# omreport chassis biossetup | grep -A 2 'HyperThreading'

If you wanted to enable hyperthreading:

# Dell R710
[[email protected] ~]# omconfig chassis biossetup attribute=cpuht setting=enabled

# Dell R720
[[email protected] ~]# omconfig chassis biossetup attribute=ProcCores setting=All

If you needed to enable or disable NUMA:

# Disable NUMA:
[[email protected] ~]# omconfig chassis biossetup attribute=numa setting=disabled

# Enable NUMA:
[[email protected] ~]# omconfig chassis biossetup attribute=numa setting=enabled

Apache quick stats

When troubleshooting performance issues with Apache that happened earlier in the day or week, it is very useful to parse the logs quickly to determine quick facts about the inbound connection rates. It may reveal a period of increased traffic that needs to be investigated further, therefore giving you that thread to begin unraveling the problem.

To get the connections per hour for a specific day, run the following:

[[email protected] ~]# grep "27/Feb" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c
  50205 03:00
  90516 04:00
  64837 05:00
  47410 06:00
  44876 07:00
  41098 08:00
  38996 09:00
  37234 10:00
  43704 11:00
  58702 12:00
  58922 13:00
  72592 14:00
  77792 15:00
  88882 16:00
  80815 17:00
  98287 18:00
 617857 19:00
  90507 20:00
  98568 21:00
 147584 22:00
 181814 23:00

Based off that output, there was a massive spike in connections during the 9:00PM hour (19:00). So now lets break the 9:00PM hour down to show the connections per minute:

[[email protected] ~]# grep "27/Feb/2017:19" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'
   1629 19:00
   1664 19:01
   1840 19:02
  10493 19:03
  13728 19:04
  17608 19:05
   1377 19:06
   2333 19:07
   1980 19:08
   2056 19:09
   2123 19:10
...
   1997 19:57
   1631 19:58
   1988 19:59

As shown above, there was some sort of traffic spike that occurred between 9:03PM – 9:05PM. As the window has been narrowed down to a 3 minute period, more specific analysis can be performed. The examples below will focus on what was happening around 9:03PM.

To list the top 10 IP’s accessing the site during around 9:03PM

[[email protected] ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $1}' | sort -nr | uniq -c |sort -nr | head

To list the top most called elements on the site:

[[email protected] ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $7}' | sort -nr | uniq -c | sort -nr | head

Changing your servers timezone

The systems timezone is usually set during installation. If the timezone needed to be changed, it can be done without rebooting the system.

Just be sure to watch for applications like MySQL and PHP that require additional steps for changing the timezone, which are noted near the bottom of this article.

CentOS 5 and CentOS 6

Modify the zone in /etc/sysconfig/clock. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[[email protected] ~]# vim /etc/sysconfig/clock
...
ZONE="America/New_York"
...

Then update /etc/localtime:

[[email protected] ~]# tzdata-update

Now sync the hardware clock against the system time:

[[email protected] ~]# hwclock --systohc

Go ahead and restart syslogd/rsyslogd and crond:

[[email protected] ~]# service crond restart
[[email protected] ~]# service rsyslog restart
[[email protected] ~]# service syslog restart

CentOS 7

Changing the timezone on CentOS 7 can be done with a few commands. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[[email protected] ~]# timedatectl set-timezone America/New_York
[[email protected] ~]# systemctl restart crond
[[email protected] ~]# systemctl restart rsyslog

Ubuntu 12.04 and Ubuntu 14.04

Modify the zone in /etc/timezone. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[[email protected] ~]# vim /etc/timezone
...
America/New_York
...

Now update active timezone:

[[email protected] ~]# dpkg-reconfigure --frontend noninteractive tzdata
Current default time zone: 'America/New_York'
Local time is now:      Tue Jan 17 01:18:04 EST 2017.
Universal Time is now:  Tue Jan 17 06:18:04 UTC 2017.

Restart rsyslog and cron:

[[email protected] ~]# service cron restart
[[email protected] ~]# service rsyslog restart

Ubuntu 16.04

Changing the timezone on Ubuntu 16.04 can be done with a few commands. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[[email protected] ~]# timedatectl set-timezone America/New_York
[[email protected] ~]# systemctl restart crond
[[email protected] ~]# systemctl restart rsyslog

MySQL, Percona and MariaDB

In order for MySQL, Percona, and MariaDB to register the new timezone settings, they need to be restarted. There really isn’t a way around this. As a temporary solution, and one that will not pick up future DST timezone changes, you can manually set the databases current time by:

# List current system date and time via MySQL:
[[email protected] ~]# mysql
mysql> SELECT @@global.time_zone;

# List the current date and time according to MySQL:
mysql> SELECT NOW();

# Update the timezone using the UTC offset:
mysql> SET @@global.time_zone = '+05:00';

Even with this temporary fix in place, unless you are using the UTC timezone, MySQL should be restarted very soon.

PHP

PHP should also have its timezone updated when you change it on the system. Determine where your php.ini resides, then update it accordingly. I am assuming CentOS 6 for this example:

[[email protected] ~]# vim /etc/php.ini
...
date.timezone = "America/New_York"
...

Then restart Apache or PHP-FPM for your system so the changes are applied. Afterwards, test the timezone change to PHP by:

[[email protected] ~]# php -i |grep date.timezone
date/time support => enabled
date.timezone => America/New_York => America/New_York

A list of supported timezone configurations for PHP can be found at:
http://www.php.net/manual/en/timezones.php

Changing the servers hostname

The hostname of a system is usually set during installation. However as time goes on, it may be determined that the hostname should be changed to something more suitable.

Outlined below is the procedure for changing the systems hostname without having to reboot.

CentOS 5 and 6

First, check to see what the existing hostname is on the server:

[[email protected] ~]# hostname

Then change the current hostname. In this example, we’re going to change the hostname to web04.mydomain.com:

[[email protected] ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[[email protected] ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[[email protected] ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Next update your network configuration:

[[email protected] ~]# vim /etc/sysconfig/network
...
HOSTNAME=web04.mydomain.com
...

Restart syslog so the new changes go into effect:

# CentOS 5
[[email protected] ~]# service syslog restart

# CentOS 6
[[email protected] ~]# service rsyslog restart

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/secure and /var/log/messages to ensure the new hostname is being used.

CentOS 7

In this example, we’re going to change the hostname to web04.mydomain.com. First run:

[[email protected] ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[[email protected] ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

Now update the hostname using the systemd command:

[[email protected] ~]# hostnamectl set-hostname web04.mydomain.com

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[[email protected] ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Now restart syslog so the new changes go into effect:

[[email protected] ~]# systemctl restart rsyslog

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.

Ubuntu 12.04 and Ubuntu 14.04

First, check to see what the existing hostname is on the server:

[[email protected] ~]# hostname

In this example, we’re going to change the hostname to web04.mydomain.com. So run:

[[email protected] ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[[email protected] ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[[email protected] ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Then update /etc/hostname accordingly:

[[email protected] ~]# vim /etc/hostname
web04.mydomain.com

Now restart syslog so the new changes go into effect:

[[email protected] ~]# service rsyslog restart

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.

Ubuntu 16.04

First, check to see what the existing hostname is on the server:

[[email protected] ~]# hostname

In this example, we’re going to change the hostname to web04.mydomain.com. So run:

[[email protected] ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[[email protected] ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

Now update the hostname using the systemd command:

[[email protected] ~]# hostnamectl set-hostname web04.mydomain.com

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[[email protected] ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Now restart syslog so the new changes go into effect:

[[email protected] ~]# systemctl restart rsyslog

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.