Things I don't want to look for twice..

Neutron Rootwrap error on CentOS 7.4

Had to upgrade to CentOS 7.4. Don’t know if it’s related, but it started just after the upgrade. So, at some point in time, neutron-l3-agent.service was filling up the logs with stuff like:

2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent [-] Error while deleting router 98debc3b-4aff-43f0-b99f-83da6e8abfe2
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-15.1.7/lib/python2.7/site-packages/neutron/agent/l3/", line 365, in _safe_router_removed
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent self._router_removed(router_id)
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-15.1.7/lib/python2.7/site-packages/neutron/agent/l3/", line 382, in _router_removed
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent self, router=ri)
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-15.1.7/lib/python2.7/site-packages/neutron/callbacks/", line 44, in notify
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent _get_callback_manager().notify(resource, event, trigger, **kwargs)
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-15.1.7/lib/python2.7/site-packages/neutron/db/", line 182, in wrapped
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent raise db_exc.RetryRequest(e)
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-15.1.7/lib/python2.7/site-packages/oslo_utils/", line 220, in exit
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent self.force_reraise()
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-15.1.7/lib/python2.7/site-packages/oslo_utils/", line 196, in force_reraise
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-15.1.7/lib/python2.7/site-packages/neutron/db/", line 177, in wrapped
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent return f(*args, **kwargs)
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-15.1.7/lib/python2.7/site-packages/neutron/callbacks/", line 127, in notify
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent raise exceptions.CallbackFailure(errors=errors)
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed-5084345 failed with "Exit code: 99; Stdin: ; Stdout: ; Stderr: /openstack/venvs/neutron-15.1.7/bin/neutron-rootwrap: Unauthorized command: kill -9 22983 (no filter matched)
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent "
2017-09-25 11:20:18.516 19686 ERROR neutron.agent.l3.agent

Ok. Long story short, it had to do with its rootwrap script. Story on how the filters are parsed, it’s here:

So I had to edit l3.filters accordingly. First, these are the default filters for metadata kill scripts:

kill_metadata: KillFilter, root, python, -15, -9
kill_metadata7: KillFilter, root, python2.7, -15, -9
kill_metadata35: KillFilter, root, python3.5, -15, -9

This takes into consideration that the OS will see python as the executable. But, well, it doesn’t. It actually sees something like:

lrwxrwxrwx 1 root root 0 Sep 25 10:43 exe -> /openstack/venvs/neutron-15.1.7/bin/python2.7

Fine, so I had to add the following filter:

kill_something: KillFilter, root, /openstack/venvs/neutron-15.1.7/bin/python2.7, -15, -9

Docs on rootwrap can be found here:


SSH Agent forwarding doesn’t work on MacOS Sierra

So, was pretty sure that “ForwardAgent yes” was added all over my SSH config. And that was working fine, on Linux. When I moved all my stuff to my newly bought Mac, well, discovered that it’s not working anymore. So, actually, 2 lines need to be added to each host configuration:

 User xxx
 IdentityFile ~/.ssh/id_rsa_xxx
 ForwardAgent yes
 AddKeysToAgent yes

Skype not working on 64bit OpenSUSE

Symptoms are:

  • no sound at all, neither at startup, nor in calls, test calls, anything
  • all devices in Options > Sound Devices show “virtual device”

This is easy, you need to install a package and restart Skype:

zypper install libpulse0-32bit

I’m currently using OpenSUSE Tumbleweed, but I’ve seen it’s happening to a lot of users, including other distros as well.

top largest / smallest files in a folder

– Largest:

find . -type f -printf '%s %p\n'| sort -nr | head -5

– Smallest:

find . -type f -printf '%s %p\n'| sort -n | head -5

Lustre 2.6 on Debian Wheezy (clients)

I had a shitty job of trying to upgrade Lustre clients to the latest 2.6. I’ve had a lot of bugs on 2.5 some of them were solved in 2.6. My configuration is the following:

– 3 CentOS6 servers (for now)

– 24 Debian Wheezy clients (for now)

Upgrading Lustre on CentOS went pretty smooth. Just found the updated packages and installed. On the other had, Debian clients, let’s say I was not so lucky. So here is what I did:

– installed a Debian 7 x86_64 on a VM (kernel version 3.2.0-4-amd64)

– found a RedHat kernel ported to Debian here (thanks Thomas Stibor) and installed linux-headers and linux-images deb packages, then booted using those:

– cloned branch 2.6 from Lustre, then:

git clone -b b2_6 git://
cd lustre-release/
./configure --enable-dist
make dist

This will create a file called lustre-2.6.0.tar.gz in lustre-release folder. Then:

mkdir BUILD
ln -s ../lustre-2.6.0.tar.gz lustre-2.6.0.orig.tar.gz
tar xzvf ../lustre-2.6.0.tar.gz

Then, the tricky part. Somehow, from the rules and makefiles in there, I found out that if you’re not changing latest version of Lustre in debian/changelog, it will build version 2.6 with a name, something like: Also, somewhere along the way, it also expects a “-” after the version. Not good. For this to change, we’ll add this to the beginning of debian/changelog (only the first line counts):

lustre (2.6.0-1) unstable; urgency=low
  * Update for an on-the-road-to-1.8.2 release
 -- Brian J. Murrell <>  Mon, 17 Aug 2009 14:54:35 -0400

After that, instead of building packages with version it will build packages with the same version as the sources are. I think it’s safe to maybe only change first line’s version and that’s it. Then:

make debs

This will create all necessary deb files and will put them in lustre-release/lustre-2.6.0/debs/ (I’ve also included the kernel debs and e2fsprogs, I needed an archive that can be installed on production servers):


That’s it. On your client machine, you should install:


Should be enough. But first, better test on a VM/test machine. To verify that the wanted version of lustre in installed:

cat /proc/fs/lustre/version 
lustre: 2.6.0
kernel: patchless_client
build:  v2_6_0_0--PRISTINE-2.6.32-rh-431.23.3-lustre-tstibor

Good luck!

Atom git commands not working on OS X Yosemite 10.10 beta 2

What I did was:

1. backup everything git* in /usr/bin

# cd /usr/bin
# mkdir temp
# mv git* temp/

2. copy everything git* from /Applications/ to /usr/bin/

# find /Applications/ -name git\* -maxdepth 1 -exec cp {} /usr/bin/ \;

So now I have:

# ls -Al git* | wc -l

And Atom seems to be working again.

PS I wanted to answer someone on stackoverflow but got lazy into formatting this on their website…

clustered LVM to non-clustered LVM

I had to re-configure clustered LVM to non-clustered LVM for some Lustre setup I’m working on. Thing is, clustered LVM can only be accessible if clvmd service is running. But I couldn’t start it, because I needed all cluster services down. After searching, I found that:

– to check if the LVM is clustered:

vgdisplay vgsrv1 --config 'global {locking_type = 0}' | grep Clustered
 WARNING: Locking disabled. Be careful! This could corrupt your metadata.
 Clustered yes

– to change it back to non-clustered LVM:

vgchange -cn vgsrv1 --config 'global {locking_type = 0}'

Now, to refresh your VG list type:

vgchange -ay

logrotate doesn’t rotate from file exists error

So, I had the following config:

/var/log/remote/*/php/*.log /var/log/remote/*/apache2/*.log { 
 rotate 7
 create 640 rsyslog rsyslog
 /etc/init.d/rsyslog restart > /dev/null

Paths were valid, ownership of folders was root.root, still, I had sample.log which wouldn’t get renamed to sample.log.1 and new file wasn’t created either. Have been sitting on this for days now. Decided to redo the config (was actually a bigger one, removed every comment, removed everything it wasn’t needed for my test and ran logrotate with the new config, forced. And eventually after carefully reading the logs (used logrotate -vf /path/to/conf) I found out that:

error: error creating output file /var/log/remote/something/apache2/error.log.1.gz: File exists

Ok, the file existed. So what? It had the necessary permissions to delete/replace it. But, of course, it didn’t. Moreover, after getting this error, creating other sample.log.1 files didn’t work either so rotation failed. So I deleted that empty file (yes, it was empty) and retried the logrotate on my test config. Of course, it worked. Surely I couldn’t find any documentation on this, so next time it happens, try to run verbosely and check for “file exists” errors. That will explain strange logrotate behavior.


deb package status

As I’m always forgetting the meaning on the letters from “dpkg –list”, here’s the meaning:

First character:

u: Unknown (an unknown state)
i: Install (marked for installation)
r: Remove (marked for removal)
p: Purge (marked for purging)
h: Hold
Second Character:

n: Not- The package is not installed
i: Inst – The package is successfully installed
c: Cfg-files – Configuration files are present
u: Unpacked- The package is stilled unpacked
f: Failed-cfg- Failed to remove configuration files
h: Half-inst- The package is only partially installed
W: trig-aWait
t: Trig-pend

Third Character:

R: Reinst-required The package must be installed.

This was taken from here.

changing PATH for Ansible and Ansible loops

So, I’ve had a problem with trying to run things as ansible and sudo. Of course, password was not required for any of my sudo commands, but I often get “command not found”. Clearly, after running “env” with ansible using sudo on a server, I noticed I was missing /usr/sbin and /sbin from my path, as opposed to root’s PATH with had both folders.

In my case, was a Debian init stript which relied on start-stop-daemon, which is in sbin, and I got the command not found error. Here is how you set it up in a playbook:

 - name: stop monit service
 shell: PATH=$PATH:/usr/sbin:/sbin sudo /etc/init.d/monit stop

Testing showed that:

ansible test-srv -s -m shell -a "PATH=$PATH:/usr/sbin:/sbin env"
test-srv | success | rc=0 >>

Today I also discovered loops. I had to run a bunch of commands on a lot of servers, searched a bit and found loops. So, if you need to run 5 commands on your servers on a single task from a playbook, do the following:

- name: multiple commands
  command: "{{item}}"
    - /path/to/command1
    - /path/to/command2
    - command3
    - command4
    - command5

Path requirement is for the case in which the specified command isn’t in user’s PATH by default and you should specify full path for the command itself.