LinuxSysAdmin

Things I don't want to look for twice..

Category Archives: Uncategorized

SSH Agent forwarding doesn’t work on MacOS Sierra

So, was pretty sure that “ForwardAgent yes” was added all over my SSH config. And that was working fine, on Linux. When I moved all my stuff to my newly bought Mac, well, discovered that it’s not working anymore. So, actually, 2 lines need to be added to each host configuration:

Host example.com
 User xxx
 IdentityFile ~/.ssh/id_rsa_xxx
 ForwardAgent yes
 AddKeysToAgent yes
Advertisements

Skype not working on 64bit OpenSUSE

Symptoms are:

  • no sound at all, neither at startup, nor in calls, test calls, anything
  • all devices in Options > Sound Devices show “virtual device”

This is easy, you need to install a package and restart Skype:

zypper install libpulse0-32bit

I’m currently using OpenSUSE Tumbleweed, but I’ve seen it’s happening to a lot of users, including other distros as well.

top largest / smallest files in a folder

– Largest:

find . -type f -printf '%s %p\n'| sort -nr | head -5

– Smallest:

find . -type f -printf '%s %p\n'| sort -n | head -5

Lustre 2.6 on Debian Wheezy (clients)

I had a shitty job of trying to upgrade Lustre clients to the latest 2.6. I’ve had a lot of bugs on 2.5 some of them were solved in 2.6. My configuration is the following:

– 3 CentOS6 servers (for now)

– 24 Debian Wheezy clients (for now)

Upgrading Lustre on CentOS went pretty smooth. Just found the updated packages and installed. On the other had, Debian clients, let’s say I was not so lucky. So here is what I did:

– installed a Debian 7 x86_64 on a VM (kernel version 3.2.0-4-amd64)

– found a RedHat kernel ported to Debian here (thanks Thomas Stibor) and installed linux-headers and linux-images deb packages, then booted using those:

http://web-docs.gsi.de/~tstibor/lustre/lustre-builds/wheezy/redhat-kernel/2.6.32-rh-431.23.3-lustre-tstibor-latest/

– cloned branch 2.6 from Lustre, then:

git clone -b b2_6 git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release/
sh autogen.sh
./configure --enable-dist
make dist

This will create a file called lustre-2.6.0.tar.gz in lustre-release folder. Then:

mkdir BUILD
cd BUILD/
ln -s ../lustre-2.6.0.tar.gz lustre-2.6.0.orig.tar.gz
tar xzvf ../lustre-2.6.0.tar.gz
./configure

Then, the tricky part. Somehow, from the rules and makefiles in there, I found out that if you’re not changing latest version of Lustre in debian/changelog, it will build version 2.6 with a name, something like: 1.8.1.50. Also, somewhere along the way, it also expects a “-” after the version. Not good. For this to change, we’ll add this to the beginning of debian/changelog (only the first line counts):

lustre (2.6.0-1) unstable; urgency=low
  * Update for an on-the-road-to-1.8.2 release
 -- Brian J. Murrell <brian@interlinx.bc.ca>  Mon, 17 Aug 2009 14:54:35 -0400

After that, instead of building packages with version 1.8.1.50-1 it will build packages with the same version as the sources are. I think it’s safe to maybe only change first line’s version and that’s it. Then:

make debs

This will create all necessary deb files and will put them in lustre-release/lustre-2.6.0/debs/ (I’ve also included the kernel debs and e2fsprogs, I needed an archive that can be installed on production servers):

e2fsprogs-1.42.9.wc1.deb
linux-headers-2.6.32-rh-431.23.3-lustre-tstibor_20140904_amd64.deb
linux-image-2.6.32-rh-431.23.3-lustre-tstibor_20140904_amd64.deb
linux-patch-lustre_2.6.0-1_all.deb
lustre_2.6.0-1_amd64.changes
lustre_2.6.0-1.dsc
lustre-client-modules-2.6.32-rh-431.23.3-lustre-tstibor_2.6.0-1_amd64.deb
lustre-dev_2.6.0-1_amd64.deb
lustre-source_2.6.0-1_all.deb
lustre-tests_2.6.0-1_amd64.deb
lustre-utils_2.6.0-1_amd64.deb

That’s it. On your client machine, you should install:

e2fsprogs-1.42.9.wc1.deb
linux-headers-2.6.32-rh-431.23.3-lustre-tstibor_20140904_amd64.deb
linux-image-2.6.32-rh-431.23.3-lustre-tstibor_20140904_amd64.deb
linux-patch-lustre_2.6.0-1_all.deb
lustre-client-modules-2.6.32-rh-431.23.3-lustre-tstibor_2.6.0-1_amd64.deb
lustre-utils_2.6.0-1_amd64.deb

Should be enough. But first, better test on a VM/test machine. To verify that the wanted version of lustre in installed:

cat /proc/fs/lustre/version 
lustre: 2.6.0
kernel: patchless_client
build:  v2_6_0_0--PRISTINE-2.6.32-rh-431.23.3-lustre-tstibor

Good luck!

Atom git commands not working on OS X Yosemite 10.10 beta 2

What I did was:

1. backup everything git* in /usr/bin

# cd /usr/bin
# mkdir temp
# mv git* temp/

2. copy everything git* from /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/ to /usr/bin/

# find /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/ -name git\* -maxdepth 1 -exec cp {} /usr/bin/ \;

So now I have:

# ls -Al git* | wc -l
 167

And Atom seems to be working again.

PS I wanted to answer someone on stackoverflow but got lazy into formatting this on their website…

clustered LVM to non-clustered LVM

I had to re-configure clustered LVM to non-clustered LVM for some Lustre setup I’m working on. Thing is, clustered LVM can only be accessible if clvmd service is running. But I couldn’t start it, because I needed all cluster services down. After searching, I found that:

– to check if the LVM is clustered:

vgdisplay vgsrv1 --config 'global {locking_type = 0}' | grep Clustered
 WARNING: Locking disabled. Be careful! This could corrupt your metadata.
 Clustered yes

– to change it back to non-clustered LVM:

vgchange -cn vgsrv1 --config 'global {locking_type = 0}'

Now, to refresh your VG list type:

vgchange -ay

logrotate doesn’t rotate from file exists error

So, I had the following config:

/var/log/remote/*/php/*.log /var/log/remote/*/apache2/*.log { 
 daily
 missingok
 rotate 7
 compress
 delaycompress
 create 640 rsyslog rsyslog
 sharedscripts
 postrotate
 /etc/init.d/rsyslog restart > /dev/null
 endscript
}

Paths were valid, ownership of folders was root.root, still, I had sample.log which wouldn’t get renamed to sample.log.1 and new file wasn’t created either. Have been sitting on this for days now. Decided to redo the config (was actually a bigger one, removed every comment, removed everything it wasn’t needed for my test and ran logrotate with the new config, forced. And eventually after carefully reading the logs (used logrotate -vf /path/to/conf) I found out that:

error: error creating output file /var/log/remote/something/apache2/error.log.1.gz: File exists

Ok, the file existed. So what? It had the necessary permissions to delete/replace it. But, of course, it didn’t. Moreover, after getting this error, creating other sample.log.1 files didn’t work either so rotation failed. So I deleted that empty file (yes, it was empty) and retried the logrotate on my test config. Of course, it worked. Surely I couldn’t find any documentation on this, so next time it happens, try to run verbosely and check for “file exists” errors. That will explain strange logrotate behavior.

 

deb package status

As I’m always forgetting the meaning on the letters from “dpkg –list”, here’s the meaning:

First character:

u: Unknown (an unknown state)
i: Install (marked for installation)
r: Remove (marked for removal)
p: Purge (marked for purging)
h: Hold
Second Character:

n: Not- The package is not installed
i: Inst – The package is successfully installed
c: Cfg-files – Configuration files are present
u: Unpacked- The package is stilled unpacked
f: Failed-cfg- Failed to remove configuration files
h: Half-inst- The package is only partially installed
W: trig-aWait
t: Trig-pend

Third Character:

R: Reinst-required The package must be installed.

This was taken from here.

changing PATH for Ansible and Ansible loops

So, I’ve had a problem with trying to run things as ansible and sudo. Of course, password was not required for any of my sudo commands, but I often get “command not found”. Clearly, after running “env” with ansible using sudo on a server, I noticed I was missing /usr/sbin and /sbin from my path, as opposed to root’s PATH with had both folders.

In my case, was a Debian init stript which relied on start-stop-daemon, which is in sbin, and I got the command not found error. Here is how you set it up in a playbook:

 - name: stop monit service
 shell: PATH=$PATH:/usr/sbin:/sbin sudo /etc/init.d/monit stop

Testing showed that:

ansible test-srv -s -m shell -a "PATH=$PATH:/usr/sbin:/sbin env"
test-srv | success | rc=0 >>
PATH=/home/ansible/ansible/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/sbin:/sbin

Today I also discovered loops. I had to run a bunch of commands on a lot of servers, searched a bit and found loops. So, if you need to run 5 commands on your servers on a single task from a playbook, do the following:

- name: multiple commands
  command: "{{item}}"
  with_items:
    - /path/to/command1
    - /path/to/command2
    - command3
    - command4
    - command5

Path requirement is for the case in which the specified command isn’t in user’s PATH by default and you should specify full path for the command itself.

incapable AT&T sysadmins

I’ve been trying for weeks to remove my provider’s external mail server IP address from AT&T’s blacklists as it’s affecting our business. Being a booking website, it’s hard not to be able to send booking confirmations and other stuff required for people to be able to complete a booking on our website. So we always get the NDR telling us we’re being rejected. I’ve tried looking for our IPs in they’re third party RBLs like SpamHaus or Symantec, with no luck. So our IP is in their own blacklist, not third-party.

Today, after about 3 weeks from when I first filled their form here, http://rbl.att.net/cgi-bin/rbl/block_admin.cgi I tried again. Both times, didn’t receive any confirmations by email, nothing, nada. This time I discovered they have some email address for requests based on what you filled in the form and I tried that. Hope this time their sysadmins will take it into consideration. Hopefully this time they’re not sleeping, not smoking weed or whatever the fuck they’re doing as they don’t seem to care about their customers and this time I have DKIM installed on both domains we’re using to send out emails. Hope it will count in their decision.

BTW, how can you blacklist an IP for a mail server for a booking company? As long as it’s not some open relay for everybody to send mail from, did they even checked the email? See what’s it about? ..or it’s just that when a client (or 2, 3) marks the mail as SPAM, AT&T adds the IP to their blacklist and never gets it out of it? Should we get a full C-class subnet to send emails from? B? Maybe customers should just switch to other services? Like Yahoo, with infinite storage, GMail, Outlook.com, install the webmail adblocker extension to remove unuseful crap (Ads) and they should have a clean interface. Also, tickets opened to them seem to have way better results at removing from blacklists as the results I’m having with AT&T.

Although I’ve been appointed by HR agencies to get hired at AT&T, I’m very glad I’ve turned down their offers. Hell no! (although, sleeping in the office and smoking weed all day seems like a nice treatment)