A returning to this blog

Just a small script to myself to remember. An elegant and playful way to internally daily backup a jira+confluence+gitlab machine - and avoid all the "unlikely happen" risks.

#!/bin/bash
BACKUPLOG=/var/log/backuplog
exec >  >(tee -ia $BACKUPLOG)
exec 2> >(tee -ia $BACKUPLOG >&2)
if [ ! -f /backup/MOUNTED ]; then  # temp solution for further use
    echo FATAL_BACKUP_NOT_MOUNTED >> $BACKUPLOG
    exit 1
fi

date
echo BACKUP_STARTED

# CONFLUENCE
MYPATH=/var/lib/confluence/backups
FILE=backup-`date +%F|sed 's/-/_/g'`
cp $MYPATH/$FILE.zip /backup/confluence
[[ `ls $MYPATH|wc -l` -gt 15 ]] && find $MYPATH -mtime +15 -delete # purge old backups only if there are new ones !
[[ `ls /backup/confluence|wc -l` -gt 60 ]] && find /backup/confluence/ -type f -mtime +60 -delete

#JIRA
MYPATH=/var/lib/jira/export/
# another nice way
rsync -avh $MYPATH /backup/jira/ # no autodelete!
[ $? -ne 0 ] && echo RSYNC_ERROR_IN_BACKUP # temp set for further use
[[ `ls $MYPATH|wc -l` -gt 41 ]] && find $MYPATH -type f -mtime +20 -delete # 2 backups daily! purge old backups only if there are new ones !
[[ `ls /backup/jira|wc -l` -gt 120 ]] && find /backup/jira -mtime +60 -delete
tar -czf /backup/jira/$FILE-data.tgz /var/lib/jira/data

# MYSQL SIMPLE MIRROR BACKUP
rsync -avh --delete /var/lib/automysqlbackup/ /backup/mysql/
sleep 3

# GITLAB
/opt/gitlab/bin/gitlab-rake gitlab:backup:create
sleep 3
mv /var/opt/gitlab/backups/* /backup/gitlab/

# etc
rdiff-backup /etc /backup/etc
rdiff-backup --remove-older-than 4W /backup/etc
echo BACKUP_ENDED
date

File access auditing on a Windows fileserver: Data Leakage Prevention

Here is a clever script concept that helps company managers notifying someone's unusual amount of file reading. That's typical behaviour for an employee who is intended to quit and try to steal all the files of that company. Such auditing softwares are on the market for several hundred or thousand bucks!
Luckily for you, I've written one in bash. OK that's not good news for ones who use only Windows. But it can be easily portable to any script language, for example, php so that it could be run directly in the Windows fileserver or DC by installing the proper runtime enviroment. (PHP, ruby, python, etc.)
Exploring that thought further, now I'm going to translate that for myself. ;) But for now, it's enough to get it work in bash.

The original idea is that we suppose that all the users open almost the same amount of files daily on their daily routines. This script always alerts when a statistical threshold percent reached per user.
In the following example you are going to see a nice solution for lab use in which I transfer the logfile from the Windows server to a Linux server to be able to run the bash script on it. You can find detailed comments inside the script.

Step-by-step installation:
1: Enable audit log policy on your Windows Server, assign it to the target folders and test it
(Note: in the above blog you can find an advanced example. In my case I look for event id 4663 because it just contains the information I need.) Set the audit rules according to your needs. The less eventrule the better. We need to trace file reads so the first rule is a must.

2: You need to export the specific events from the security log to a plain file. So create a getsec.ps1 file in c:\script\ with the following content:
Get-EventLog security -After (Get-Date).AddDays(-1) -InstanceId 4663 |select Message|ft -AutoSize -Wrap > c:\auditing\report.txt
3: Also, don't forget to create that c:\auditing folder and then put an empty file into it named: mounted

 4: Schedule the script to run at the end of the working hours or at midnight. The command is to be: (e.g.) C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe and the argument (e.g.): -executionpolicy bypass -file c:\scripts\getsec.ps1  2>&1 > C:\scripts\log.txt
5: Share c:\auditing folder with a dedicated user that is intended to be used only by the Linux server, e.g.: linuxsrv
6: On your linux box, install the following packages: cifs-utils dos2unix mutt iconv
7: Test your connection:
 [ -f /mnt/mounted ] || mount.cifs //192.168.xx.xx/auditing/ /mnt/ -o username=linuxsrv,password=Sup3rS3cur3P4$$,domain=contoso
8: Create the base directories in, e.g.
mkdir /root/auditor && cd /root/auditor
mkdir archive average stat users; echo "0" > counter

Having succeeded, congratulations, now you are ready to track your file access activity and watch out for possible data stealing FOR FREE!


Here is the mighty script. See comments inline!

Three small scrips

Hey there, long time no see. Nothing special news here, just wanted to see googlebots heading here. So here comes my first new year post, 3 minor scripts. First one is for Exchange: adds new email aliases for users from a .csv and makes it default address.

Import-Csv c:\scripts\data.csv -Header UZER,ADDRZ| Foreach{
Set-Mailbox $_.UZER -EmailAddressPolicyEnabled $false
   $user = Get-Mailbox -Identity $_.UZER
   $user.EmailAddresses+=$_.ADDRZ
   Set-Mailbox $user -PrimarySmtpAddress $_.ADDRZ
}


example of data.csv: (without any headers!)
mgibson,mel.gibson@mighty.com
ctom,tom.cruise@mighty.com
cjoe,joe.cool@mighty.com


Second one is a bit tricky. I wanted to list all my distribution groups and their members. There are lots of solutions for this, e.g. you can find an edifying blog entry here. Unfortunatelly most of these scripts don't work nowadays at Office365 Exchange because of this unpleasing nastiness:
Cannot process argument transformation on parameter 'Identity'. Cannot convert value to type "Microsoft.Exchange.Configuration.Tasks.DistributionGroupMemberIdParameter". Error: "Cannot convert hashtable to an object of the following type: Microsoft.Exchange.Configuration.Tasks.DistributionGroupMemberIdParameter. Hashtable-to-Object conversion is not supported in restricted language mode or a Data section."                                                       

Explained on reddit by PsTakuu: It's not the object being passed into the Get-DistributionGroupMember by the pipeline that is causing the issue, it's that you are shoving an entire object into the first positional parameter (Identity) and it doesn't accept hash tables.
Here's a way to recreate your issue:

Get-DistributionGroup | select -First 1 | %{Get-DistributionGroupMember $_}

Here's the way to fix:

Get-DistributionGroup | select -First 1 | %{Get-DistributionGroupMember $_.identity} 

So here is the final working solution:

foreach ($group in Get-DistributionGroup) { get-distributiongroupmember $group.displayname | ft @{expression={$_.displayname};Label="$group"}}

The results can be redirected to file like this: $( foreach (............) )|out-file file.txt
or

$result = foreach (...)

$result | out-file file.txt -append

A +1 powerlist, for bonus: 

Get-DistributionGroup|format-table -wrap -property name,emailaddresses,hiddenfromaddresslistsenabled,RequireSenderAuthenticationEnabled > c:\groups.txt

 The third supersimple linuxer script adds users to a linux system and into samba fileserver database. I don't care about real names, room numbers and so on. That also creates tricky .bat files to make it easier to attach the network drive to windows users later.

#!/bin/bash
while read line; do
uzer=$(echo $line|cut -d ':' -f1)
pazz=$(echo $line|cut -d ':' -f2)
useradd -p $(openssl passwd -1 $pazz) $uzer --shell /bin/false --no-create-home --no-user-group
echo -ne "$pazz\n$pazz\n" | smbpasswd -a -s $uzer
echo "cmdkey /add:192.168.85.254 /user:workgroup\\$uzer /pass:$pazz" > /root/batz/$uzer.bat
echo "net use m: \\\192.168.85.254\\workz /P:Yes" >> /root/batz/$uzer.bat
done < users.txt

example of users.txt:
melbigson:jydac3sS
tomcruise:hEieafS
joecool:nhi252ax

Playing around with pattern subtitution

The other day I was given a cool task that I should replace the every second occurance of a character in a line. If there are only one of that special char (e.g. a colon) then do nothing. The list itself had tousands of newlines. Digging deep into this task I've collected some nice tricks around the net I wanted to record here.
#!/bin/bash

xxx="This:is:a:test"
echo "0:" `grep -o ":" <<< "$xxx" | wc -l ` # simple count
y="${xxx//[^:]}"        #pattern matching, y= all the chars that matches the char itself
echo "1: " "$y" # prints :::
echo "2: " ${#y} # stands for the lenght of a string = 3
echo "3: " `echo $xxx | awk -F":" '{print $NF}'` # finds the last occurence and cut the original string after there = test
echo "4: " `echo $xxx | awk -F":" '{print length($0)-length($NF)}' ` # similar to above but prints the found char position in the string = 14
end=${xxx##*:}
echo "5: Last : is in column $((${#xxx} - ${#end}))" # same as above
echo "6: " `sed 's/\(.*\):.*/\1/' <<< $xxx` # cuts the string at the last occurence of : and prints the first part
echo "7: " `sed 's/.*\:/\ /g' <<< $xxx` # cuts the string at the last occurence of : and prints the rest = test
echo "8: " `sed 's/\(.*\):/\1!/' <<< $xxx` # replaces the _last_ occurence of : with a !
echo "9: " $xxx| sed 's/t$/!/' # same as above what have to specify the last char
echo "10: " $xxx| sed 's/:/!/2' # replaces the second occurence of : with !
echo "11: " ${xxx##*:} # cuts the string at the last : and prints the rest = test
echo "12: " "${xxx#*:}" # cuts out the first word, prints the rest = "is:a:test"
echo "13: " ${xxx%:*}!!!${xxx##*:} # replaces the last occurence of : with the string: !!!
echo "14: " "${xxx%?}!"  # replaces the very last character of the string with !
echo "15: " ${xxx%:*} # cuts out the last part of the string using separator : ,selecting the first parts.
echo "16: " $xxx | sed "s/:[^:]*$//"  # cuts out the last part of the string using separator : ,selecting the first parts.
echo "17: " `sed -r "s/([^:]*:){2}//" <<< $xxx` # removes the first two parts separeted by : and prints the rest= "a:test"
echo "18: " "${xxx/:/!}" # replace the first occurence without using sed
echo "19: " "${xxx//:/!}" # replace all occurences of : without using sed
echo "20: " ${xxx:5:2} # for the sake of completion, prints = is. (2 chars from the 6th char)
echo "21: " ${xxx,} # converts the first char to lowercase
echo "22: " ${xxx,,} # concerts all to lowercase
echo "23: " "${0##*/}" # prints the name of the script without using basename
#echo $xxx | awk -F: '{print $1 $2 FS $3 $4}'

Little dear ones of mine

How to take actions on a directory that contains hundreds of subdirectories, named like this...

0001
0002
....
0100
0101
...
0999
1000
...

...but just on the first some hundreds of them so their proper naming could be an issue. Solution:

#!/bin/bash

for i in $(seq 500);do
lngt=`expr length $i`
case $lngt in
    1)
    i=000$i
    ;;
    2)
    i=00$i
    ;;
    3)
    i=0$i
    ;;
esac
ls /home/samba/archive/$i -LRs >> /root/content.txt
done

It could be more solid but you know, always Keep It Simple&Stupid. :-)
Here is a more complex one. It's a cron driven script that checks your openvpn logfile and email you if an event (e.g. if certain user connects) found. It remembers its last run so that a logline never get processed twice. Also handles logrotate events.

#!/bin/bash

cd /var/log/openvpn
[ -e temp ] && rm temp
echo "" > connectionz
NOW=`cat openvpn.log|wc -l`
LAST=`cat last`
CHECK=$(($NOW-$LAST))
    if [[ $CHECK -ge 0 ]]; then # change found
    echo $NOW > last # if 0 then doesn't matter but no harmful
    else
    echo 0 > last # logrotation happened, nulling last
    LAST=0
    fi
tail -$CHECK openvpn.log|grep Initiated > connectionz
while read line
 do
 USER=`echo $line|cut -d '[' -f2|cut -d ']' -f1`
 DATUM=`echo $line|cut -d ' ' -f2-5`
  if [ $USER = "JohnSmith" ] || [ $USER = "PeteSmith" ] || [ $USER = "JaneSmith" ] ; then
   echo "A user connected:"$USER" event time:"$DATUM >> temp
   echo "" >> temp
  fi
 done < connetionz
[ -e temp ] && cat temp | mail -n -s "OPENVPN CONNETION initiated" myemail@mydomain.com,yourdomain@yourdomain.com

Usefulz

Quicksort by size
du -shc * | sort -h
change on file group owner if group=x
find . -type f -group X-exec chgrp Y {} \;


I can't remember how it's made but I must have been in a hurry. :)

#!/bin/bash
rm /root/mailbe &> /dev/null
[ -f /root/okk ] && exit 0
k=0
megye=0
for i in Bacs Baranya Bekes Heves;do
 mtomb[$k]=$i
 cd /atvitel/pdf/npd/backup/$i
 cou=0
 for j in *`date --date="tomorrow" +%d`*.pdf;do
  cou=`expr $cou + 1`
  done
   mtomb[$k]=`echo ${mtomb[$k]} $cou`
   k=`expr $k + 1`

done
  for k in `seq 0 8`;do
   pdfek=`echo ${mtomb[$k]}| cut -d ' ' -f 2`
   echo ${mtomb[$k]} >> /root/mailbe
   [ "$pdfek" -ge "16" ] && megye=`echo "$megye + 1 "|bc`

  done

if [ $megye = 9 ];then

  echo "Bacs Baranya Bekes Heves megvan ,OK"|mail -s "goto home" "viktornak@email.hu"

  touch /root/okk

else

  cat /root/mailbe|mail -s "Megyek kuldese allapot jelentes" "viktornak@email.hu"

fi