2015. szeptember 28., hétfő

Playing around with pattern subtitution

The other day I was given a cool task that I should replace the every second occurance of a character in a line. If there are only one of that special char (e.g. a colon) then do nothing. The list itself had tousands of newlines. Digging deep into this task I've collected some nice tricks around the net I wanted to record here.
#!/bin/bash

xxx="This:is:a:test"
echo "0:" `grep -o ":" <<< "$xxx" | wc -l ` # simple count
y="${xxx//[^:]}"        #pattern matching, y= all the chars that matches the char itself
echo "1: " "$y" # prints :::
echo "2: " ${#y} # stands for the lenght of a string = 3
echo "3: " `echo $xxx | awk -F":" '{print $NF}'` # finds the last occurence and cut the original string after there = test
echo "4: " `echo $xxx | awk -F":" '{print length($0)-length($NF)}' ` # similar to above but prints the found char position in the string = 14
end=${xxx##*:}
echo "5: Last : is in column $((${#xxx} - ${#end}))" # same as above
echo "6: " `sed 's/\(.*\):.*/\1/' <<< $xxx` # cuts the string at the last occurence of : and prints the first part
echo "7: " `sed 's/.*\:/\ /g' <<< $xxx` # cuts the string at the last occurence of : and prints the rest = test
echo "8: " `sed 's/\(.*\):/\1!/' <<< $xxx` # replaces the _last_ occurence of : with a !
echo "9: " $xxx| sed 's/t$/!/' # same as above what have to specify the last char
echo "10: " $xxx| sed 's/:/!/2' # replaces the second occurence of : with !
echo "11: " ${xxx##*:} # cuts the string at the last : and prints the rest = test
echo "12: " "${xxx#*:}" # cuts out the first word, prints the rest = "is:a:test"
echo "13: " ${xxx%:*}!!!${xxx##*:} # replaces the last occurence of : with the string: !!!
echo "14: " "${xxx%?}!"  # replaces the very last character of the string with !
echo "15: " ${xxx%:*} # cuts out the last part of the string using separator : ,selecting the first parts.
echo "16: " $xxx | sed "s/:[^:]*$//"  # cuts out the last part of the string using separator : ,selecting the first parts.
echo "17: " `sed -r "s/([^:]*:){2}//" <<< $xxx` # removes the first two parts separeted by : and prints the rest= "a:test"
echo "18: " "${xxx/:/!}" # replace the first occurence without using sed
echo "19: " "${xxx//:/!}" # replace all occurences of : without using sed
echo "20: " ${xxx:5:2} # for the sake of completion, prints = is. (2 chars from the 6th char)
echo "21: " ${xxx,} # converts the first char to lowercase
echo "22: " ${xxx,,} # concerts all to lowercase
echo "23: " "${0##*/}" # prints the name of the script without using basename
#echo $xxx | awk -F: '{print $1 $2 FS $3 $4}'

2015. szeptember 24., csütörtök

How to perform an automated brick-level (mailbox level) Exchange 2003 backup

Ohh those were the easy, happy and uncomplicated times when people used Windows 2003 SBS and Exchange 2003 servers. Even if it's EOL now there are still many companies out there where managers don't give a heck to security considerations and warnings.
Restoring a relatively large Exchange database from ntbackup is one of those things that none of the sysadmins are raving about. I mean, restoring the whole database just because a skilled user accidentally deleted an "extreme-important-and-high-business-valuable" email.
It's a known sad fact that Exchange 2003 lacks the feature of keeping soft-deleted items in the database for the retention period. So in the above example you don't have any other choice than restoring everything into a second recovery database. That would be funnier if your server partitions are going full and you have no free space to fill with a second multi-gigs database.
One solution would be to use Exmerge but scripting it is maybe the largest pain in the ass I've ever seen and it still can't export mailboxes larger than 2Gigs. Forget it.
But here is my genious method to backup your users emailing daily. All you need is a Windows backup PC on the network with two hard drives: a smaller for your system partition and a larger one to store the backups. And, an Outlook 2010 installed in that system. (Ehm, just a sidenote: you don't need to activate that Outlook anyway.)

First, you need an account which has all the necessary rights to export databases. Create a user named, for example, exmerge with a super-secure password. Just to be an the safe side and be careless enough, add it to your Administrators group.
Open your System Manager and give all rights to exmerge on your Mailbox Store.




That was everything on your server. Go to your backup PC. Open your Outlook 2010 and set up the account of your exmerge user. Older versions of Outlook are no good because they don't cache shared mailboxes for offline use.
Having done, go and get a coffee.Then:

  •     In Outlook click File tab in the Toolbar
  •     Click Account Settings button, select Account Settings
  •     Select the E-Mail tab
  •     Highlight your mailbox, click the Change button
  •     Click the More Settings button
  •     Select the Advance tab
  •     Click the Add button
  •     Type the first characters of your first user's name and let Outlook resolve it with Add button.
  •     Repeat previous step again and again for all the users in your organization
  •     Click the Apply and Ok buttons
  •     Click Next, Finish, and Close buttons
Now let this PC alone and don't touch it during the next 24 hours. Hopefully one day will be enough to download all the emails your users have. It's a good idea to encrypt both hard disks in this machine because, as you may guessed already, all those highly confidental emails will get in those Outlook and your local system hard disk. The exact location you can find that cache file having .ost extension at is something like:
C:\%your user profile%\Local Settings\Application Data\Microsoft\Outlook\Outlook.ost
It will grow pretty large, similar to the size of your exchange priv1.edb file.

Okay, one day later you will have all emails cached and the Outlook GUI responsible again. Now you need a simple scheduled .bat to start Outlook. Outlook needs a few quiescent hour to syncronize all mailboxes. Let it do its jobs.
Some hours later stop it gracefully via, e.g. a runme.bat file including:
@echo off
cscript "c:\scripts\CloseOutlook.vbs"
:EXIT

and that CloseOutlook.vbs contains:
Dim oOL
Set oOL = CreateObject("Outlook.Application")
oOL.Quit

Then grab your whole folder on your C: (if you want to be sure) and copy it with a cleverly parametered xcopy or with any free backup software (e.g. Cobian Backup) onto your second drive. Don't run out of space! Make sure you keep just the sufficent number of versions of the .ost file.
How to restore? It's easy! DO NOT START your Outlook! Instead, open your Control Panel and find Mail. Open it and select Email accounts.

  • Select the Exchange account, and then click Change.
  • Click More Settings. 
  • Choose whether to work offline or online each time you start Outlook     Click Manually control connection state, and then select the Choose the connection type when starting check box.
  • Exit
  • Start your Outlook and select Offline mode.
  • Find the missing emails within the mailbox in question.
  • I am a hell damn genious!



2015. szeptember 18., péntek

How to re-check a resized virtual disk in linux

To recognize a newly added disk:

root@host:#echo "- - -" > /sys/class/scsi_host/host*/scan

To recognize the modified size of old disk:

root@host:# fdisk -l

[...]
Disk /dev/sdb: 11.7 GB, 10737418240 bytes
[...]

Disk /dev/sdb: 214.7 GB, 214748364800 bytesroot@host:# ls /sys/class/scsi_disk/
0:0:0:0  0:0:1:0
root@host:# echo '1' > /sys/class/scsi_disk/0\:0\:1\:0/device/rescan
root@host:# fdisk -l

[...]

Disk /dev/sdb: 236.5 GB, 214748364800 bytes

2015. szeptember 15., kedd

Exchange Survival Kit 3. - hardening and log searching

If your servers have any sensitive data about their services (e.g. version numbers) to hide from from the wide world then you definitely want to change some default settings. First, it's adviseble to change your default Exchange SMTP banners and HELO string to hide your long and ugly default intro string.

For the Send Connector(s):

Open your EAC - Mail Flow - Send Connectors - Select your SEND connector and click on Scoping. On the bottom, find FQDN field and fill it implicitly.


For the Receive Connector(s):

You won't be able to change your internal hostname to your FQDN because your will get an obfuscating error. The phenomenon and the solution detailed in this blog. It's a nice trick but personally I don't care about keeping the timestamp and so on. What's more, I don't think anyone care about them.
So simply open your Exchange Powershell and:
Get-ReceiveConnector|select identity,bindings
Find your connector which bound to port 25 and:
Set-ReceiveConnector <ConnectorIdentity> -Banner "220 go ahead and make my day."

Hide your client's IP 


"In practice that means if you sent an email from Outlook, Outlook Web App (OWA) or an ActiveSync-connected smartphone while on the Corporate Wi-Fi, your device’s Corporate Wi-Fi IP address will be contained in the email. If you were connected to your home Internet at the time, your (public) home Internet IP address will be in the email.
This may give a recipient, or any party snooping up the email while in transit, decent clues of the network you were connected to and the whereabouts of your staff and you. " (all credits go to Will Neumann including the pics)





Searching logs for emails

An example worth thousand words! Note the tricky subject selector expression: selects both the "robbery" subjects AND the empty subjects. (because of the -or operator)

Get-MessageTrackingLog -Server [YOUR.CAS.SERVERNAME] -ResultSize Unlimited -Recipients [your.user@domain.com] -Start "9/12/2015 08:59:59" -End (Get-Date).AddHours(-72) | where{$_.sender -like "*@sender.com"}|where{$_.eventid -like "*eceiv*"}|Where-Object {$_.MessageSubject -match "robbery" -or $_.MessageSubject -notlike ""} select eventid,sender,recipients,messagesubject,timestamp -autosize | ConvertTo-Html > "C:\reports\track.html"

It hits and displays the first AND/OR (disjunction again, my favourite operation!) second matched recipients in a GUI:

Get-MessageTrackingLog -recipients john.snow@got.com,aragorn@mordor.org | select-object eventid,timestamp,messageid,sender,recipient,messagesubject | out-gridview