2015. szeptember 28., hétfő

Playing around with pattern subtitution

The other day I was given a cool task that I should replace the every second occurance of a character in a line. If there are only one of that special char (e.g. a colon) then do nothing. The list itself had tousands of newlines. Digging deep into this task I've collected some nice tricks around the net I wanted to record here.

echo "0:" `grep -o ":" <<< "$xxx" | wc -l ` # simple count
y="${xxx//[^:]}"        #pattern matching, y= all the chars that matches the char itself
echo "1: " "$y" # prints :::
echo "2: " ${#y} # stands for the lenght of a string = 3
echo "3: " `echo $xxx | awk -F":" '{print $NF}'` # finds the last occurence and cut the original string after there = test
echo "4: " `echo $xxx | awk -F":" '{print length($0)-length($NF)}' ` # similar to above but prints the found char position in the string = 14
echo "5: Last : is in column $((${#xxx} - ${#end}))" # same as above
echo "6: " `sed 's/\(.*\):.*/\1/' <<< $xxx` # cuts the string at the last occurence of : and prints the first part
echo "7: " `sed 's/.*\:/\ /g' <<< $xxx` # cuts the string at the last occurence of : and prints the rest = test
echo "8: " `sed 's/\(.*\):/\1!/' <<< $xxx` # replaces the _last_ occurence of : with a !
echo "9: " $xxx| sed 's/t$/!/' # same as above what have to specify the last char
echo "10: " $xxx| sed 's/:/!/2' # replaces the second occurence of : with !
echo "11: " ${xxx##*:} # cuts the string at the last : and prints the rest = test
echo "12: " "${xxx#*:}" # cuts out the first word, prints the rest = "is:a:test"
echo "13: " ${xxx%:*}!!!${xxx##*:} # replaces the last occurence of : with the string: !!!
echo "14: " "${xxx%?}!"  # replaces the very last character of the string with !
echo "15: " ${xxx%:*} # cuts out the last part of the string using separator : ,selecting the first parts.
echo "16: " $xxx | sed "s/:[^:]*$//"  # cuts out the last part of the string using separator : ,selecting the first parts.
echo "17: " `sed -r "s/([^:]*:){2}//" <<< $xxx` # removes the first two parts separeted by : and prints the rest= "a:test"
echo "18: " "${xxx/:/!}" # replace the first occurence without using sed
echo "19: " "${xxx//:/!}" # replace all occurences of : without using sed
echo "20: " ${xxx:5:2} # for the sake of completion, prints = is. (2 chars from the 6th char)
echo "21: " ${xxx,} # converts the first char to lowercase
echo "22: " ${xxx,,} # concerts all to lowercase
echo "23: " "${0##*/}" # prints the name of the script without using basename
#echo $xxx | awk -F: '{print $1 $2 FS $3 $4}'

2015. szeptember 24., csütörtök

How to perform an automated brick-level (mailbox level) Exchange 2003 backup

Ohh those were the easy, happy and uncomplicated times when people used Windows 2003 SBS and Exchange 2003 servers. Even if it's EOL now there are still many companies out there where managers don't give a heck to security considerations and warnings.
Restoring a relatively large Exchange database from ntbackup is one of those things that none of the sysadmins are raving about. I mean, restoring the whole database just because a skilled user accidentally deleted an "extreme-important-and-high-business-valuable" email.
It's a known sad fact that Exchange 2003 lacks the feature of keeping soft-deleted items in the database for the retention period. So in the above example you don't have any other choice than restoring everything into a second recovery database. That would be funnier if your server partitions are going full and you have no free space to fill with a second multi-gigs database.
One solution would be to use Exmerge but scripting it is maybe the largest pain in the ass I've ever seen and it still can't export mailboxes larger than 2Gigs. Forget it.
But here is my genious method to backup your users emailing daily. All you need is a Windows backup PC on the network with two hard drives: a smaller for your system partition and a larger one to store the backups. And, an Outlook 2010 installed in that system. (Ehm, just a sidenote: you don't need to activate that Outlook anyway.)

First, you need an account which has all the necessary rights to export databases. Create a user named, for example, exmerge with a super-secure password. Just to be an the safe side and be careless enough, add it to your Administrators group.
Open your System Manager and give all rights to exmerge on your Mailbox Store.

That was everything on your server. Go to your backup PC. Open your Outlook 2010 and set up the account of your exmerge user. Older versions of Outlook are no good because they don't cache shared mailboxes for offline use.
Having done, go and get a coffee.Then:

  •     In Outlook click File tab in the Toolbar
  •     Click Account Settings button, select Account Settings
  •     Select the E-Mail tab
  •     Highlight your mailbox, click the Change button
  •     Click the More Settings button
  •     Select the Advance tab
  •     Click the Add button
  •     Type the first characters of your first user's name and let Outlook resolve it with Add button.
  •     Repeat previous step again and again for all the users in your organization
  •     Click the Apply and Ok buttons
  •     Click Next, Finish, and Close buttons
Now let this PC alone and don't touch it during the next 24 hours. Hopefully one day will be enough to download all the emails your users have. It's a good idea to encrypt both hard disks in this machine because, as you may guessed already, all those highly confidental emails will get in those Outlook and your local system hard disk. The exact location you can find that cache file having .ost extension at is something like:
C:\%your user profile%\Local Settings\Application Data\Microsoft\Outlook\Outlook.ost
It will grow pretty large, similar to the size of your exchange priv1.edb file.

Okay, one day later you will have all emails cached and the Outlook GUI responsible again. Now you need a simple scheduled .bat to start Outlook. Outlook needs a few quiescent hour to syncronize all mailboxes. Let it do its jobs.
Some hours later stop it gracefully via, e.g. a runme.bat file including:
@echo off
cscript "c:\scripts\CloseOutlook.vbs"

and that CloseOutlook.vbs contains:
Dim oOL
Set oOL = CreateObject("Outlook.Application")

Then grab your whole folder on your C: (if you want to be sure) and copy it with a cleverly parametered xcopy or with any free backup software (e.g. Cobian Backup) onto your second drive. Don't run out of space! Make sure you keep just the sufficent number of versions of the .ost file.
How to restore? It's easy! DO NOT START your Outlook! Instead, open your Control Panel and find Mail. Open it and select Email accounts.

  • Select the Exchange account, and then click Change.
  • Click More Settings. 
  • Choose whether to work offline or online each time you start Outlook     Click Manually control connection state, and then select the Choose the connection type when starting check box.
  • Exit
  • Start your Outlook and select Offline mode.
  • Find the missing emails within the mailbox in question.
  • I am a hell damn genious!

2015. szeptember 18., péntek

How to re-check a resized virtual disk in linux

To recognize a newly added disk:

root@host:#echo "- - -" > /sys/class/scsi_host/host*/scan

To recognize the modified size of old disk:

root@host:# fdisk -l

Disk /dev/sdb: 11.7 GB, 10737418240 bytes

Disk /dev/sdb: 214.7 GB, 214748364800 bytesroot@host:# ls /sys/class/scsi_disk/
0:0:0:0  0:0:1:0
root@host:# echo '1' > /sys/class/scsi_disk/0\:0\:1\:0/device/rescan
root@host:# fdisk -l


Disk /dev/sdb: 236.5 GB, 214748364800 bytes

2015. szeptember 15., kedd

Exchange Survival Kit 3. - hardening and searching

If your servers have any sensitive data about their services (e.g. version numbers) to hide from from the wide world then you definitely want to change some default settings. First, it's adviseble to change your default Exchange SMTP banners and HELO string to hide your long and ugly default intro string.

For the Send Connector(s):

Open your EAC - Mail Flow - Send Connectors - Select your SEND connector and click on Scoping. On the bottom, find FQDN field and fill it implicitly.

For the Receive Connector(s):

You won't be able to change your internal hostname to your FQDN because your will get an obfuscating error. The phenomenon and the solution detailed in this blog. It's a nice trick but personally I don't care about keeping the timestamp and so on. What's more, I don't think anyone care about them.
So simply open your Exchange Powershell and:
Get-ReceiveConnector|select identity,bindings
Find your connector which bound to port 25 and:
Set-ReceiveConnector <ConnectorIdentity> -Banner "220 go ahead and make my day."

Hide your client's IP 

"In practice that means if you sent an email from Outlook, Outlook Web App (OWA) or an ActiveSync-connected smartphone while on the Corporate Wi-Fi, your device’s Corporate Wi-Fi IP address will be contained in the email. If you were connected to your home Internet at the time, your (public) home Internet IP address will be in the email.
This may give a recipient, or any party snooping up the email while in transit, decent clues of the network you were connected to and the whereabouts of your staff and you. " (all credits go to Will Neumann including the pics)

Searching logs for emails

An example worth thousand words! Note the tricky subject selector expression: selects both the "robbery" subjects AND the empty subjects. (because of the -or operator)

Get-MessageTrackingLog -Server [YOUR.CAS.SERVERNAME] -ResultSize Unlimited -Recipients [your.user@domain.com] -Start "9/12/2015 08:59:59" -End (Get-Date).AddHours(-72) | where{$_.sender -like "*@sender.com"}|where{$_.eventid -like "*eceiv*"}|Where-Object {$_.MessageSubject -match "robbery" -or $_.MessageSubject -notlike ""} select eventid,sender,recipients,messagesubject,timestamp -autosize | ConvertTo-Html > "C:\reports\track.html"

It hits and displays the first AND/OR (disjunction again, my favourite operation!) second matched recipients in a GUI:

Get-MessageTrackingLog -recipients john.snow@got.com,aragorn@mordor.org | select-object eventid,timestamp,messageid,sender,recipient,messagesubject | out-gridview

2015. augusztus 28., péntek

Exchange 2013 Survival Kit 2. - restoring

Just found a great MS doc that efficiently explains the basics of how Exchange 2013 handles Recoverable Items Folder. In short: if one user asks you to restore some accidentely deleted and purged email, you no more need to restore the whole database from Windows Backup and mount it to be able to restore the whole mailbox into a former state. At least, in theory.
If you are lucky enough your user remembers the properties of the emails he purged:
- the senders names, or
- the subject strings, or
- the date interval in which the email(s) was.
Unfortunately, Exchange 2013 can't restore a subfolder in your mailbox. Find out why here.
"This seems like it would be a simple enhancement into the cmdlet since the attribute exists on the mail item object.  It would be my vote to make this enhancement since it make single-item restores almost worthless if a folder is accidentally deleted. [...] Thanks for making my life more difficult than it needs to be Microsoft."
(/me also grateful.)

Clearing a Recoverable Items Folder while Single Item recovery is enabled is a bit problematic. See Use the Shell to clean up the Recoverable Items folder for mailboxes that are placed on hold or have single item recovery enabled

Easiest way to export only the Recoverable Items Folder from the mailbox to a .pst:
New-MailboxExportRequest -mailbox joecool -filepath \\localhost\backup\joe.pst -IncludeFolders "Recoverable Items"
An other interesting method explained here using In Place eDiscovery but there are some limitations. According to MS: "You can use In-Place eDiscovery in the Exchange admin center (EAC) to search for missing items. However, when using the EAC, you can’t restrict the search to the Recoverable Items folder. Messages matching your search parameters will be returned even if they’re not deleted. After they’re recovered to the specified discovery mailbox, you may need to review the search results and remove unnecessary messages before recovering the remaining messages to the user’s mailbox or exporting them to a .pst file.
For details about how to use the EAC to perform an In-Place eDiscovery search, see Create an In-Place eDiscovery search. "
Frankly, I've never done a search like this in EAC. Instead, doing a similar thing in Powershell:
First, search your RIF and place the results to Discovery mailbox.
Search-Mailbox "Joe Cool" -SearchQuery "from:'Sam Knows' AND keyword1" -TargetMailbox "Discovery Search Mailbox" -TargetFolder "JoeRecovery" -LogLevel Full Second, search the Discovery again with the same phrase and put the results back into your user (or anyone's) mailbox. The results will show in a strange folder structure: in the upper level there is a short report about the search, a .csv attached with the matching files and somewhere deep in the folders you will find the actual mails.
Search-Mailbox "Discovery Search Mailbox" -SearchQuery "from:'Sam Knows' AND keyword1" -TargetMailbox "Joe Cool" -TargetFolder "Recovered Messages" -LogLevel Full -DeleteContent
(Note the DeleteContent switch: it's important to clear up the Discovery Search Mailbox after yourself.)
Putting the results directly into a .pst:
New-MailboxExportRequest -Mailbox "Discovery Search Mailbox" -SourceRootFolder "April Stewart Recovery" -ContentFilter {Subject -eq "April travel plans"} -FilePath \\MYSERVER\HelpDeskPst\AprilStewartRecovery.pst

You can use the EstimateOnly switch to return only get an estimate of the search results and not copy the results to a discovery mailbox. So, just simulating a search to see what would actually happen: (Examples from Microsoft):
New-MailboxSearch "FY13 Q2 Financial Results" -StartDate "04/01/2013" -EndDate "06/30/2013" -SourceMailboxes "DG-Finance" -SearchQuery '"Financial" AND "Fabrikam"' -EstimateOnly -IncludeKeywordStatistics Start-MailboxSearch "FY13 Q2 Financial Results"
Get-MailboxSearch "FY13 Q2 Financial Results" | FL Name,Status,LastRunBy,LastStartTime,LastEndTime,Sources,SearchQuery,ResultSizeEstimate,ResultNumberEstimate,Errors,KeywordHits

To check a user state:
Get-Mailbox "Joe Cool" | FL SingleItemRecoveryEnabled,RetainDeletedItemsFor
To enable a single user:
Set-Mailbox -Identity "Joe Cool" -SingleItemRecoveryEnabled $true
To enable everybody and raise the default retention time limit:
Get-Mailbox -ResultSize unlimited -Filter {(RecipientTypeDetails -eq 'UserMailbox')} | Set-Mailbox -SingleItemRecoveryEnabled $true -RetainDeletedItemsFor 30
Some more advanced search examples here.

How to destroy your mailboxes permanently

Just find your disconnected mailboxes:
Get-MailboxStatistics -Database "Database name" | where {$_.disconnectdate -ne $null} | select displayname,MailboxGUID

How to purge them:
Get-MailboxStatistics –Database <DB NAME> | where {$_.disconnectdate –ne $null} | select displayname,MailboxGUID Remove-StoreMailbox –Database <Database-Name> -Identity <MailboxGUID-from-the-previous-cmdlet> -MailboxState Disabled (The Remove-StoreMailbox only works against Disconnected and soft-deleted mailboxes!)

Remove all soft-deleted mailboxes:
Get-MailboxStatistics -Database MBD01 | where {$_.DisconnectReason -eq "SoftDeleted"} | foreach {Remove-StoreMailbox -Database $_.database -Identity $_.mailboxguid -MailboxState SoftDeleted}
Get-MailboxStatistics -Database MDB01 | where {$_.DisconnectReason -eq "disabled"} | foreach {Remove-StoreMailbox -Database $_.database -Identity $_.mailboxguid -MailboxState disabled -Confirm:$False}  
Hard delete a mailbox (no option to restore it from the actual database!)
Remove-Mailbox <Mailbox> -Permanent:$True

2015. augusztus 19., szerda

ntopng install on Debian Sqeeze

If you are careless enough to just follow a step-by-step tutorial like this being on a good old Squeeze you surely will end up with a failing and buggy ntopng. E.g. you won't be able to see your newly created users (users tab is totally empty: No Results Found)
Looks somewhat broken
or can not switch between your monitored interfaces. If you start ntopng from shell you may see something like this:
19/Aug/2015 13:28:28 [src/Redis.cpp:170] ERROR: ERR unknown command 'HSET' [HSET ntopng.host_labels ]
19/Aug/2015 13:28:28 [src/Redis.cpp:170] ERROR: ERR unknown command 'HSET' [HSET ntopng.host_labels ]
19/Aug/2015 13:28:30 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:30 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:30 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:30 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:36 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:36 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:36 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'

This whole thing is because your Redis installation is out of date. Another nice thing in Debian Squeeze is its repositories includes Version: 2:1.2.6-1 Redis. Simply fix that with:
echo "deb http://backports.debian.org/debian-backports squeeze-backports main" >> /etc/apt/sources.list
apt-get update
apt-get -t squeeze-backports install redis-server

Now it is:
redis-server                       2:2.4.15-1~bpo60+2    
How to reset your forgotten ntopng admin password.
You might don't want to bother with compiling ntopng-2.0 packages on a simple standard Squeeze. In that case here are the x64 and x86 versions. You're welcome.

2015. július 28., kedd

Ugly bug in Draytek Vigor firewall?

One day I came across a unique error. A client reported that they were unable to query any nameserver outside their network, except for the case they query standard A records. So, A records worked fine but, e.g. NS or MX records failed with timeout. Local DNS servers was properly set with valid forwarders.
So, we experienced:
Default Server:  dc01.hq.local           

> google.org
Server:  dc01.hq.local                   

Non-authoritative answer:                       
Name:    google.org                             

> set type=mx 
> google.org                                 
Server:  dc01.hq.local                   

DNS request timed out.                              
timeout was 2 seconds.                      
*** Request to dc01.hq.local timed-out   

> server                                   
Default Server:  google-public-dns-a.google.com           

> google.org                            
Server:  google-public-dns-a.google.com        

DNS request timed out.                              
timeout was 2 seconds.                      
*** Request to google-public-dns-a.google.com timed-out      

What a riddle! Guess that! :)
After three hours it turned out that in their Vigor 2925 firewall router there was a built-in rule called "xNETBios > DNS" in the section called "Data filter" (very informative names by Draytek guys, phuhh). That blocked such special DNS queries - even if it was DISABLED!
Default factory settings

Factory settings

In the end I had to disable the entire Data Filter section - in that way, external DNS queries got to work as expected. I'm still unable to find any explanation for this.

Model Name : Vigor2925n
Firmware Version : 3.7.6
Build Date/Time : Nov 17 2014 17:20:57