Ubiquiti Unify nuisances and the attack of the Martians

Some weeks ago I was given a nice task. A client of ours wants us to set up two new Unify APs in its network with two new wireless networks: one for guest and one for internal use. They had a Vigor 2925 to be used for firewall and DHCP role. For those who are not familiar with the wireless products named ubiquiti unify APs, here are a few links to inform:
How do I configure a "Guest Network" on UniFi AP?
Instructive reading. Based on the infos here I finally decided not to use the internal "firewall" in the APs and let the Vigor do the network separation.
UniFi - Does the controller need to be running at all times?
Official answer says "no, most of the times it is not necessary." Unfortunately this isn't entirely true with the latest firmwares. :/
No worries, since the client already had a Linux server, it looked so simple to install the controler software and setup the nodes. Sadly, everything went a different way.
For a mystical reason I couldn't make the controller software, running on the Linux, see its APs, even they were in the same subnet by their IPs and in the same broadcast domain, for the sake of Layer2 communications. I spent two days just on this riddle. Maybe it was a misconfiguration of the D-Link switches or maybe an insolvable incompatibility issue. I don't know why to this very day. :( I tried everything but the time run out so I had to find a quick solution.
So I decided to use a different network card and a second subnet on the Linux only to control the APs.
I ended up with this config:

My interfaces were:
em1       Link encap:Ethernet  HWaddr 00:25:90:xx:xx:xx
          inet addr:172.16.20.30  Bcast:172.16.20.255  Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fed3:930c/64 Scope:Link
..
em2       Link encap:Ethernet  HWaddr 00:25:90:xx:xx:xx
          inet addr:192.168.3.200  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fed3:930d/64 Scope:Link
..
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
..

APs looked working. I think Connected(limited) state is normal in this case. Note that I didn't use the built-in "guest network" feature because it's just ridiculous.

 

 So everything seemed properly set. But to my greatest astonishment I coudn't reach network share of my server lying on 172.16.20.30 from my internal wifi client 192.168.3.11. When I started to ping and tcpdump'ed on the server I saw that echo requests came in but replies never went back. I thought to myself: Seeing the fact that the kernel wanted to reply on the other interface (192.168.3.x) it's hardly surprising that it didn't work.
So I set IP policy routing: if the packet comes from 192.168.3.11 on em1, reply to it on the same interface -em1- instead of em2. You know, all the iptables mangle MARK and ip route add default via 172.16.20.1 dev em1 table ... stuff, etc. etc. etc.
It didn't work either. Suddenly a light dawned on me. I turned on kernel martian packet logging with echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
and VOILA I saw:

Mar  2 20:08:03 superserver kernel: [ 2755.407570] IPv4: martian source 192.168.3.11 from 192.168.3.200, on dev em1
Mar  2 20:08:03 superserver kernel: [ 2755.407590] ll header: 00000000: ff ff ff ff ff ff 00 25 90 d3 93 0d 08 06        .......%......
Mar  2 20:08:04 superserver kernel: [ 2756.424025] IPv4: martian source 192.168.3.11 from 192.168.3.200, on dev em1
Mar  2 20:08:04 superserver kernel: [ 2756.424048] ll header: 00000000: ff ff ff ff ff ff 00 25 90 d3 93 0d 08 06        .......%......
Mar  2 20:08:05 superserver kernel: [ 2757.421639] IPv4: martian source 192.168.3.11 from 192.168.3.200, on dev em1
Mar  2 20:08:05 superserver kernel: [ 2757.421661] ll header: 00000000: ff ff ff ff ff ff 00 25 90 d3 93 0d 08 06        .......%......
It's a bit confusing isn't it ?!?! 192.168.3.200 is my own server!
I tried to turn off the Martian protection with echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
but I learned: such a bad routing problem can not be solved with a simple fix like this.
I was thinking very hard for an hour and finally I faked the kernel with an another subnet set on my second interface:

auto em2
iface em2 inet static
    address 192.168.3.200
    netmask 255.255.255.192

Now it's working as expected:
root@:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.20.1     0.0.0.0         UG    0      0        0 em1
172.16.20.0     0.0.0.0         255.255.255.0   U     0      0        0 em1
192.168.3.192   0.0.0.0         255.255.255.192 U     0      0        0 em2

These were all done on a:
Linux 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux