tzlee.com/blog

  • Happy Holidays, Welcome 2010

    Welcome back…

    I know you have been faithful readers.

    See? You’re smiling. Stop denying it. You love my blog, don’t you?

    It has been a crazy month. I’m part of a small team of three and two of my colleagues went away – one to Turkey and another to reservist so he could help weed out terrorist. I was, of course, left all alone to take on some good amount of work that, well,  came all at the same time. Screw Murphy. Hate that guy.

    So it’s December and everybody’s in a holiday mood. I wish I was too, but it’s not quite easy with a fucking noisy neighbour living right upstairs. They make so much noise and vibration by dragging furniture, slamming their doors and let their kids run about into the wee hours that even my window grilles rattle at times. After approaching them three times and calling the cops once, I decided to approach them one last time which ended in a yelling session, with the typical remark from these inconsiderate bastards like “this is my house, I’ll do whatever I want” or “go buy a condo“. Things improved a bit after the yelling, but it still happens. When the night’s all quiet, these sudden bumps and squeaks really make you to jump.

    So it seems nothing really changed in 2009 other than my marital status and an empty bank from a property purchase and renovation. It’s time to work out something new in 2010. My public list of items goes like this:

    • Get back on track with some research/dev type projects. I’ll be meeting an NUS professor before 2009 ends, so I’ll post an update here.
    • Learn to cook now that I’ve got a kitchen to myself. I’ve also promised to post some of my successful attempts out of many other unsuccessful ones.
    • Get the wedding banquet done.
    • Build new sources of revenue. I’ll start small, but aim big.
    • Get back to flying R/C occasionally as a hobby. I’ve stopped flying for the entire of 2009 without even realizing it.

    And of course, before 2009 ends, here’s a short to do list.

    • Pack up the study room. It’s in an absolute mess now.
    • Get another two sets of Wii Remote + Nunchuck since the two I ordered from HK over a month ago didn’t arrive at all.
    • Get my old Cello bows rehaired. Already sent and will only be done in January 2010.

  • Hardening Linux and Apache Servers for DDoS

    In my earlier entry I discussed an interesting topic on firewalls and why we don’t need them. I put a small LAMP server to the test and got my results.

    Attack Information:

    • Type: TCP SYN flood
    • Max performance: 26Kpps (8Mbps)
    • Source IP Spoofing: Yes

    Victim A Specifications:

    • VMware Guest on a Single Core Opteron 1.8GHz Sun X2100
    • CentOS 4.x + Apache 2.x
    • 768MB RAM
    • Tuned (see below)

    Here’s what I’ve added to tune the Linux TCP stack in /etc/sysctl.conf:


    net.ipv4.tcp_abort_on_overflow = 1
    net.ipv4.tcp_fin_timeout = 15
    net.ipv4.tcp_low_latency = 1
    net.ipv4.tcp_syncookies = 1
    net.ipv4.tcp_max_syn_backlog = 2048
    net.ipv4.tcp_synack_retries = 3
    net.ipv4.tcp_sack = 0
    net.ipv4.ip_conntrack_max = 65535
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_wmem = 4096 65536 16777216
    net.ipv4.ip_local_port_range = 1024 65000
    net.ipv4.tcp_keepalive_intvl = 15
    net.ipv4.tcp_keepalive_probes = 4
    net.ipv4.tcp_keepalive_time = 1800

    Here’s what I’ve added to the top of my iptables configuration in /etc/sysconfig/iptables as well:


    -N SYN
    -A SYN -m limit --limit 20/s --limit-burst 50 -j RETURN
    -A SYN -j DROP
    -A INPUT -p tcp --syn -j SYN

    * Note: During my testing, I added a log entry before dropping the packet as this floods the logs and kills the CPU and I/O so I highly discourage doing so.

    I repeated the same test on another VM running in a much more powerful Dell 2850 and with no modifications to the kernel or iptables.

    Victim B Specifications:

    • VMware Guest on a 2 x Dual Core Xeon 3.2GHz Dell 2850
    • CentOS 5.x + Apache 2.x
    • 256MB RAM
    • No Tuning

    Results:

    • Victim A held up to 16Kpps SYN flood (approx 5Mbps) but slowed down a little
    • Victim A held up to respond at 26Kpps SYN flood (approx 8Mbps) but was extremely slow
    • Victim B held up to 26Kpps SYN flood (approx 8Mbps) and did not slow down at all

    At this point in time, I couldn’t generate any more SYN packets as I lacked the hardware to do so, but it has given some conclusive results that a reasonably powerful LAMP hardware could take on modest DDoS attacks if configured correctly. I would expect a bare metal hardware with decent CPU performance to hold up much much more than what I’ve tested.

    Time to ditch that firewall!

  • Being Ignorant About DDoS and Why Firewalls Suck

    I’ve just attended a one day “seminar” with folks at Arbor Networks and it has been insightful.

    It seems people are still pretty ignorant about DDoS attacks. Unlike the 1999 CIH virus that was programmed to take out a computer by corrupting it’s BIOS EEPROM, most of the viruses, worms, malwares and whatnots on the Internet today are around for one simple reason – money. Obviously if you’re good enough to write worms, you’d think “why write a worm for fun, when I can make money?” These worms infect computers to build Botnets, and Botnets are sold for real money on the black market to take down sites (via a DDoS), send spam, and all sorts of other things.

    There was one point in particular though that caught my attention, and it was that firewalls (or in fact any type of inline device such as load balancers) are potentially targets for DDoS attacks. To make matters worse, the higher the OSI layer the firewall capability goes, the worse it gets in terms of performance and reliability.

    Believe it or not, firewalls are vulnerable to serious security issues like buffer overflows just like any other server or appliance with an IP address. So it turns out that firewalls are the biggest marketing scam in the history of IT security because companies have spent millions and millions of dollars on these stuff that don’t offer much protection than say, iptables.

    Just about a month ago, I spoke to one of our customers who experienced a DDoS attack launched towards their co-location in the USA. The DDoS traffic was approximately 500Mbps and it completely took out the firewall. This site provided online payment services to customers and was up and down for days. Their firewall was tiny in comparison to the DDoS they got – on paper specs states performance capabilities of 90Mbps or 30Kpps at 2.8K sessions/sec with a max of 8K sessions at any time. Of course, these are lab specifications and real world traffic wouldn’t be as forgiving.

    A simple DDoS attack that’s merely 10Mbps in traffic volume would have generated millions of packets per second with a 1-byte  UDP or ICMP packet. Taking down such a firewall would be a breeze. In fact, a single modern day computer on a broadband connection could probably do the job.

    If it was a TCP SYN flood, it would have been way easier. Sending 2K TCP SYN packets per second is child’s play, so filling the firewall’s state table really takes no more than 10 seconds.

    I had a chat with my wife who audits financial institutions (FIs) based on the PCI-DSS standard. Most FIs providing payment card services will have to conform to this standard. This standard, however, mandates that a firewall is required to comply. Unfortunately, most FIs have a pretty average Internet connectivity pipe – somewhat in the range of 20Mbps to 100Mbps. They scale their firewalls to their connectivity, so what they have, well, closely resembles the one I described earlier.

    So why were firewalls invented?

    Early operating systems didn’t provide packet filtering capabilities, so the early firewalls were really just stateless packet filters that basically routed (not NAT’ed) traffic and dropped unwanted requests based on simple IP, protocol and port numbers to services that weren’t supposed to be public. Then the idea of NAT came about (remember the days of WinRoute) to allow multiple computers on a LAN to share a single IP address on a WAN link. Some smart guy then figured, “oh well, let’s put servers on a private subnet and use the NAT technology to map public and private address spaces. This way, we’re safer!” Agreeably, that was the dumbest idea ever and is a PITA to manage, but millions of servers are configured this way today. Over time, these features were slowly incorporated into the all-in-one junkbox we now call the Firewall. Sweet.

    Personally, I don’t have a firewall sitting in front of my servers. All my servers are individually configured to run iptables (or ipfilter on Solaris, etc.). I am going to test the Linux TCP stack with Apache from a default CentOS install to see how much SYN flood it can hold up before giving up and maybe post some results here, including what I tweaked in the kernel.

  • Three Week Summary

    Renovation has officially completed; we made our final payment last Thursday and the final touches were done yesterday. All the furniture are in as well. Mode Studio didn’t screw up and delivered the remaining furniture as scheduled.

    I pay a lot of attention to detail and there were some minor issues with the renovation (as usual). Carpentry, plumbing and electrical works needed personal supervision but overall we rate our interior designer highly for their design creativity and project management. We are especially pleased with their flooring, painting and solid surface works.

    I’ve been very busy but I will post some photos of the completed work soon.

    Work as usual, hamsters are still alive and kicking. I’m trying to find some time to pick up my R/C hobby again as the sloping season’s back but my time has been all tied up by the house renovation. Cleaning a new house has proved to be quite a chore as the dust settles each day.

    I have also attempted cooking. BTW, it’s not exactly cheap for two persons to cook at home. Breakfast is easy but I haven’t been very successful with my first dinner dish. However, the wife makes nice Chinese soup dishes!

    It’s just one more month before the year comes to an end. Time to close yet another chapter and kick start 2010. It will be a very exciting new year!

  • Mode Studio and Red Apple Furniture Woes

    We bought furnitures from Mode Studio, a subsidary of Red Apple Furniture on Aug 24, 2009. It has been exactly two months and we have yet to recieve full shipment of our order.

    The initial delivery was expected before the end of  September (one month from the date of order). When we called to confirm our delivery, the sales person denied having agreed to such a date and told us we would only receive our order by mid October. Our sales person then went MIA after the first week of October and we had to follow up with another guy.

    Our sofa and dining table arrived two weeks ago. We’re still waiting for our dining chair and coffee table. Oh, it’s the China holidays, it’s the bad weather, order’s not submitted, order went missing, communication problem, shipping delay, wrong container. It’s just excuses and delays after delays.

    Their logistics company called to schedule the remainder of our order to be delivered yesterday but I won’t be home, so I asked to have them shipped tomorrow morning.

    Anyway, there’s a thread in HardwareZone Forums discussing some other customers’ experience with Red Apple Furniture and I would recommend that you read it.

    This is what I wrote about my sales experience:

    I didn’t ask for a ridiculous discount. In fact, I was “offerred” a package when I stepped in. A dining table, 4 chairs, L sofa and coffee table. All for approx $3.2k. Then the sales person added a GST after we agreed to the price. BTW, this is a sales gimmick. If you have ever visited my retail store www.whymobile.com, you will know that I am very familiar with these sales tricks especially in my trade. If you want more info, read the Mission Statement under “About Us”.

    Anyway, after giving them a weird face (I almost wanted to walk out), they said they will waive the delivery charge. IMHO, $3.2k is not a ridiculous price. You get about that kind of prices for china made furniture everywhere. ($1.4k sofa, $800 dining table, $400 for 4 chairs, $400 for coffee table, with some spare change for delivery.) I credit them for better design but certainly if you were to compare Ikea’s prices, product quality and after-sales service, they’re no where near.

    Here’s another thread at RenoTalk.

    I never had a hiccup with Ikea’s logistics. Their products are of great quality considering the prices. I never really had an Ikea thing fail on me like many others said they would. I have a 10+ year old Ikea study table still straight and solid. Like Ikea furnitures today, it’s made of the same MBF 10 years ago. There’s also an Ikea bookshelve at the back of WhyMobile. It’s battered day in and out in a busy retail store and it’s still stands straight.

    I’m happy to have spent over $2k on Ikea furnitures for my new home with no hiccups at all. After all these fiasco, I declare myself a supporter of Ikea furnitures and food.

    If you’re buying Ikea, don’t expect your $39 table to last a lifetime, duh. Buy something of higher quality at Ikea.

  • Flash Sites are Passé; The DOs and DON’Ts of Web Design

    I’m surprised to find Renoma Paris’s site (in English) made entirely out of Flash. While it took ages to load, it also played an annoying music that I couldn’t turn off unless I turned down my speakers.

    Once the page loaded, I was presented with a scrolling marquee of images. They were so small that I couldn’t figure out what they were, so I clicked on any random image that passes and it brought me to yet another page that required loading. I sat waiting and stared at the red squares in the middle of the screen as more of them appeared after several seconds.

    Frustrated, I closed my browser tab. I was on the site for barely two minutes.

    This is a classic example of how your site can literally drive people away. Try it yourself – go visit that site.

    Many business owners don’t understand that what they like to have on their own site isn’t necessarily what people want to see.

    Here’s some of my personal DOs and DON’Ts of web design.

    • DON’T use flash for your entire site. It’s not only slow and heavy on a computer’s CPU, it doesn’t scroll well within a browser, it renders fonts differently from browsers making them difficult to read at times, the back and forward buttons don’t work, etc. The list of problems are endless. Oh, and did I mention that those Flash guys charge an arm and two legs? Don’t use flash. Period.
    • DON’T embed audio into your pages. It might give an old lady a heart attack, or simply just piss young people off by distorting whatever Wonder Girls track they’re listening to at the moment.
    • DON’T use a splash page. They only serve to delay a user’s entrance into your site. 9 in 10 splash pages I’ve seen have no real purpose other than the intent to create a “grand entrance” to a site. People visit web sites in search for content and will gladly click on the first sight of an “ENTER” button.
    • DON’T upload full resolution photos and simply use the HTML width and height attributes to resize your images. Resize  images using an image editing program like Adobe Photoshop or GIMP to achieve optimal image quality and file size.
    • DON’T underestimate the power of image compression. Choose wisely between GIF, JPEG and PNG compression and experiment which works best for you. GIF generally works well with text, JPEG works well with photos and PNG works well if transparency is involved. When used incorrectly, your images will not only look bad, it will consume unnecessary storage and bandwidth.
    • DON’T pop shit windows up. It’s not only annoying but confusing. Open the next page in the same window – people know how to use the back button on the browser.
    • DON’T use FORM POSTs excessively. This is what most Java and ASP.NET developers don’t quite understand. FORM POSTs (or POSTBACKs) not only prevent the back button on the browser from working, they also prevent caches from doing their jobs.
    • DO engage a third party to check for grammar, spelling and content accuracy. Badly written content translates to a bad user experience.
    • DO test your web site over a real Internet connection at home to check its loading time. Most sites load in a split second over a LAN but not over the Internet.
    • DO read up on how to make your site cache friendly, especially if your site handles lots of traffic. ISPs spend tonnes of money on web caches to conserve their bandwidth and yet web caching is one of the most misunderstood technology on the Internet. When your site is made cache friendly, ISP caches will greatly improve your users’ experience especially if they are far away.
    • DO add more line spacing. It’s easier on the eyes.

    There’s much more to web design than this short list though. Here’s my golden rule – humans like control. Give it to them.

    On a side note, I provide consultation for web marketing. Feel free to drop me a (private) message.