Distributed Denial of Service – December 26 to 27

Last December 26 and 27, we experienced a Distributed Denial Of Services Attack (DDoS). This led to problems with accessing our site, and Time Proof.

The attack came from bot network infected and under control of the perpetrators. All were focused to send out HTTP GET, POST, CONNECT and OPTIONS or other means necessary to flood (in our case, tsunami) our servers with these request and overwhelm it with replies, leaving the users unable to connect to our site.

Other parts of the attack include SQL injection and Browser hijacking by means of Header and User Agents.

The intent of the attack was to overwhelm the servers, and deny users the use of onlinejobs. It was not an attempt, not was there any data breach from the site.

Timeline:

PST = Pacific Standard Time (UTC -8)

Dec 26 at around 8:00 AM PST

A DoS attack was observed coming from Pakistan and presumably a test before the second attack came later that same day.

The attack was mitigated by blocking a set of Pakistan IP Blocks to stop it. It was successful and there was a short downtime of about 8 seconds due to webserver reload for the block and adjustments to take effect.

Dec 26 at round 4:00 PM PST

Another DoS attack was observed this time coming from the Philippines, which is presumably a test before the full attack.

The attack was mitigated by blocking a set of PLDT (ISP) IP blocks to stop it.

Dec 26 at 6:00 PM PST

The Distributed-Denial-of-Service Attack started, coming from all over the world and specifically from the following countries (not in order)

China

Russia

India

Indonesia

United Kingdom

Singapore

Philippines

United States of America

Canada

Netherlands

Germany

Thailand

Vietnam

Brazil

And other countries

Dec 26 at 6:10 PM PST

Local server firewall installed cannot effectively block the attacks anymore and had led to the server resources such as CPU at RAM to max at 100% causing downtime and effectively overwhelming services and functionality of the site. The best way to stop it is to shutdown the entire server to possibly let the attack pass and stop in a few hours.

However, attacks continued until the next day and we can only dodge what was coming in by blocking IP address manually

Dec 27 at around 11:00 AM PST

After further deliberations to mitigate the attacks, the IT team concluded to sign up with Cloudflare services and started configuring what is necessary for it to protect our site.

Dec 27 at around 3:00 PM PST

Additional configuration was set at Cloudflare and waiting for activation of the service which took almost 24 hours.

Attacks still ongoing as of the date above and was only observing and adjusting what is necessary to manage all server resources adequately

Dec 27 at around 11:00 PM PST

Cloudflare is activated and the IT team started configuring Firewalls as well as enabling WAF and other attack mitigating functionalities

Dec 28 at around 7:00 AM PST

IT team continued to configure all the settings required for Cloudflare, AWS and the server to finally stop all the attacks.

Dec 28 at around 9:00 PM PST

All adjustments and configuration was running and the site is back online, Cloudflare is blocking all unnecessary traffic and exploits

The following graph shows the timeline as it happened above.

ddos-summary


Here are some added illustrations on DDOS Attacks:

How DDoS works

DDoS with valid users

And here are the steps taken to mitigate and protect our site.

CF attack mitigationProtection by CF

Comments

  1. Enrick Neil Reinoso says

    Thank you for providing the update and being transparent to what happened during those times.

    It was an obvious attack because of how our site is currently being utilized by a lot of jobseekers and employers!

    Keep up the good work, Onlinejobs.ph Team!

  2. Carl Arjona says

    Good job on IT team and rest of the OLJ on protecting the website and the dreams of our fellow freelancers, Mabuhay kayo and more power!

  3. shem says

    I now understand why during those days, we’re not able to open the site I thought I lost the account already. So thank you for these updates and this is a great explanation about what happen. I’m glad the site is okay now! God bless to you guys!

  4. Ruth Serrano Raquel says

    thank you so much for the efforts of protecting the system.
    it means a lot to small freelancers like myself.

    KUDOS to the ADMIN

    Your Freelancer,
    Ruth Serrano Raquel

  5. DC says

    Good job team! IDK what is the motive of the attack other than to sabotage the site. This is well performing site that aims to connect work to Filipinos without cutting their throats with large fees. We need this platform so thank you for securing it. Greatly appreciated!

  6. Jeffrey Eugenio says

    Good job mitigating these attacks. Despite these, timeproof’s offline tracking still works and thankfully, everything goes back to normal.

    Keep up the good work!

  7. Ayan Yazon says

    I think the attackers from all those countries are the same and just using an IP hider. I hope everything is alright now!
    PS hoping to find a new job here! Hahaha

  8. Elmer says

    Thanks for the updates that sent to us hope everything will be fine, Perhaps another option can be implemented in case of attacks by creating backup servers for continuity of business and to avoid downtime.

  9. Pink says

    Thank you for letting us informed of what happened about those days where I thought my access to the site was forever be denied.

  10. Elisa Fe A.Daguinod says

    The tedious tasks done by the admin. to battle up with the attack of the hackers for about three days was not an easy one.I could feel the care/concern of the administrators to their benefactors…this is just a sign that you are the unsung heroes of these online job seekers & with online jobs…job well done & to God be the glory…I really appreciate your selfless efforts,though I’m not hired yet,as I’m just on my way of completing my application,I can sense that you’re happy serving your constituents….Good luck,God bless & more power!!!

  11. Gilbert says

    can’t understand why they can be so harsh to a website that offers help and support to millions of Filipinos who wish a decent life for themselves and their families, in exchange for decent work and service. thanks onlinejobs.ph for holding the fort.

  12. Ritchelle says

    I thought I had an acct logging in problem as I can’t access nor browse job on those dates. Thank you so much for the information. Hope everything is okay now.

    Job well done for OLJ IT team and for the whole team, thank you for keeping the website safe. This website is a big help to freelancers like me. A BIG thank you everyone!

    God Bless!

  13. Marvin L. says

    Wew very nice and clear exposition from the IT team. They must have been CompTIA Security+ certified.

    Thanks for the continued service guys.

  14. Harryjo F Estremera says

    Thanks for the hard work you’re doing out there to keep this site up. Kudos to the team for the job well done

  15. Robert says

    Thanks for the info and this will help us to understand why there is some slowness during those times. Hope everything is ok.

  16. Jo Apelanio says

    And also my account posted some “negative” feedback from “scammer – a client wanna be. Please this Onlinejobs admin also helps and protects workers/freelancers inside this platform and not so favors on the outside visitors like the client specially.

  17. Waren says

    Awesome! It could be a big data breach if those hackers succeed on their attacks. Don’t forget that there are some cybersecurity professional freelancers on your system including me. If you need help you can also contact me.

    What all I can say is to protect the freelancer’s information here at all cost.

    Kudos to those sleepless IT in your department. I know what it looks like hahaha

  18. bev says

    i can not download timeproof app. I already tried many times but I was not able to download it.

    What will I do with this?

    I badly need the app right now because i need to track my job. Thanks.

Leave a Reply

Your email address will not be published. Required fields are marked *