Last December 26 and 27, we experienced a Distributed Denial Of Services Attack (DDoS). This led to problems with accessing our site, and Time Proof.
The attack came from bot network infected and under control of the perpetrators. All were focused to send out HTTP GET, POST, CONNECT and OPTIONS or other means necessary to flood (in our case, tsunami) our servers with these request and overwhelm it with replies, leaving the users unable to connect to our site.
Other parts of the attack include SQL injection and Browser hijacking by means of Header and User Agents.
The intent of the attack was to overwhelm the servers, and deny users the use of onlinejobs. It was not an attempt, not was there any data breach from the site.
Timeline:
PST = Pacific Standard Time (UTC -8)
Dec 26 at around 8:00 AM PST
A DoS attack was observed coming from Pakistan and presumably a test before the second attack came later that same day.
The attack was mitigated by blocking a set of Pakistan IP Blocks to stop it. It was successful and there was a short downtime of about 8 seconds due to webserver reload for the block and adjustments to take effect.
Dec 26 at round 4:00 PM PST
Another DoS attack was observed this time coming from the Philippines, which is presumably a test before the full attack.
The attack was mitigated by blocking a set of PLDT (ISP) IP blocks to stop it.
Dec 26 at 6:00 PM PST
The Distributed-Denial-of-Service Attack started, coming from all over the world and specifically from the following countries (not in order)
China
Russia
India
Indonesia
United Kingdom
Singapore
Philippines
United States of America
Canada
Netherlands
Germany
Thailand
Vietnam
Brazil
And other countries
Dec 26 at 6:10 PM PST
Local server firewall installed cannot effectively block the attacks anymore and had led to the server resources such as CPU at RAM to max at 100% causing downtime and effectively overwhelming services and functionality of the site. The best way to stop it is to shutdown the entire server to possibly let the attack pass and stop in a few hours.
However, attacks continued until the next day and we can only dodge what was coming in by blocking IP address manually
Dec 27 at around 11:00 AM PST
After further deliberations to mitigate the attacks, the IT team concluded to sign up with Cloudflare services and started configuring what is necessary for it to protect our site.
Dec 27 at around 3:00 PM PST
Additional configuration was set at Cloudflare and waiting for activation of the service which took almost 24 hours.
Attacks still ongoing as of the date above and was only observing and adjusting what is necessary to manage all server resources adequately
Dec 27 at around 11:00 PM PST
Cloudflare is activated and the IT team started configuring Firewalls as well as enabling WAF and other attack mitigating functionalities
Dec 28 at around 7:00 AM PST
IT team continued to configure all the settings required for Cloudflare, AWS and the server to finally stop all the attacks.
Dec 28 at around 9:00 PM PST
All adjustments and configuration was running and the site is back online, Cloudflare is blocking all unnecessary traffic and exploits
The following graph shows the timeline as it happened above.
Here are some added illustrations on DDOS Attacks:
And here are the steps taken to mitigate and protect our site.
Hope everything is secured.
Thank you for providing the update and being transparent to what happened during those times.
It was an obvious attack because of how our site is currently being utilized by a lot of jobseekers and employers!
Keep up the good work, Onlinejobs.ph Team!
Good job on IT team and rest of the OLJ on protecting the website and the dreams of our fellow freelancers, Mabuhay kayo and more power!
I now understand why during those days, we’re not able to open the site I thought I lost the account already. So thank you for these updates and this is a great explanation about what happen. I’m glad the site is okay now! God bless to you guys!
?
thank you so much for the efforts of protecting the system.
it means a lot to small freelancers like myself.
KUDOS to the ADMIN
Your Freelancer,
Ruth Serrano Raquel
Good job IT Team!!!
Thanks for the share. This makes sense and really helpful.
Thanks for your inisghts.This is definitely a great help for everyone.
Good job team! IDK what is the motive of the attack other than to sabotage the site. This is well performing site that aims to connect work to Filipinos without cutting their throats with large fees. We need this platform so thank you for securing it. Greatly appreciated!
Good job mitigating these attacks. Despite these, timeproof’s offline tracking still works and thankfully, everything goes back to normal.
Keep up the good work!
I think the attackers from all those countries are the same and just using an IP hider. I hope everything is alright now!
PS hoping to find a new job here! Hahaha
Thanks for the updates that sent to us hope everything will be fine, Perhaps another option can be implemented in case of attacks by creating backup servers for continuity of business and to avoid downtime.
Good Job!! Thank you for protecting this site!
I am glad that the site was secured promptly. I hope these perpetrators would stop doing this.
Thank you for letting us informed of what happened about those days where I thought my access to the site was forever be denied.
The tedious tasks done by the admin. to battle up with the attack of the hackers for about three days was not an easy one.I could feel the care/concern of the administrators to their benefactors…this is just a sign that you are the unsung heroes of these online job seekers & with online jobs…job well done & to God be the glory…I really appreciate your selfless efforts,though I’m not hired yet,as I’m just on my way of completing my application,I can sense that you’re happy serving your constituents….Good luck,God bless & more power!!!
Thanks for the update, great job!
can’t understand why they can be so harsh to a website that offers help and support to millions of Filipinos who wish a decent life for themselves and their families, in exchange for decent work and service. thanks onlinejobs.ph for holding the fort.
I just experience it. I was wondering and now I know. Thank you.
You’re choice for Cloudflare is great! You should have done earlier I guess.
Thankyou for doing all this steps to protect the website… 2 thumbs up to your team
Thanks for the update, it explains everything that happens on that dates. Thanks
Thank you for the update. Good job IT and all OLJ staff. Keep up the good work and more power!
Thank you for the update 🙂
Warmest regards to the I.T. team! Keep it up!
The transparency is highly appreciated, OnlineJobs.ph team.
I thought I had an acct logging in problem as I can’t access nor browse job on those dates. Thank you so much for the information. Hope everything is okay now.
Job well done for OLJ IT team and for the whole team, thank you for keeping the website safe. This website is a big help to freelancers like me. A BIG thank you everyone!
God Bless!
Thank you so much for sharing 🙂
Thank you so much for the efforts of protecting us and also for your insights.
More Power and God Bless!
Thank You so much for the effort, it really means a lot for us people working at home. God Bless
Good job guys!
Thank you so much for your hardwork.
Thank you for all Your efforts OLJ!!!!
Thanks for sharing this information.
Wew very nice and clear exposition from the IT team. They must have been CompTIA Security+ certified.
Thanks for the continued service guys.
Thanks for your inisghts.This is definitely a great help for everyone.
Thanks for the hard work you’re doing out there to keep this site up. Kudos to the team for the job well done
Thanks for the info and this will help us to understand why there is some slowness during those times. Hope everything is ok.
Thank You!
Thanks for that update! Very helpful and relieves some anxiety about the situation.
thank you for the update, it means a lot. God Bless
And also my account posted some “negative” feedback from “scammer – a client wanna be. Please this Onlinejobs admin also helps and protects workers/freelancers inside this platform and not so favors on the outside visitors like the client specially.
Hi Jo, you can email our support team (support@onlinejobs.ph) so that they can help you with your account and check it out as well
Thanks for notifying us.
Thank you IT team for protecting the system.
Mabuhay ang mga home based Pinoy.
Thank you for the update. Hope this will not happen again
We are so glad to know that everything is okay with OnlineJobs website now. Good luck OnlineJobs.
Awesome! It could be a big data breach if those hackers succeed on their attacks. Don’t forget that there are some cybersecurity professional freelancers on your system including me. If you need help you can also contact me.
What all I can say is to protect the freelancer’s information here at all cost.
Kudos to those sleepless IT in your department. I know what it looks like hahaha
thank you for protecting the site.
Thank you!
i can not download timeproof app. I already tried many times but I was not able to download it.
What will I do with this?
I badly need the app right now because i need to track my job. Thanks.
Hi Bev, you can email our support team (support@onlinejobs.ph) so that they can assist you on your TimeProof right away