Our top consumer of open-source contributions this year is...
ApisCP incorporates a mixture of licenses allowing us to give back some parts of the platform to the community. Since moving packages to GitHub in 2017, we've contributed 56 packages including a fork of mod_evasive - a lightweight brute-force deterrence for Apache. Some packages exist solely for nostalgia, such as the original backend broker from 2003 while others are minor package updates over what's included with CentOS 7 such as mod_security with JIT PCREs. Many of these packages however serve to the betterment of the community at large, which is flattering when such work falls under the observation of larger competitors.
It comes with great honor to announce the 2020 top consumer of our GitHub contributions as early as January!
Drum roll please... 🥁🥁🥁
Congratulations DirectAdmin on using our fork of mod_evasive for your hosting platform some 17 years later. It's good to see such an initiative taken after receiving an email from a consumer of your software bemoaning problems with bots behaving badly, specifically:
Main issue we are currently facing are the DOS attacks from bot(nets) scanning for mainly WordPress bugs/SQL injection/file searches.
And it comes as no surprise to see this module integrated into DirectAdmin's Apache server a couple weeks later while performing a routine performance audit between DirectAdmin and ApisCP (more on that later).
If there's one takeaway from this, there is a hierarchy and so long as DirectAdmin continues to feed off our work, ApisCP will have the established leadership in delivering safe, secure hosting environments.
Update: since publishing this announcement DirectAdmin has scrubbed mod_evasive from its CustomBuild release and denied ever having used it. Opportunities to take jabs at competitors are in good sport, but by all means I recognize a 429 response when I see one whether it's at midnight or noon. CentOS does not stash Apache modules under /usr/lib on 64-bit platforms.Without such a package in its mirrors at present it's difficult to validate this statement. The above passage should be construed as satire to illustrate the emergent need for controlling resource usage.
Second Update: on further evaluation, mod_evasive modules compile differently on DirectAdmin and ApisCP platforms. Different flags result in different symbol layouts. The mod_evasive module referenced in the screenshot is indubitably fde81f and pulled down with a CustomBuild script update following install on Jan 15. It was likely removed later in the day following a remark on WHT about drafting off our work. It's OK to imitate us, but just be direct!
wget https://raw.githubusercontent.com/apisnetworks/mod_evasive/fde81f3f88f3cb9c68d90f0443c7f0e617ef85a4/SOURCES/mod_evasive.c ; apxs -c mod_evasive.c ; strings .libs/mod_evasive.so | grep -B10 -A7 DOSCanwill produce demonstrably different code on an ApisCP platform or CentOS in general, but identical layout on DirectAdmin indicating the module wasn't mainline and had been compiled on a DA platform. Idiosyncratic compiler flags produce markedly different code!
In addition to developing ApisCP, I've run Hostineer, a hosting firm split from Apis Networks in 2017, since 2002. This stack is methodically selected from a hosting provider's perspective to curb threats and squeeze performance every way possible. Better yet, the stack has proven to work year over year in reliability handling whatever comes its way by diminishing monopolization effects.
A need to restrain monopolization
mod_evasive is an important piece in controlling monopolization, which between the excerpt above from the email I received and DirectAdmin's hurried adoption without much consideration for tighter integration - is startling to see such a crucial piece go unchecked for so long in any stack. Bandwidth is cheap, latency an order of magnitude better than a decade ago, and still in the early stages of decentralization through virtualization. More self-anointed system administrators whose experience begins and ends with following a random blog post on setting up WordPress the wrong way or PHP + NGINX incorrectly, thus spawning CVE-2019-11043, hints at a larger problem with the current state of web security.
Hosting is a complicated field. Certain liberties can be taken to simplify system design such as zero configuration and package-based updates where appropriate, but by and large any platform must focus first on security and second on controlling monopolization. Monopolization, colloquially known as a "denial of service" attack, is any activity in which the output cost far exceeds the input cost. If an input cost of 1 generates an output cost of 4, all things being equal, 1 machine could saturate 4 machines preventing any further activity from the 4 machines. In real world applications, these figures are skewed to much larger orders of magnitude (1:1000 or more!).
Attacks come in a variety of forms whether it be from a bad query that joins 10 billion records thus creating excessive swapping problems and thus IO contention, low cost ESMTPA attempts driving up server process counts to handle each connection in its own process space, fork bombs, or the perennial plague of throttling abusive web requests without disrupting normal traffic flows.
Monopolization illustrated
Let's step back and look at SMTP authentication attempts, which is often overlooked as part of any threat deterrence package.
smtp-auth.php queries a local SMTP server. "postfix.service" is part of the systemd cgroup layout in /sys/fs/cgroup/memory/system.slice/postfix.service.
Querying SMTP cost 420 microseconds while Postfix incurred ~10240 microseconds in processing the SMTP authentication challenge as well as incidentals. Performing a few extra checks, let's control for incidental processing costs in Postfix, what it does while it's idle, with a rudimentary calculation in bash. At idle, Postfix consumes ~20-30 microseconds per second of CPU time per its usage controller.
# for i in $(seq 1 10) ; do let T1=$(cat cpuacct.usage) ; sleep $SLEEP ; let T2=$(cat cpuacct.usage) ; echo $((($T2-$T1)/$SLEEP/1000)) ; done
22
29
33
22
19
68
20
24
10
15
Substract those numbers and it's still clear that the input is less than the output costs, but wait - there's more! cgroups don't update counters immediately, there is a slight delay. Let's add a 10 second window then recalculate to find the actual drain by just Postfix - ignoring the drain on third-party authentication services and syslog:
(224673630235 ns - 224623061891 ns)/1000 ns/us - 10 seconds x 30 microseconds incidental usage/second = 50268 microseconds of usage per ~400 microseconds input, a 125x output cost.
As stacks grow in complexity, dependency chains evolve, and services bear the cumulative burden of finalizing an action it's important to take steps up front to filter good from the bad.
Any system that accepts arbitrary input from untrusted sources must take reasonable action to stem abuse from untrusted sources. A malicious actor is more likely to exist anonymously than one identified. Netiquette is dead, bots don't follow the two concurrent requests recommendation from the IE7 days. In fact, HTTP/2 multiplexing has obviated it altogether.
mod_evasive bean counting
HTTP is by far the most common and convoluted pipeline in processing requests. There's overhead from the HTTP server, database server, application server, perhaps proxy server, IO from logging, cache server, and so on. HTTP is a lucrative target too allowing attackers to host phishing sites and gain confidential information - including medical records. It's a huge problem. Bots attack frenetically to tease out credentials.
In fact most of the hacked email accounts I see on Hostineer have had their credentials previously dumped on haveibeenpwned.com. There's also a correlation between large breaches and an uptick in hacked accounts.
mod_evasive tallies accounts over a window and makes brute-force very costly for bots much like what reCAPTCHA did for bots algorithmic ability in solving CAPTCHAs. For example, the following is a request that triggered an automated block with increased sensitivity on xmlrpc.php, which is considered a high-value URI.
123.148.211.92 [18/Jan/2020:14:52:28 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:29 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:29 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:30 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:51:54 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:51:58 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:02 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:04 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:10 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:11 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:15 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:17 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:18 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:20 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:23 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:25 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:26 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:27 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
123.148.211.92 [18/Jan/2020:14:52:31 -0500] "POST /xmlrpc.php HTTP/1.1" 200 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
ASN points back to AS4837, China168 Backbone. Surprise! In total, 1,922 requests were from malicious IPs out of 487,647 over half a day in which 19 IPs were banned for violating generous allowances (20 same URI requests in 2 seconds or 300 requests in 4 seconds) resulting in significant CPU cycle savings. Saving CPU up front keeps a server running smoothly and your customers happy.
What good is a downed server anyway?
On those benchmark numbers, a 2 GB NVMe high-frequency compute node from Vultr with PHP-FPM 7.3, ApisCP churned out 61.67 requests per second, DirectAdmin 40.15 requests per second. A discrepancy exists between total data transferred as the blog title differed.
Concurrency Level: 1
Time taken for tests: 24.907 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 25772000 bytes
HTML transferred: 25504000 bytes
Requests per second: 40.15 [#/sec] (mean)
Time per request: 24.907 [ms] (mean)
Time per request: 24.907 [ms] (mean, across all concurrent requests)
Transfer rate: 1010.49 [Kbytes/sec] received
versus
Concurrency Level: 1
Time taken for tests: 16.215 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 25947000 bytes
HTML transferred: 25630000 bytes
Requests per second: 61.67 [#/sec] (mean)
Time per request: 16.215 [ms] (mean)
Time per request: 16.215 [ms] (mean, across all concurrent requests)
Transfer rate: 1562.64 [Kbytes/sec] receivedThey still have more catching up to do, which might just be easier for them to whitelabel ApisCP going forward.
cpcmd scope:set core panel_brand "DirectAdmin"