3.2.48: The C Release

I'm happy to announce ApisCP 3.2.48 has been released after a multimonth development marathon. This release introduces hidden critical changes to spur development over the next few releases as it transitions to ApisCP 4.0.

NSS/PAM rewrite

Previously, any NSS lookups to translate virtual logins into system userdata would save the root directory with fopen(), enter a chroot() syscall to load the appropriate passwd file, perform a normal getpwnam() request, then exit from chroot() with an fchdir() syscall. It's a fairly straightforward process that is one of many techniques to break out of chroots but with serious limitations:

• Non-reentrant. If a signal is received that processes a code segment elsewhere, then that code is stuck with that new root context.
• In a multi-threaded environment any such chroot() would change root context for all threads pooled in the same process space. Any parallel processing would adopt the new root context thus violating thread safety.

chroot() has been moved from NSS to PAM. Any lookups now report a reusable username composed of username@siteXX. The old chroot behavior can be restored by passing env _NSS_AUTOJAIL=1 to a process.

Previously, gentent msaladna@apiscp.com would return msaladna:x:9999:1000::/home/msaladna:/bin/bash, which loses the site identifier. Once the login changes, it becomes incompatible with future lookups. Now, login resolution reports a schema that can be looked up while preserving the correct data, msaladna@site1:x:9999:1000::/home/msaladna:/bin/bash.

Further, translations are durable: site identifiers never change once an account is created on a server. User IDs are also durable; usernames are not. Postfix delivers locally to UID@SITEXX ensuring deliverability in rare circumstances where mail cannot be immediately delivered (temp fail: overquota, bad .mailfilter grammar, Postgres down), then during that window the domain is dropped or username changed. Previously, it would result in an undeliverable situation.

Durable logins when used with systemd's RootDirectory= directive now allow for services to launch direct from systemd - including timers. That means the next release of ApisCP will enhance security further by supporting multiple PHP pools per account as well as assigning pool ownership to any secondary user. A future release will further improve performance by swapping individual crond processes with monolithic systemd timers.

mod_shield WAF

Bot spam is only getting worse. mod_shield is a reimage of mod_evasive that shares request data across workers, implements TTL/LRU auto-eviction in a lightweight cyclic buffer, and introduces new scoring dimensions that target both HTTP response codes and response times.

CPU cycles are never free. To contend with traditional botnets of compromised smart devices, we're now inundated with a deluge of derelict VPSes as well as the LLM arms race. It's not pretty. Shield is a unique module to control abusive IPs while permitting genuine traffic.

Let's look at a common request pattern from 52.169.252.46, a compromised Azure server performing random probes against apiscp.com running Laravel.

52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.0536 -0400] "GET /wp-content/themes/signify/firkon.php HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.1554 -0400] "GET /wp-content/shell20211028.php HTTP/1.1" 301 255 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.0536 -0400] "GET /wp-content/shell20211028.php HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.1554 -0400] "GET /wp-content/ice.php HTTP/1.1" 301 245 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.0536 -0400] "GET /wp-content/ice.php HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.1554 -0400] "GET /wp-admin/css/colors/sunrise/ HTTP/1.1" 301 255 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.0536 -0400] "GET /wp-admin/css/colors/sunrise/ HTTP/1.1" 301 254 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.0536 -0400] "GET /wp-admin/css/colors/sunrise HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.1554 -0400] "GET /wp-includes/css/dist/components/ HTTP/1.1" 301 259 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.0536 -0400] "GET /wp-includes/css/dist/components/ HTTP/1.1" 301 258 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:12.0536 -0400] "GET /wp-includes/css/dist/components HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.0998 -0400] "GET /wp-content/xleet.php HTTP/1.1" 301 247 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.1905 -0400] "GET /wp-content/xleet.php HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.0998 -0400] "GET /wp-content/plugins/tunnelforms/lib.php HTTP/1.1" 301 265 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.1905 -0400] "GET /wp-content/plugins/tunnelforms/lib.php HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.0998 -0400] "GET /wp-content/plugins/linkpreview/ HTTP/1.1" 301 258 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.1905 -0400] "GET /wp-content/plugins/linkpreview/ HTTP/1.1" 301 257 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.1905 -0400] "GET /wp-content/plugins/linkpreview HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.0998 -0400] "GET /wp-admin/images/ HTTP/1.1" 301 243 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.1905 -0400] "GET /wp-admin/images/ HTTP/1.1" 301 242 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:13.1905 -0400] "GET /wp-admin/images HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.0534 -0400] "GET /wp-includes/customize/ HTTP/1.1" 301 249 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.1440 -0400] "GET /wp-includes/customize/ HTTP/1.1" 301 248 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.1440 -0400] "GET /wp-includes/customize HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.0534 -0400] "GET /wp-content/plugins/cakil/ HTTP/1.1" 301 252 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.1440 -0400] "GET /wp-content/plugins/cakil/ HTTP/1.1" 301 251 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.1440 -0400] "GET /wp-content/plugins/cakil HTTP/1.1" 404 1331 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.0534 -0400] "GET /wp-includes/random_compat/about.php HTTP/1.1" 301 262 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.1440 -0400] "GET /wp-includes/random_compat/about.php HTTP/1.1" 429 227 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.0534 -0400] "GET /wp-content/updates.php HTTP/1.1" 429 227 "-" "-"
52.169.252.46 apiscp.com - [29/Jul/2025:12:37:14.0534 -0400] "GET /wp-content/plugins/HelloDollyV2/hello_dolly_v2.php HTTP/1.1" 429 227 "-" "-"

First, what is the cost of a request? It's about 40 ms.

curl -o/dev/null -s -w '%{time_total}s' https://apiscp.com/wp-includes/random_compat/about.php

That's not bad, but over the span of 1 second given the hit rate (10 req/sec), it's approximately 400 ms of processing less asynchronous IO. Let's look at a WordPress site, for each invalid request a separate dispatcher (index.php) is invoked. Likewise, a modest number of plugins are loaded.

# wp-cli plugin list
+--------------------------+----------+-----------+---------+----------------+-------------+
| name                     | status   | update    | version | update_version | auto_update |
+--------------------------+----------+-----------+---------+----------------+-------------+
| advanced-accordion-block | inactive | none      | 5.0.4   |                | off         |
| akismet                  | active   | none      | 5.5     |                | off         |
| xxxxxxxxxxxxx            | active   | none      | 1.2     |                | off         |
| classic-widgets          | active   | none      | 0.3     |                | off         |
| find-my-blocks           | inactive | none      | 4.0.3   |                | off         |
| jetpack                  | active   | none      | 14.8    |                | off         |
| jetpack-boost            | active   | none      | 4.2.1   |                | off         |
| just-writing-statistics  | active   | none      | 5.4     |                | off         |
| mailchimp                | active   | none      | 1.9.0   |                | off         |
| reading-time-wp          | active   | none      | 2.0.17  |                | off         |
| responsive-lightbox      | active   | none      | 2.5.2   |                | off         |
| google-site-kit          | inactive | available | 1.157.0 | 1.158.0        | on          |
| amazon-s3-and-cloudfront | active   | none      | 3.2.11  |                | off         |
| db-error.php             | dropin   |           |         |                | off         |
+--------------------------+----------+-----------+---------+----------------+-------------+

It's 282 ms. 7x worse than a simple Laravel site. Now let's add Wordfence, a popular WAF that depends upon the WordPress to protect itself.

# wp-cli plugin install --activate wordfence

Downloading installation package from https://downloads.wordpress.org/plugin/wordfence.8.0.5.zip...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'wordfence'...
Plugin 'wordfence' activated.
Success: Installed 1 of 1 plugins.

Zero further configuration. Retest. 387 ms per request. Latency has increased 37%; this impacts every pageview. Shield provides universal protection, virtually no overhead (μs in patrol mode, negative overhead in blocking mode), and pierces Cloudflare proxies. From cursory testing, mod_shield handles over 40k rps on a single logical core Ryzen 7 5700X without any performance degradation.

Or go nose-first into the docs

D-Bus PECL memleaks

I ❤️ systemd. It solves issues that have plagued SysV distributions: streamlines complex dependency management, adds service grouping, init script overrides, resource tracking, socket-based activation, degraded service remediation, et cetera. systemd also hooks into the D-Bus wire protocol for seamless IPC communication.

ApisCP began communicating over D-Bus in 3.2.36.1 but had some rather nasty memory leaks that scaled proportional to service counts. Those have been resolved in this release.

Now that systemd can translate any user on any site into its corresponding passwd entry - and memory isn't randomly going out to lunch - it's possible to shift away from per-site vixie-cron instances to systemd timers in a future release. What this means is:

• Tasks have a replayable log buffer without specifying MAILTO=
• Tasks can have dependency chains– e.g. don't run X unless service Y is active, such as Discourse
• Last run, last run state. Presently it's approximated naively
• No per-site cron daemons so no filesystem ghosting. File handles are returned at end of execution. Previously with vixie-cron libraries could remain loaded such as with libc RPM updates