Debugging basics

📝 This is part of an initial documentation release. See DEBUGGING.md for up-to-date information.

ApisCP may optionally emit debugging information when debug mode is enabled. Debug mode may be set per-request lifetime using the DEBUG environment variable or enabled globally using the cp.debug Scope. Backtraces, which are code pathways, can be enabled by setting [core] => debug_backtrace_qualifier. Backtraces give context to how an error was encountered and should always be enabled when debugging problematic code.

cpcmd scope:set cp.debug true enables debugging mode globally.

cpcmd scope:set cp.config core debug_backtrace_qualifier -1 enables backtraces for all reporting classes (fatal, error, warning. info, deprecated, and debug) as well as exceptions. Increasing verbosity levels inherit lower levels. Backtraces are mandatory when reporting issues.

The following table is only relevant when panel debugging is enabled.

env DEBUG=1 cpcmd common:whoami executes the whoami method in common module. This method simply returns the current username. A special module test is available in debug mode to facilitate development. Only this request operates in debug mode ensuring appropriate isolation in a production environment.

# env DEBUG=1 cpcmd test:benchmark common_whoami
benchmark common_whoami
time: 0.01 sec (1000 rounds; 0.0051 ms each; 195429.32 per second)

0.0051169395446777

# env DEBUG=0 cpcmd test:benchmark common_whoami
ERROR   : test_benchmark: command does not exist
----------------------------------------
MESSAGE SUMMARY
Reporter level: ERROR
----------------------------------------
ERROR: test_benchmark: command does not exist

When in debug mode, housekeeping and cron services are disabled as well as job runner. Housekeeping/cron tasks may not be invoked traditionally. Job runner invocation is covered under Jobs.

Targeted frontend debug

misc:debug-session(string $id, bool $mode = true) enables debugging for a frontend session (DAV, UI, SOAP). Session identifier may be retrieved from a browser session by accessing the session.id property. session.debug encodes whether debug mode is enabled.

# Enable debugging for active session aE2...JL9r in browser
env DEBUG=1 misc:debug-session aE2uNGVkvpathoXqhofnsKDDXGNBJL9r true

Log locations

ApisCP logs messages in a few places. Respective services use their preferred logging locations. This table summarizes common services and their log locations.

All locations are within /var/log unless noted. siteXX is shorthand for /home/virtual/siteXX/fst/. siteXX is the site ID identifier that can be resolved using get_site_id domain.com. "..." following siteXX is short-hand for /var/log thus siteXX ... log indicates /home/virtual/siteXX/fst/var/log/log. CP_ROOT is the panel home, typically either /usr/local/apnscp or /usr/local/apiscp. Words fully capitalized are symbolic.

Automated email reporting

ApisCP may be configured to forward a copy of all unhandled errors - PHP notices/errors and exceptions - to an email address. Set [core] => bug_report in config.ini. This should only be used for developers as it generates false positives that are encountered during typical operation.

cpcmd scope:set cp.config core bug_report email+bugs@domain.com

Given the volume generated, plussed address notation or a separate email address is recommended to facilitate mail filtering by your SMTP provider.

Backend

Backend broker may be started from command-line in the foreground. Backend broker handles elevation requests from the frontend via query() in the API. When in debugging mode, cron and housekeeping tasks are disabled as are jobs.

systemctl stop apiscp
cd /usr/local/apnscp

env DEBUG=1 ./apnscpd -f restart

PHP-FPM status

FPM pools are grouped by site ID identifier and name. By default, one pool is created named after the primary domain. php:pools() lists active pools for a site.

php:pool-status(string $pool = '') provides the internal PHP-FPM pool status as reported via systemd's notify feature. These values are real-time metrics as seen by the pool manager.

php:pool-info(string $pool = '') reports service information from systemd. This command is equivalent to systemctl show POOLNAME. StatusText is the plaintext value of php:pool-status.

Jobs

Laravel Horizon is used for jobs unless has_low_memory is enabled during Bootstrapping or [cron] => low_memory is set in config.ini. Horizon may be launched manually using:

./artisan horizon

Pending work queue is reported by misc:get-job-queue(). Pending jobs may be processed using ./artisan queue:work. An optional flag, --once, processes these jobs singularly.

# Disable job runner, housekeeping, and cron to prevent jobs from starting
cpcmd scope:set cp.debug true
systemctl restart apiscp

# Validate the queue is empty
cpcmd misc:get-job-queue
# Jobify a command
cpcmd misc:jobify 'common_whoami'
# Validate the queue has 1 job
cpcmd misc:get-job-queue
# Run first job in queue, enable verbose output
./artisan queue:work --once -vvv
# Validate queue is now empty
cpcmd misc:get-job-queue

Jobs are unavailable when the panel is in debug mode unless Horizon has been manually started.

Web App installation

Web Apps engaged through the UI are dispatched to a job runner. Web Apps may be installed directly using the API commands. Each web app maps to a module named after itself. All installations follow a common interface:

NAME:install(string $hostname, string $path = '', array $options = [])

Applications support specific options as well as generalized options. The following are common options found in Web > Web Apps:

Jobs are unavailable when the panel is in debug mode unless Horizon has been manually started.

Command listing

misc:list-commands(string $filter = '') is a role-aware helper that displays available commands for a given role. Used in conjunction with cpcmd, it provides a convenient interface to filter available commands.

# Show commands available to Appliance Administrator ("admin" username)
cpcmd misc:list-commands
# Show commands available to site1 Site Administrator
cpcmd -d site1 misc:list-commands
# Show commands available to site1 in the "ghost" module
cpcmd -d site1 misc:list-commands 'ghost:*'
# "l" is an alias and equivalent to the above command
cpcmd -d site1 misc:l 'ghost:*'

Introspection

misc:command-info(string $filter = '') provides verbose information on the command including its method signature and documentation. This can be used to elucidate what parameters an API command anticipates. Method usage is similar to list-commands:

# Show signature for ghost:install as Site Administrator
cpcmd -d site1 misc:command-info ghost:install
# Show command signature for all commands in "admin" module
cpcmd misc:command-info 'admin:*'
# "i" is an alias and equivalent ot the above command
cpcmd misc:i 'admin:*'

User preferences

Preferences are stored in siteXX/info/USER. common:load-preferences() is a convenient interface to show these preferences.

common:get-user-preferences(string $user) allows for a Site Administrator access to a user's preferences.

YAML_INLINE is an environment variable that controls array folding depth. Increasing folding depth improves readability. The default value is 2.

# Show Appliance Administrator's preferences
cpcmd common:load-preferences
# Show preferences for site1's Site Administrator
cpcmd -d site1 common:load-preferences
# Show preferences for user "foobar" on site1
cpcmd -d site1 -u foobar common:load-preferences
# The following command is equivalent
cpcmd -d site1 common:get-user-preferences foobar
# Use YAML_INLINE=n to expand collapsed fields
env YAML_INLINE=4 cpcmd -d site1 common:load-preferences

Connecting to Redis

Redis manages caching as well as job queues over a UNIX domain socket. Database 1 is assigned to ApisCP, 2 to jobs, and 3 to rspamd if utilized. Database volatility is arranged in decreasing priority with database 1 most volatile and database 3 least: do not issue the FLUSHALL command as this will purge rspamd logical replication from PostgreSQL.

redis-cli -s /usr/local/apnscp/storage/run/redis.sock
# Show memory usage
info memory
# Show stored keys
keys *

API bypasses

API permissions may be side-stepped by using a surrogate module. A surrogate allows for rapid prototyping of individual API methods that may be restricted through normal operation. Surrogates are covered in detail in PROGRAMMING.md.

<?php declare(strict_types=1);

	class Dns_Module_Surrogate extends Dns_Module {
		public function __construct() {
			parent::__construct();
			// ensure we always win permissions
			$this->exportedFunctions = ['*' => PRIVILEGE_ALL];
		}
		
		public function t() {
			return $this->_cron();
		}
	} 

Now the _cron method, normally restricted by API accessibility rules, may be interacted with directly from command-line:

env DEBUG=1 cpcmd dns:t

Be careful with this approach. Any cross-module calls inherit the current role, which means you might try calling an API method intended for PRIVILEGE_SITE as PRIVILEGE_ADMIN. Cross-module violations are still blocked unless those permissions are also overwritten with a surrogate.