Operations Manual

Installation

Step 0 - Check dependencies are installed

Member Audit is a plugin for Alliance Auth. If you don’t have Alliance Auth running already, please install it first before proceeding. (see the official AA installation guide for details)
Member Audit needs the app django-eveuniverse to function. Please make sure it is installed, before before continuing.

Step 1 - Install app

Make sure you are in the virtual environment (venv) of your Alliance Auth installation. Then install the newest release from PyPI:

pip install aa-memberaudit

Step 2 - Configure Auth settings

Configure your Auth settings (local.py) as follows:

Add 'memberaudit' to INSTALLED_APPS
Add below lines to your settings file:

CELERYBEAT_SCHEDULE['memberaudit_run_regular_updates'] = {
    'task': 'memberaudit.tasks.run_regular_updates',
    'schedule': crontab(minute=0, hour='*/1'),
}

Optional: Add additional settings if you want to change any defaults. See Settings for the full list.

Step 3 - Finalize App installation

Run migrations & copy static files

python manage.py migrate
python manage.py collectstatic

Restart your supervisor services for Auth

Step 4 - Update EVE Online API Application

Update the Eve Online API app used for authentication in your AA installation to include the following scopes:

esi-assets.read_assets.v1
esi-bookmarks.read_character_bookmarks.v1
esi-calendar.read_calendar_events.v1
esi-characters.read_agents_research.v1
esi-characters.read_blueprints.v1
esi-characters.read_contacts.v1
esi-characters.read_corporation_roles.v1
esi-characters.read_fatigue.v1
esi-characters.read_fw_stats.v1
esi-characters.read_loyalty.v1
esi-characters.read_medals.v1
esi-characters.read_notifications.v1
esi-characters.read_opportunities.v1
esi-characters.read_standings.v1
esi-characters.read_titles.v1
esi-clones.read_clones.v1
esi-clones.read_implants.v1
esi-contracts.read_character_contracts.v1
esi-corporations.read_corporation_membership.v1
esi-industry.read_character_jobs.v1
esi-industry.read_character_mining.v1
esi-killmails.read_killmails.v1
esi-location.read_location.v1
esi-location.read_online.v1
esi-location.read_ship_type.v1
esi-mail.organize_mail.v1
esi-mail.read_mail.v1
esi-markets.read_character_orders.v1
esi-markets.structure_markets.v1
esi-planets.manage_planets.v1
esi-planets.read_customs_offices.v1
esi-search.search_structures.v1
esi-skills.read_skillqueue.v1
esi-skills.read_skills.v1
esi-universe.read_structures.v1
esi-wallet.read_character_wallet.v1

Step 5 - Verify Celery configuration

Please note that Member Audit will not work with the process based celery setup from the official AA installation guide!

If you have not yet switched to a thread-based celery setup, please see this step-by-step guide on how to do so: Configuring celery workers

Step 6 - Load Eve Universe map data

In order to be able to select solar systems and ships types for trackers you need to load that data from ESI once. If you already have run those commands previously you can skip this step.

Load Eve Online map:

python manage.py eveuniverse_load_data map

python manage.py memberaudit_load_eve

You may want to wait until the loading is complete before continuing.

Hint

These command will spawn a thousands of tasks. One easy way to monitor the progress is to watch the number of tasks shown on the Dashboard.

Step 7 - Enable optional features

Some features are disabled by default. Please review and decide which features you want to enable for your installation:

Corporation roles: Add MEMBERAUDIT_FEATURE_ROLES_ENABLED = True to your local settings.

Step 8 - Setup permissions

Finally you want to setup permission to define which users / groups will have access to which parts of the app. Check out permissions for details.

Congratulations you are now ready to use Member Audit!

Updating

To update your existing installation of Member Audit first enable your virtual environment.

Then run the following commands from your AA project directory (the one that contains manage.py).

pip install -U aa-memberaudit

python manage.py migrate

python manage.py collectstatic

Finally restart your AA supervisor services.

Permissions

For this app there are two types of permissions:

Feature permissions give access to a feature
Scope permissions give access to scope

To define a role you will mostly need at least one permission from each type. For example for the recruiter role you will want finder_access, that gives access to the character finder tool, and view_shared_characters, so that the recruiter can see all shared characters.

The exception is the basic role, basic_access, that every user needs just to access the app. It does not require any additional scope roles, so a normal user just needs that role to be able to register his characters.

Permission list

Name	Description	Type
`basic_access`	Can access this app and register and view own characters	Feature
`share_characters`	Can share his characters. Note that others need the `view_shared_characters` permission to see them.	Feature
`finder_access`	Can access character finder features for accessing characters from others	Feature
`reports_access`	Can access reports features for seeing reports and analytics.	Feature
`characters_access`	Can access characters owned by others.	Feature
`exports_access`	Can access data exports. Warning: This permission gives access to all data from all characters and does not require any additional scope permissions.	Feature
`view_shared_characters`	All characters, which have been marked as shared & can access these characters	Feature & Scope
`view_same_corporation`	All mains - incl. their alts - of the same corporations the user’s main belongs to	Scope
`view_same_alliance`	All mains - incl. their alts - of the same alliances the user’s main belongs to	Scope
`view_everything`	All characters registered with Member Audit	Scope
`notified_on_character_removal`	Get a notification when someone drops a character.	Feature
`view_skill_sets`	Can view skill set for a character.	Scope

Hint

All above mentioned permissions can be found under the category “memberaudit | general”.

Example Roles

To further illustrate how the permission system works, see the following list showing which permissions are needed to define common roles:

Role	Description	Permissions
Normal user	Can use this app and register and access own characters	`basic_access`
Recruiter	Can access shared characters	`basic_access` `finder_access` `view_shared_characters`
Corporation Leadership	Can access reports for his corporation members (but can not access the characters)	`basic_access` `reports_access` `view_same_corporation`
Corp Leadership & Recruiter	Can access shared characters	`basic_access` `finder_access` `view_shared_characters` `reports_access` `view_same_corporation`
Alliance Auditor	Can search for and access all characters of his alliance	`basic_access` `finder_access` `characters_access` `view_same_alliance` `notified_on_character_removal`

Note

Naturally, superusers will have access to everything, without requiring permissions to be assigned.

Management Commands

The following management commands are available to perform administrative tasks:

Hint

Run any command with --help to see all options

memberaudit_data_export

Export data into a CSV file for use with external applications. Will include data from all characters in the database.

Currently supports the wallet journal only.

memberaudit_load_eve

Pre-loads data required for this app from ESI to improve app performance.

memberaudit_reset_characters

This command deletes all locally stored character data, but maintains character skeletons, so they can be reloaded again from ESI.

Warning

Make sure to stop all supervisors before using this command.

memberaudit_stats

This command returns current statistics:

Object counts for all character sections (e.g. asset items)
Current Member Audit settings
Current stale minutes configuration

memberaudit_update_characters

Start updating characters from ESI. You can choose to update all or specific sections.

Configuring celery workers

Celery workers can be setup in different ways. The two main flavors are thread based and process based. Thread based workers perform better when tasks are primarily I/O bound. And process based workers perform better with tasks that are primarily CPU bound.

AA tasks are primarily I/O bound (most tasks are fetching data from ESI and/or updating the database). That is why thread based setups for AA will usually outperform process based setups by a wide margins in terms of task throughput per CPU second. Some popular community apps, which make heavy use of tasks like Member Audit will not even work properly with a process based setup on most systems.

Step 1 - Configure workers

Hint

The celery worker configuration can be found in the supervisor.conf file. This file is usually located in your AA root folder, see same folder where you also find the manage.py file. When you followed the default installation guide the path is: /home/allianceserver/myauth/supervisor.conf. This file contains the configuration for several programs. The celery worker configuration can be found under [program:worker].

The first step is to change the configuration of your celery worker. Here is an example configuration:

[program:worker]
command=/home/allianceserver/venv/auth/bin/celery -A myauth worker -P threads -c 10 -l INFO -n %(program_name)s_%(process_num)02d
directory=/home/allianceserver/myauth
user=allianceserver
numprocs=2
process_name=%(program_name)s_%(process_num)02d
stdout_logfile=/home/allianceserver/myauth/log/worker.log
stderr_logfile=/home/allianceserver/myauth/log/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
killasgroup=true
priority=998

This example configuration will spawn up to 20 concurrent worker threads running on two processes with up to 10 threads each.

Depending on your system you may want to adjust the configuration:

numprocs: Number of processes. Must not be higher then the number of available CPU cores.
-c: Number of threads per process. Can be decreases/increased to adjust how much load the celery workers will utilize.

Step 2 - Setup memmon

In addition we strongly recommend to setup measures to protect your celery workers from consuming too much memory.

This is not a built in feature and requires the 3rd party extension superlance, which includes a set of plugin utilities for supervisor. The one that watches memory consumption is memmon.

To setup install superlance into your venv with:

pip install superlance

You can then add memmon to your supervisor.conf. Here is an example setup with a worker:

[eventlistener:memmon]
command=/home/allianceserver/venv/auth/bin/memmon -p worker_00=512MB -p worker_01=512MB
directory=/home/allianceserver/myauth
events=TICK_60
stderr_logfile=/home/allianceserver/myauth/log/memmon.log

This setup will check the memory consumption of two “worker” programs every 60 secs and automatically restart it if is goes above 512 MB. Note that it will use the stop signal configured in supervisor, which is TERM by default. TERM will cause a “warm shutdown” of your worker, so all currently running tasks are completed before the restart.

This configuration is for exactly two worker processes. If you have more or less, please adjust accordingly.

The 512 MB is just an example and should be adjusted to fit your system configuration. Note that the total memory consumption is the sum of all thresholds per worker process, e.g. here we allow a maximum of 1 GB.

Step 4 - Update connection pool for ESI

If you have more than 10 worker threads you also need to increase the connection pool for django-esi accordingly. You can do this by adding the following line to your local settings (e.g. for a total of 20 worker threads):

ESI_CONNECTION_POOL_MAXSIZE = 20

Step 5 - Enable changes

Finally, to enable all changes your will need to reload your supervisor.

Let’s first shutdown your AA programs normally.

sudo supervisorctl stop myauth:

Then we reload the supervisor configuration, which will also restart all programs:

sudo supervisorctl reload

To verify this worked well, you can check the status of your AA programs with:

sudo supervisorctl status myauth:

All programs should be shown with RUNNING.

In case you encounter any issues you can find the relevant logs here:

supervisor: sudo journalctl -u supervisor
worker: worker.log in myauth/log
memmon: memmon.log in myauth/log
gunicorn: gunicorn.log in myauth/log

Configuring update frequency

In general most character sections are updated every 4 hours, with some heavy sections like assets updated every 6 hours, and some sections where more current information is needed, like location, updated every hour.

Hint

You can see the current stale minutes configuration for all sections by running the memberaudit_stats command.

If you have different requirements, you can change the update frequency for each and every section. E.g. decreasing the update frequency will help with reducing the overall task load.

How often a section is updated is determined by their “stale minutes” value. That value defines how long it takes for a section to “turn” stale after it’s last successful update was completed. Stale sections will be updated with the next run of the periodic update task, which normally runs every 15 minutes.

To set a custom stale minutes values for a sections add the name of the section and the new value to the MEMBERAUDIT_SECTION_STALE_MINUTES_CONFIG setting. That setting is a dict and expects the name of the section as key.

For example let’s set the stale minutes for the asset section to 12 hours:

MEMBERAUDIT_SECTION_STALE_MINUTES_CONFIG = {"assets": 720}

Multiple entries need to be separated with a comma. For example if we also want to set the stale minutes for the contacts section to 6 hours we get:

MEMBERAUDIT_SECTION_STALE_MINUTES_CONFIG = {"assets": 720, "contacts": 360}

Note

Sections, which are not defined here will use their default.

Hint

You can see the current stale minutes configuration for all sections by running the memberaudit_stats command.

Settings

Here is a list of available settings for this app. They can be configured by adding them to your AA settings file (local.py).

Note

All settings are optional and the app will use the documented default settings if they are not used.

Settings for Member Audit.

MEMBERAUDIT_APP_NAME: Name of this app as shown in the Auth sidebar and page titles.

MEMBERAUDIT_BULK_METHODS_BATCH_SIZE = 500: Technical parameter defining the maximum number of objects processed per run of Django batch methods, e.g. bulk_create and bulk_update.

MEMBERAUDIT_DATA_EXPORT_MIN_UPDATE_AGE = 60: Minimum age of existing export file before next update can be started in minutes.

MEMBERAUDIT_DATA_RETENTION_LIMIT = 360: Maximum number of days to keep historical data for mails, contracts and wallets. Minimum is 7 day.

MEMBERAUDIT_FEATURE_ROLES_ENABLED = False: Feature flag to enable or disable the corporation roles feature.

MEMBERAUDIT_LOCATION_STALE_HOURS = 24: Hours after a existing location (e.g. structure) becomes stale and gets updated e.g. for name changes of structures.

MEMBERAUDIT_MAX_MAILS = 250: Maximum amount of mails fetched from ESI for each character.

MEMBERAUDIT_NOTIFY_TOKEN_ERRORS = True: When enabled will automatically notify users when their character has a token error. But only once per character until the character is re-registered or this notification is reset manually by admins.

MEMBERAUDIT_SECTION_STALE_MINUTES_CONFIG = {}

Custom configuration of stale minutes for each section, which will override the respective defaults.

Tip: Please run the command memberaudit_stats to see the currently effective configuration.

MEMBERAUDIT_SECTION_STALE_MINUTES_GLOBAL_DEFAULT = 240

Default time in minutes after the last successful update at which a section is considered stale and therefore needs to be updated. All sections, which do not have a specific default value and are not configured differently will use this value.

Tip: Please run the command memberaudit_stats to see the currently effective configuration.

MEMBERAUDIT_STORE_ESI_DATA_CHARACTERS = []: List character IDs to filter storing debug data for. An empty list means all characters.

MEMBERAUDIT_STORE_ESI_DATA_ENABLED = False

Set to true to store incoming data from the ESI API to disk for debugging.

The data will be stored in JSON files under: ~/myauth/temp/memberaudit_log.

Warning: Storing this data can quickly occupy a lot of disk space. We strongly recommend to also define filters for sections and/or characters to limit what data is stored.

MEMBERAUDIT_STORE_ESI_DATA_SECTIONS = []: List sections to filter storing debug data for. An empty list means all sections.

MEMBERAUDIT_TASKS_HIGH_PRIORITY = 3: Priority for high priority tasks, e.g. user requests an action.

MEMBERAUDIT_TASKS_LOW_PRIORITY = 7: Priority for low priority tasks, e.g. updating characters.

MEMBERAUDIT_TASKS_MAX_ASSETS_PER_PASS = 2500: Technical parameter defining the maximum number of asset items processed in each pass when updating character assets. A higher value reduces duration, but also increases task queue congestion.

MEMBERAUDIT_TASKS_NORMAL_PRIORITY = 5: Priority for normal tasks, e.g. updating characters.

MEMBERAUDIT_TASKS_TIME_LIMIT = 7200: Global timeout for tasks in seconds to reduce task accumulation during outages.

Uninstalling

Here is how you can uninstall Member Audit cleanly from your system.

Step 1 - Shut down supervisors

First you should shut down all AA services.

sudo supervisorctl stop myauth:

Step 2 - Remove tables from the database

Next we will remove Member Audit’s table from the database.

Important: You will loose all Member Audit related data. You can not undo this step, unless you have a backup.

First, fetch the SQL script:

wget https://gitlab.com/ErikKalkoken/aa-memberaudit/-/blob/master/memberaudit/tools/drop_tables.sql

Next, log into mysql, switch to your database and then and run the script to drop all tables:

sudo mysql -u root
use alliance_auth;
source drop_tables.sql;

Check that all tables have been deleted.

show tables;

Then, switch back to bash and delete the sql script (it has served it’s purpose):

exit;
rm drop_tables.sql

Finally, we inform Django that the Member Audit tables have been deleted:

python manage.py migrate zero --fake

Step 3 - Remove app from Django

After the migration to zero is completed, we can now remove the Django app in your local settings.

For that remove the "memberaudit" in INSTALLED_APPS and any other setting about Member Audit, e.g. definition for tasks.

Step 4 Uninstall Python package

Finally, we can uninstall the Python package:

pip uninstall aa-memberaudit

Step 5 - Restart AA services

Member Audit is now fully uninstalled and you can restart your AA services:

sudo supervisorctl start myauth: