system:mailman3
Table of Contents
Mailman 3
Debian: 10 Buster LTS EOL 2024-06-30 First stable to support mailman3 last to support mailman 2 balug-sf-lug-v2.balug.org (balug.org) mailman3: run mailman command with env LANG=en_US.UTF-8, as user list (by default on Debian), probably umask 022 $ expand -t 4 < ~root/bin/Mailman #!/bin/sh umask 022 && unset LC_ALL && LANG=en_US.UTF-8 export LANG && case "$(id -n -u)" in list) umask 022 && exec mailman "$@" ;; *) umask 022 && exec sudo -u list \ mailman "$@" ;; esac $ # /usr/share/mailman3-web/manage.py # use no options/arguments for help # /usr/share/mailman3-web/manage.py createsuperuser # /usr/share/mailman3-web/manage.py changepassword login_name_of_user databases: # sqlite3 /var/lib/mailman3/data/mailman.db # sqlite3 /var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db # or /var/lib/mailman3/web/mailman3web.db backup, e.g.: sqlite3 /var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db ".backup '/var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.BACKUP.db'" created list: balug-test3@lists.balug.org https://lists.balug.org/mailman3/ https://lists.balug.org/mailman3/admin/ Y posting: X-Archive: no if header is set to value no (regardless of case), doesn't archive Y posting: X-No-Archive: if is header present, doesn't archive Y posting: if X-Archive and X-No-Archive conflict, or multiple conflicting X-Archive present, appears to not archive unless only X-Archive header(s) are present and the first does not have value of no (case insensitive) see also: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/docs/archives.html Y (Almost?) Everything you wanted to know about Mailman 3 imports/archives (but were afraid to ask?): Various things happen between import and "export" ("Download" of archives). import appends, existing archive items aren't dropped Note also that there may be a raw(er) form in the database. some analysis (notably mtime, etc.) implies that the imported data is stored in: /var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db (sqlite3/postorious database: # sqlite3 /var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db) somewhere around, e.g. here: sqlite> SELECT sql FROM sqlite_master WHERE tbl_name = 'hyperkitty_email' AND type = 'table'; CREATE TABLE "hyperkitty_email" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "message_id" varchar(255) NOT NULL, "message_id_hash" varchar(255) NOT NULL, "subject" varchar(512) NOT NULL, "content" text NOT NULL, "date" datetime NOT NULL, "timezone" smallint NOT NULL, "in_reply_to" varchar(255) NULL, "archived_date" datetime NOT NULL, "thread_depth" integer NOT NULL, "thread_order" integer NULL, "sender_id" varchar(255) NOT NULL REFERENCES "hyperkitty_sender" ("address"), "thread_id" integer NOT NULL REFERENCES "hyperkitty_thread" ("id"), "sender_name" varchar(255) NULL, "mailinglist_id" integer NOT NULL REFERENCES "hyperkitty_mailinglist" ("id"), "parent_id" integer NULL REFERENCES "hyperkitty_email" ("id")) sqlite> select * from hyperkitty_email WHERE id = 54; 54|1720513097.9ExYtTWvb7bqxG6Y@balug-sf-lug-v2.balug.org|PTLURYJSWOCAWEZ5WDK5PBHHCQ3R66HW|test - ignore - archive import testing|body 1720513097 postmaster@example.com @example.com last line of body |2024-07-09 08:18:17|0||2024-07-09 08:18:17|0|0|test-trnzipzntiaotvhgjmee@balug.org|49|test-trnzipzntiaotvhgjmee@balug.org|2| E.g. some inconsistent email address obfuscation between GUI display and Download (export), hints that there's lower level raw(er) in there somewhere - maybe that can also be usefully accessed? So, between import and Download, at least some headers and bodies are altered: To: is unconditionally set to only the list CC: is unconditionally stripped. Various headers may be added/dropped/altered, reordered plain email with no MIME may get converted to Content-Type: multipart/mixed, even if it only then has the one part X-No-Arcive: unconditionally stripped (and then handled as if not present) X-Archive: unconditionally stripped (and then handled as if not present) Other conversions/changes, etc. may occur. In any case does end up imported into archive Y Do users have any ability to access (and backup) entire list's archive as (e.g.) .mbox format file (like Mailman 2 can do)? On archive page there's a Download drop-down to download entire archive, no auth needed for public archives, URL also works (e.g. with curl), e.g.: https://lists.balug.org/mailman3/hyperkitty/list/balug-test3@lists.balug.org/export/balug-test3@lists.balug.org.mbox.gz N Are such archives in original unobfuscated (e.g. email addresses) format? Although headers aren't obfuscated, bodies are subject to some @ --> (a) email obfuscation, Web GUI display of archive does similar, but not even same matched set! E.g. body munging: displays on web GUI: obfuscated: postmaster(a)[127.0.0.1] postmaster(a)example.com unobfuscated: <postmaster@localhost> @example.com @localhost postmaster@localhost And in what it allows to be downloaded via web, ugh, slightly worse, but not even the same set!: obfuscated: <postmaster(a)localhost> postmaster(a)example.com postmaster(a)localhost unobfuscated: @example.com @localhost postmaster@[127.0.0.1] And same results whether posted via SMTP/mail, or GUI. Y imports appear to apply same @ --> (a) email address obfuscation (at least compared to GUI display and Download) ? is actual data in database unadulterated? Y how do we remove posting or archive? Looks like simplest way is from admin web interface, e.g.: "Delete this message" link, e.g.: https://lists.balug.org/mailman3/hyperkitty/list/balug-test3@lists.balug.org/message/DIQNAQZNUJSVN7CRA2NZ2JLQ7UCFFSTM/delete Note that that's distinct from "Delete this thread" links. lock (to avoid race conditions) can delete from archive (e.g. in database), e.g.: sqlite> DELETE from hyperkitty_email WHERE id = 55; that deletes an individual email - may or may not delete associated data (e.g. attachments). ? Note that it doesn't immediately disappear all related information on web pages, such as "RECENTLY ACTIVE DISCUSSIONS", though the links to the email message will no longer work, and seems after a while these clean themselves up (likely some periodic scheduled job or the like) ? Something like this may also be useful (unverified/untested): in a django management shell, you could try something like import datetime from hyperkitty.models import Email for msg in Email.objects.filter(mailinglist_id=X): if msg.date < datetime.datetime(Y, M, D, tzinfo=datetime.timezone.utc): msg.delete() where X is the id of the mailing list you want to prune, or if you wanted to do all lists, just use Email.objects.all() instead of filtering on mailinglist_id. And, Y, M and D are integer Year, Month and Day for the cutoff. Note that something like from hyperkitty.models import MailingList for ml in MailingList.objects.all(): print('{}: {}'.format(ml.name, ml.id)) will print all the list names and their ids. ? How do we rename a list (including its archive)? Doesn't appear to be well covered, see: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/Q3YHKZKUALBWIESNOQLRBFRNJ6F3O77U/ Let's attempt - rename of balug-test4@lists.balug.org to balug-test5@lists.balug.org Y sanity check balug-test4@lists.balug.org still working Y allow postings but prevent additional archiving, this, e.g.: https://lists.balug.org/mailman3/postorius/lists/balug-test2.lists.balug.org/settings/archiving change setting from: Archive policy: Public archives Active archivers: (checkbox checked) hyperkitty to: Archive policy: Public archives Active archivers: (checkbox unchecked) hyperkitty suffices to prevent new postings from being archived (and if reactivated, doesn't cause retroactive archiving) ? how does that interact with, e.g.: https://lists.balug.org/mailman3/admin/hyperkitty/mailinglist/4/change/ change settings from: Archive policy: public to: Archive policy: never ? will that prevent additional archiving while allowing posting. ? Are both of those needed, or will one suffice? ? What exactly is the difference? Y import adds (doesn't clobber existing) ? localpart+tag@ email addresses supported? N by MTA in current configuration? Doesn't look super easy to add at current, will probably defer and reevaluate later Y by Mailman 3 Looks like Mailman 3 itself probably handles it fine and doesn't care (treats it as local part, including part after +) Y how do we export roster of members? E.g.: # (for list in $(Mailman lists -a -q); do echo "$list:"; Mailman members --role any "$list"; done) or via URLs, e.g. https://lists.balug.org/mailman3/postorius/lists/balug-test3.lists.balug.org/csv_view/ Y don't reject empty body (e.g. mailman commands don't require body to be present) DMARC - tested (mostly with @yahoo.com), seems these are best compromise settings: DMARC mitigation action: Wrap the message in an outer message From: the list. DMARC Mitigate unconditionally: No DMARC rejection notice: DMARC wrapped message text: select documentation: https://docs.mailman3.org/en/latest/install/virtualenv.html https://docs.mailman3.org/projects/hyperkitty/en/latest/install.html https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/postorius.html https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/docs/archives.html https://docs.mailman3.org/projects/mailmanclient/en/latest/ https://docs.mailman3.org/projects/postorius/en/latest/ https://wiki.archlinux.org/title/Postorius https://www.balug.org/doc/ https://www.balug.org/doc/mailman3-doc/ https://www.balug.org/doc/mailman3-full/ https://www.balug.org/doc/mailman3-web/ https://www.balug.org/doc/mailman3/html/README.html#table-of-contents https://www.balug.org/doc/mailman3/html/src/mailman/model/docs/mlist-addresses.html https://www.balug.org/doc/python3-django-hyperkitty/html/ https://www.balug.org/doc/python3-django-postorius/html/ https://www.balug.org/doc/python3-django-postorius/html/deployment.html Packages: apache2 exim4-daemon-heavy lynx mailman3 mailman3-doc mailman3-full mailman3-web python-django-common python3-django python3-django-allauth python3-django-appconf python3-django-compressor python3-django-extensions python3-django-filters python3-django-gravatar2 python3-django-guardian python3-django-haystack python3-django-hyperkitty python3-django-mailman3 python3-django-picklefield python3-django-postorius python3-django-q python3-djangorestframework python3-mailman-hyperkitty python3-mailmanclient
Mailman 2 --> 3 migration
Mailman 2 --> 3 migration checklist/status
list | test | admin | talk | announce |
---|---|---|---|---|
lock old | y | y | y | y |
handle pending | y | y | y | y |
create/rename target | y | y | y | y |
migrate list configuration | y | y | y | y |
migrate list archive | y | y | y | y |
index archive | y | y | y | y |
archive *.pck files | y | y | y | y |
archive relevant bits on informational pages | y | y | y | y |
remove Mailman 2 list do not remove archives | y | y | y | y |
update http[s]://lists.balug.org/ | y | y | y | y |
update links on https://www.balug.org/#Lists | y | y | y | y |
Mailman 2 –> Mailman 3 http[s] redirects | ir | ir | ir | ir |
update Mailman 2 archive locations to not link to obsolete information | n | n | n | n |
customize templates | i | i | i | i |
update monthly stats reporting | - | y |
notes/keys:
- n - No / Not done
- i - in progress or incomplete
- ir - added redirects for main list page + each lists's main page
- y - Yes / done
- - N/A Not Applicable
- lock old: comment out of /etc/aliases
- pending: handle pending as warranted (e.g. held pending moderator approval)
Mailman 3: fix gravatar.com information leakage | y |
fix what I broke in attempting to fix the immediately above | y |
Mailman 3: check for any additional information leaks | n |
stop Mailman 2 services # systemctl stop mailman.service | y |
disable Mailman 2 services # systemctl disable mailman.service | y |
disable the remaining Mailman 2 aliases (mailman & mailman-*) /etc/aliases | y |
Apache 2 configuration - remove no longer relevant Mailman 2 portions | y |
fix what I broke in the immediately above, notably links so static web archive content will still work | y |
disable nntp runner | y |
remove but do not purge mailman2 and no longer needed reverse dependencies packages | y |
fix minor HyperKitty archive bugs present on Debian 11 Bullseye | n |
fix HyperKitty failure (configuration?) from Debian 11 Bullseye –> Debian 12 Bookworm upgrade | y |
balug-announc@lists.balug.org set users with explicit moderation of hold (imported from Mailman 2) to List default (discard) | y |
notes/keys:
- n - No / Not done
- i - in progress or incomplete
- y - Yes / done
- - N/A Not Applicable
Various notes on migration, etc. See also the earlier: https://www.wiki.balug.org/wiki/doku.php?id=balug:mail_and_lists Debian: 10 Buster last to support mailman 2, also (first to?) supports mailman3 LTS EOL 2024-06-30 balug-sf-lug-v2.balug.org (balug.org) Location of list configuration in Mailman 2 /var/lib/mailman/lists/*/config.pck Location of list archives in Mailman 2 /var/lib/mailman/archives/private/*.mbox/*.mbox Location of the bin/ commands in Mailman 2 /usr/lib/mailman/bin run commands as user list (on Debian): Y How do we do "locking" or list suspensions/shutdowns to avoid race conditions? removing Mailman 2 aliases from /etc/aliases (and running newaliases) prevents posts, etc. to corresponding Mailman 2 lists, etc. Create the target list on Mailman 3, e.g.: foo-list@example.com, via either: https://lists.balug.org/mailman3/postorius/lists/new/ # or: Mailman create foo-list@example.com Y if same list name exists on mailman2 and mailman3, and post is sent to list, which does it use?: Mailman 2 if still in /etc/aliases, else Mailman 3 Migrate the list configuration from Mailman 2 to Mailman 3 by running the following command (use the appropriate file for .pck and .mbox in the below) # Mailman import21 foo-list@example.com /var/lib/mailman/lists/*/config.pck Ugh. mailman import12 may also fail, e.g.: File "/usr/bin/mailman", line 11, in <module> load_entry_point('mailman==3.2.1', 'console_scripts', 'mailman')() File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/lib/python3/dist-packages/mailman/commands/cli_import.py", line 64, in import21 pickle_file, encoding='utf-8', errors='ignore') ModuleNotFoundError: No module named 'Mailman' see also, e.g.: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/JEPMB3HW4FI57EUMOST4L7BD2ILIIS3P/ and this seemed sufficien to work around the issue: executed as id list: echo 'bounce_info = {}' > reset_bounceinfo.py config_list -i reset_bounceinfo.py balug-talk Y import21 does mostly bring over the following per-member options from Mailman2 to Mailman3: Mailman 2 <--> Mailman 3 one-to-one mappings where present: Mailman 2 Mailman 3 delivery enable/disable Delivery status Enabled/Disabled (preserved) digest no/yes Regular / Plain Text Digests / Mime Digests (preserved, see plain below) mod Hold for moderation (preserved) user_options (bit mapped) 2 not metoo Receive own postings No/Yes (preserved) 4 ack Acknowledge posts Yes/No (preserved) 8 plain (if digest mode enabled, sets type above - preserved) 16 hide Hide address Yes/No (preserved) 256 no dupes Receive list copies No/Yes (preserved) ? note that some attributes/settings may not be imported in some circumstances, these seem to be the cases: ? if nomail is set, no options may end up set (e.g. none of enabled/disabled, yes/no set) ? if no options are set, may end up with just Delivery status Enabled, Delivery mode Regular, and no other options set at all (similar to above) ? if digest is set and no other options set, may end up with just Delivery status Enabled, Delivery mode Mime Diegest, and no other options set at all (similar to above) Migrate the list archives from Mailman 2 to Mailman 3 by running the following command: Note that the import and (re)indexing need to be run as www-data on Debian $ (unset LC_ALL && LANG=en_US.UTF-8 export LANG && /usr/share/mailman3-web/manage.py hyperkitty_import -l foo-list@example.com --since 1968 /var/lib/mailman/archives/private/*.mbox/*.mbox) Notes: on mbox import format as processed by hyperkitty_import: The leading From (envelope from, not header From:) can be entirely empty - just "From " and it accepts that (apparently doesn't use it other than to determine where that mail item starts) So, it apparently just used Date: field for time and timezone information, and From: header (or possibly similar) to identify sender email and name. Also note on import and database - though it does save raw unaltered body, only some header data is saved and even some/much of that isn't saved in original raw form. Also, database can have line(s) that start with "From " in body (content), but they can't be imported that way (as in mbox format, that would mark start of a mail item) rebuild the index for this list: $ (unset LC_ALL && LANG=en_US.UTF-8 export LANG && /usr/share/mailman3-web/manage.py update_index_one_list foo-list@example.com) may need to do, e.g.: $ (unset LC_ALL && LANG=en_US.UTF-8 export LANG && /usr/share/mailman3-web/manage.py update_index -s 1970-01-01T00:00:00) Delete Mailman 2 list: $ /usr/lib/mailman/bin/rmlist foo-list Y Old Mailman 2 archives remain after Mailman 2 list is removed with rmlist, unless the -a option is given (in which archives are also removed). ? other bits, e.g. informational page(s) we should update along the way? Y prepare changes of redirection/content on http[s]://lists.balug.org/ for migration & legacy, etc. ? prepare replacement reporting, e.g. monthly stats. Existing Mailman 2 have: /etc/cron.d/local-lists /var/local/sbin/lists_monthly2 to be: ? /var/local/sbin/lists_monthly3 Note: import will append, not replace At least partially lock out old (remove aliases from /etc/aliases and run newaliases) ? backward compatibility with old links, e.g. archives, do we do as noted/suggested here: https://docs.mailman3.org/en/latest/migration.html and here: https://mail.python.org/pipermail/security-sig/ the files to be customized would be the *.html files under /var/lib/mailman/archives/private/list_name/ ? update links on, e.g. https://www.balug.org/#Lists ? Announce ? Talk ? Admin ? Test ? Is it at all reasonably feasible to mass export/import user preferences, passwords, etc.? There isn't 1-to-1 mapping between Mailman 2 and Mailman 3 on user logins, preferences, etc. some key differences: Mailman 2 logins are per-list Mailman 3 logins aren't required and if present are per user Mailman 3 can have multiple distinct emails associated with same user account, Mailman 2 can't do that ? What bits do have one-to-one mappings on per-list basis? Mailman 2 has (approximately?) this: Mailman 2 <--> Mailman 3 one-to-one mappings where present: Mailman 2 Mailman 3 delivery enable/disable delivery enable/disable digest no/yes digest no/plain/MIME/summary (see also Mailman 2 plain/MIME) user_options (bit mapped) 2 not metoo Receive own postings no/yes 4 ack Acknowledge posts yes/no 8 plain No equivalent except if digest, can be set to plain or MIME 16 hide Hide address yes/no 256 no dupes Receive list copies no/yes Other bits we'll want to map as/where feasible (not per list): usernames passwords example files: Mailman 2: /var/lib/mailman/archives/private/balug-test2/pipermail.pck /var/lib/mailman/lists/balug-test2/config.pck $ /usr/lib/mailman/bin/dumpdb --pickle /var/lib/mailman/lists/balug-test2/config.pck provides users (members), their passwords, and options. Before migrating list archive, check, clean, etc., see: https://docs.mailman3.org/en/latest/migration.html#id2 Mailman 3 gravatar information leakage HyperKitty archive pages are (at least by default) served up with links to gravatar.com (looks like) they leak information to gravatar.com - looks like each link is (probably) a hash of the user/member's email address Earlier versions of Mailman 3 don't have a "fix" for that. Later versions allow it to be disabled in configuration: https://docs.mailman3.org/projects/mailman-web/en/latest/settings.html#mailman_web.settings.mailman.HYPERKITTY_ENABLE_GRAVATAR Looks like that went in with HyperKitty 1.3.4: https://docs.mailman3.org/projects/hyperkitty/en/latest/news.html#news-1-3-4 Debian 10.13 --> 11.10 gave us upgrade python3-django-hyperkitty:all 1.2.2-1+deb10u1 --> 1.3.4-4 So, should now be able to fix the gravatar issue. Minor HyperKitty archive bugs on Debian 11 Bullseye From archive page, e.g.: https://lists.balug.org/mailman3/hyperkitty/list/balug-test@lists.balug.org/ clicking the Threads by month dropdown doesn't do anything. After clicking All Threads which goes to, e.g. https://lists.balug.org/mailman3/hyperkitty/list/balug-test@lists.balug.org/latest once there, clicking on a year shows the months under year, but those immediately collapse before one can click on any of those months, this seems to be the case regardless of browswer. These issues weren't present on Debian Buster 10, and hopefully are fixed by Debian 12 Bookworm balug-announc@lists.balug.org set users with explicit moderation of hold (imported from Mailman 2) to List default (discard) as user list: (umask 022 && unset LC_ALL && LANG=en_US.UTF-8 export LANG && sqlite3 /var/lib/mailman3/data/mailman.db "UPDATE member SET moderation_action = NULL WHERE list_id = 'balug-announce.lists.balug.org' and moderation_action = 0;") templates: from mailman3 package: /usr/lib/python3/dist-packages/mailman/templates/en/*.txt VM: debian10mailman23 select documentation: https://docs.debops.org/en/stable-2.1/ansible/roles/mailman/mailman2-migration.html https://docs.mailman3.org/en/latest/faq-migration.html https://docs.mailman3.org/en/latest/migration.html https://github.com/jmuhlich/mailman-archive-migration Packages: mailman
system/mailman3.txt · Last modified: 2024-09-07T18:59:41+0000 by michael_paoli_admin