2024-02-06

What does private mean at package level in Scala 3?


TL; DR:

  • private declarations at a top-level scope of a package in Scala 3 are equivalent to a private[pkg] in other contexts.
  • They are accessible to everything within the package and its subpackages, but nothing else.

In Scala 2, to place a declaration at the "package" level, one would define a "package object":

package top

package object pkg {
  private val Hush = 0
  val Loud = Int.MaxValue
}

Given this

  • one might refer to Loud from anywhere with fully-qualified name top.pkg.Loud
  • import top.pkg._ would pick it up
  • inside the package top.pkg one coul refer to it simply as Loud

So far, so intuitive.

In Scala 2, the semantics of private val Loud was also intuitive. A package object is just an object. A private member of an object is only visible within that object's scope. While the Scala compiler does some magic to make nonprivate declarations more broadly visible, access to private members of the package object was restricted to the object in the ordinary way.

But Scala 3 introduces "naked" top-level declarations, which I find I use constantly.

So the declarations above might translate to:

package top.pkg

private val Hush = 0
val Loud = Int.MaxValue

There is no object scope! So what does private even mean in this context.

I could imagine four possibilities:

  1. private to a virtual object scope constituted of all top-level declaraions
  2. private to the top-level of the current compilation unit (i.e. file)
  3. private to the current compilation unit (including nested scopes)
  4. private to the package as a whole, i.e. the same as private[pkg]

Playing around, it looks like #4 is the winner.

A private top-level declaration seems visible to any code in the package, even if defined in other files or directories. It is visible from anywhere in the pkg or subpackages of pkg.

So now I know! And so do you!

2024-02-04

Style-by-mail in feedletter


If, my most patient dear reader, you followed the feedletter tutorial in the previous post, you saw that we styled feedletter newsletters by starting up a development webserver, which would serve up HTML for an example newsletter.

This is still the best to get started styling your newsletters, because you can iteratively edit your untemplate, then just hit refresh in your web browser to very quickly play with your style and layout.

However, there are differences between how web browsers and e-mail clients render HTML. Getting things to look great in your browser, both at wide and mobile-like narrow widths, is not enough to guarantee that things will look good as emails, on desktop or mobile e-mail clients.

So, feedletter v0.0.8 now supports styling by e-mail.

Instead of firing up a webserver to preview your newsletter, if you give feedletter-style commands --from and --to arguments, an example e-mail will be sent. So you can now accurately preview and tweak exactly how newsletters will look in the mail clients they are actually sent to.

To put that more specifically, just replace a command like...

$ ./feedletter-style compose-single --subscribable-name lgm --port 45612

with

$ ./feedletter-style compose-single --subscribable-name lgm --from feedletter@mchange.com --to swaldman@mchange.com

No development webserver will be spun up. Instead, a sample e-mail will be sent.

2024-01-29

Feedletter tutorial


I've been working for some time on a service to turn RSS feeds into e-mail newsletters, which I've called feedletter.

The service watches any number of RSS feeds, can host a variety of subscription types for each feed, including one e-mail per article, daily or weekly digests, compendia of every n posts, etc. It can also notify other services, like Mastodon, of new posts. It lets you define, for each feed, a notion of when an item is stable and finalized, and takes great care never to e-mail or notify the same item more than once.

Great minds think alike! After weeks of working on this, I discovered a similar project with the very same name.

Here I want to go through the process of setting up a feedletter instance, configuring it, tweaking or customizing the newsletter style, and running it.

You can host feedletter on any Linux/UNIX-ish server. For completeness, I'm going to set up a server from scratch, from a fresh Digital Ocean droplet. But of course you can run feedletter along side other services on an existing machine, and skip a lot of these steps. feedletter's main prerequisite is postgres, but we'll make use of nginx, certbot, systemd etc. as we go along.

Much of the code and config we develop will be memorialized in this github repo.

Let's go!

Table of contents

  1. Set up a server with a DNS name
  2. Download dependencies
  3. Create user feedletter
  4. Install feedletter
  5. Prepare the postgres database
  6. Set up feedletter-secrets.properties
  7. Get an https certificate
  8. Configure nginx to forward to the API
  9. Initialize the feedletter database
  10. Perform in-database configuration
  11. Add feeds to watch
  12. Define "subscribables" to feeds
  13. Enable feedletter as a systemd daemon
  14. Let users subscribe to your subscribables!
  15. Tweak the newsletter styles
  16. Advanced: Customize the content
  17. Conclusion

1. Set up a server with a DNS name

We launch a "droplet" from Digital Ocean. You can use whatever Linux flavor you like. We'll pick the latest Ubuntu.

Screenshot of Digital Ocean droplet setup

And we go ahead and give it a name.

Screenshot of FastMail DNS setup

2. Download dependencies

We login as root to our new droplet (however we've configured that), and download a bunch of stuff we'll need:

# apt install postgresql
# apt install openjdk-17-jre-headless
# apt install nginx
# apt install certbot
# apt install emacs

While we're at it, let's upgrade everything on the server and restart.

# apt upgrade
# shutdown -r now

3. Create user feedletter

We'll create a passwordless user:

# adduser --disabled-password feedletter
info: Adding user `feedletter' ...
info: Selecting UID/GID from range 1000 to 59999 ...
info: Adding new group `feedletter' (1000) ...
info: Adding new user `feedletter' (1000) with group `feedletter (1000)' ...
info: Creating home directory `/home/feedletter' ...
info: Copying files from `/etc/skel' ...
Changing the user information for feedletter
Enter the new value, or press ENTER for the default
	Full Name []: 
	Room Number []: 
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] Y
info: Adding new user `feedletter' to supplemental / extra groups `users' ...
info: Adding user `feedletter' to group `users' ...

4. Install feedletter

We'll become user feedletter, and download a local installation of the feedletter app:

# su - feedletter
feedletter@feedletter-play:~$ git clone https://github.com/swaldman/feedletter-install.git feedletter-local
Cloning into 'feedletter-local'...
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 46 (delta 19), reused 38 (delta 11), pack-reused 0
Receiving objects: 100% (46/46), 8.75 KiB | 2.19 MiB/s, done.
Resolving deltas: 100% (19/19), done.

The first time you run feedletter, it will take a couple of minutes to download its dependencies and compile stuff.

Although we can't meaningfully use it yet, let's give the feedletter applicaton a test run:

$ cd feedletter-local/
$ ./feedletter
...
...
Missing expected command (add-feed or alter-feed or daemon or db-dump or db-init or db-migrate or define-email-subscribable or define-mastodon-subscribable or drop-feed-and-subscribables or drop-subscribable or edit-subscribable or export-subscribers or list-config or list-feeds or list-items-excluded or list-subscribables or list-subscribers or list-untemplates or send-test-email or set-config or set-extra-params or set-untemplates or subscribe)!

Usage:
    feedletter [--secrets <propsfile>] add-feed
    feedletter [--secrets <propsfile>] alter-feed
    feedletter [--secrets <propsfile>] daemon
    feedletter [--secrets <propsfile>] db-dump
    feedletter [--secrets <propsfile>] db-init
    feedletter [--secrets <propsfile>] db-migrate
    feedletter [--secrets <propsfile>] define-email-subscribable
    feedletter [--secrets <propsfile>] define-mastodon-subscribable
    feedletter [--secrets <propsfile>] drop-feed-and-subscribables
    feedletter [--secrets <propsfile>] drop-subscribable
    feedletter [--secrets <propsfile>] edit-subscribable
    feedletter [--secrets <propsfile>] export-subscribers
    feedletter [--secrets <propsfile>] list-config
    feedletter [--secrets <propsfile>] list-feeds
    feedletter [--secrets <propsfile>] list-items-excluded
    feedletter [--secrets <propsfile>] list-subscribables
    feedletter [--secrets <propsfile>] list-subscribers
    feedletter [--secrets <propsfile>] list-untemplates
    feedletter [--secrets <propsfile>] send-test-email
    feedletter [--secrets <propsfile>] set-config
    feedletter [--secrets <propsfile>] set-extra-params
    feedletter [--secrets <propsfile>] set-untemplates
    feedletter [--secrets <propsfile>] subscribe

Manage e-mail subscriptions to and notifications from RSS feeds.

Options and flags:
    --help
        Display this help text.
    --secrets <propsfile>
        Path to properties file containing SMTP, postgres, c3p0, and other configuration details.

Environment Variables:
    FEEDLETTER_SECRETS=<path>
        Path to properties file containing SMTP, postgres, c3p0, and other configuration details.

Subcommands:
    add-feed
        Add a new feed from which mail or notifications may be generated.
    alter-feed
        Alter the timings of an already-defined feed.
    daemon
        Run daemon that watches feeds and sends notifications.
    db-dump
        Dump a backup of the database into a configured directory.
    db-init
        Initialize the database schema.
    db-migrate
        Migrate to the latest version of the database schema.
    define-email-subscribable
        Define a new email subscribable, a mailing lost to which users can subscribe.
    define-mastodon-subscribable
        Define a Mastodon subscribable, a source from which Mastodon feeds can receive automatic posts..
    drop-feed-and-subscribables
        Removes a feed, along with any subscribables defined upon it, from the service.
    drop-subscribable
        Removes a subscribable from the service.
    edit-subscribable
        Edit an already-defined subscribable.
    export-subscribers
        Dump subscriber information for a subscribable in CSV format.
    list-config
        List all configuration parameters.
    list-feeds
        List all feeds the application is watching.
    list-items-excluded
        List items excluded from generating notifications.
    list-subscribables
        List all subscribables.
    list-subscribers
        List all subscribers to a subscribable.
    list-untemplates
        List available untemplates.
    send-test-email
        Send a brief email to test your SMTP configuration.
    set-config
        Set configuration parameters.
    set-extra-params
        Add, update, or remove extra params you may define to affect rendering of notifications and messages.
    set-untemplates
        Update the untemplates used to render subscriptions.
    subscribe
        Subscribe to a subscribable.
1 targets failed
runMain subprocess failed

All good!

5. Prepare the postgres database

We'll exit back to root, become user postgres, and create a feedletter database that user feedletter can command:

$ exit
# su - postgres
$ createdb feedletter
$ createuser feedletter
$ psql
psql (15.5 (Ubuntu 15.5-0ubuntu0.23.10.1))
Type "help" for help.

postgres=# ALTER DATABASE feedletter OWNER TO feedletter;
ALTER DATABASE
postgres=# ALTER USER feedletter WITH PASSWORD 'not-actually-this';
ALTER ROLE
postgres=# \q

6. Set up feedletter-secrets.properties

feedletter expects passwords and some other configuration information in a "secrets" file, in Java properties file format. You can place this anywhere you want (feedletter will look for a command-line argument or an environment variable), but by default it looks for /etc/feedletter/feedletter-secrets.properties or /usr/etc/feedletter/feedletter-secrets.properties.

The file must belong to the user who will run feedletter, and it must have restrictive permission, readable and optionally writable by the user only.

The contents of the file will be something like this:

feedletter.secret.salt=Arbitrary secret string
mail.smtp.user=not-actually-this
mail.smtp.password=not-actually-this
mail.smtp.host=not-actually-this
mail.smtp.port=465
#mail.smtp.port=587
mail.smtp.debug=false
c3p0.jdbcUrl=jdbc:postgresql://localhost:5432/feedletter
c3p0.user=feedletter
c3p0.password=not-actually-this
c3p0.testConnectionOnCheckout=true

You’ll want to fill in your real SMTP authentication configuration. For information about this configuration, see mailutil.

You can configure database access via any and all c3p0 configuration properties.

So, let's do it! We exit from our last stint as user postgres first, then...

$ exit
# mkdir /etc/feedletter/
# emacs /etc/feedletter/feedletter-secrets.properties

Here we pause to edit the file, see the template above...

# chown -R feedletter:feedletter /etc/feedletter
# chmod go-wrx /etc/feedletter/feedletter-secrets.properties
# ls -l /etc/feedletter/
total 8
-rw------- 1 feedletter feedletter 370 Jan 25 18:59 feedletter-secrets.properties
-rw-r--r-- 1 feedletter feedletter 372 Jan 25 18:57 feedletter-secrets.properties~

Oops! emacs created a backup file with open permissions. Let's get rid of it so those secrets don't leak.

# rm /etc/feedletter/feedletter-secrets.properties~

7. Get an https certificate

We gave our server the name play.feedletter.org.

feedletter offers a web API to manage subscriptions. We'll want that to use https rather than http for privacy's sake.

Let's acquire a free Let's Encrypt certificate. I prefer to pause nginx to acquire and renew certificates, and use certbot's standalone server to verify control of the domain, rather than have certbot mess around with my nginx config.

So...

# systemctl stop nginx
# certbot certonly -d play.feedletter.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1 
Requesting a certificate for play.feedletter.org

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/play.feedletter.org/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/play.feedletter.org/privkey.pem
This certificate expires on 2024-04-24.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.
We were unable to subscribe you the EFF mailing list because your e-mail address appears to be invalid. You can try again later by visiting https://act.eff.org.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# systemctl start nginx

8. Configure nginx to forward to the API

By default, feedletter's API is bound to localhost on port 8024. If you need to, you can customize the web API port or interface, run ./feedletter set-config --help to see how. We'll stick with that default.

As root, we create and edit a file /etc/nginx/conf.d/play.feedletter.org.conf:

# emacs /etc/nginx/conf.d/play.feedletter.org.conf

It should look like this:

    # play.feedletter.org
    server {
        listen 80;
        listen [::]:80;
        server_name play.feedletter.org;
        return 301 https://play.feedletter.org$request_uri;
    }
    server {
        listen       443 ssl http2;
        listen       [::]:443 ssl http2;
        server_name  play.feedletter.org;
        ssl_certificate /etc/letsencrypt/live/play.feedletter.org/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/play.feedletter.org/privkey.pem;
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  10m;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
            proxy_pass http://127.0.0.1:8024/;
            proxy_set_header  X-Real-IP $remote_addr;
            proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header  Host $http_host;
        }
    }

Then we restart nginx:

root@feedletter-tutorial:/etc/nginx# systemctl restart nginx

9. Initialize the feedletter database

If we've set up the database and secrets file property, it should be as easy as

# su - feedletter
$ cd feedletter-local
$ ./feedletter db-init

10. Perform in-database configuration

Some of feedletter's config sits in the secrets file, but much lives in the application's database itself.

We can see feedletter's current (default) configuration simply by running

$ ./feedletter list-config
+-----------------------+-------------------------------------------------------+
¦ Configuration Key     ¦ Value                                                 ¦
+-----------------------+-------------------------------------------------------+
¦ ConfirmHours          ¦ 48                                                    ¦
¦ DumpDbDir             ¦ throws com.mchange.feedletter.db.ConfigurationMissing ¦
¦ MailBatchDelaySeconds ¦ 600                                                   ¦
¦ MailBatchSize         ¦ 100                                                   ¦
¦ MailMaxRetries        ¦ 5                                                     ¦
¦ MastodonMaxRetries    ¦ 10                                                    ¦
¦ TimeZone              ¦ Etc/UTC                                               ¦
¦ WebApiBasePath        ¦ /                                                     ¦
¦ WebApiHostName        ¦ localhost                                             ¦
¦ WebApiPort            ¦ None                                                  ¦
¦ WebApiProtocol        ¦ http                                                  ¦
¦ WebDaemonInterface    ¦ 127.0.0.1                                             ¦
¦ WebDaemonPort         ¦ 8024                                                  ¦
+-----------------------+-------------------------------------------------------+

The WebApi* keys are used to construct URLs that point back to the application (for creating, confirming, and removing subscriptions).

  • We won't want these to be localhost URLs, so we'll modify WebApiHostName
  • We'll want WebApiProtocol to be https rather than http
  • I'd prefer the timezone (used to format dates, and to decide the boundries of days and weeks for daily and weekly roundups) be America/New_York

Let's checkout the set-config command:

$ ./feedletter set-config --help
[49/49] runMain 
Usage: feedletter set-config [--confirm-hours <hours>] [--dump-db-dir <directory>] [--mail-batch-size <size>] [--mail-batch-delay-seconds <seconds>] [--mail-max-retries <times>] [--time-zone <zone>] [--web-daemon-interface <interface>] [--web-daemon-port <port>] [--web-api-protocol <http|https>] [--web-api-host-name <hostname>] [--web-api-base-path <path>] [--web-api-port <port>]

Set configuration parameters.

Options and flags:
    --help
        Display this help text.
    --confirm-hours <hours>
        Number of hours to await a user confiration before automatically unsubscribing.
    --dump-db-dir <directory>
        Directory in which to create dump files prior to db migrations.
    --mail-batch-size <size>
        Number of e-mails to send in each 'batch' (to avoid overwhelming the SMTP server).
    --mail-batch-delay-seconds <seconds>
        Time between batches of e-mails are to be sent.
    --mail-max-retries <times>
        Number of times e-mail sends (defined as successful submission to an SMTP service) will be attempted before giving up.
    --time-zone <zone>
        ID of the time zone which subscriptions based on time periods should use.
    --web-daemon-interface <interface>
        The local interface to which the web-api daemon should bind.
    --web-daemon-port <port>
        The local port to which the web-api daemon should bind.
    --web-api-protocol <http|https>
        The protocol (http or https) by which the web api is served.
    --web-api-host-name <hostname>
        The host from which the web api is served.
    --web-api-base-path <path>
        The URL base location upon which the web api is served (usually just '/').
    --web-api-port <port>
        The port from which the web api is served (usually blank, protocol determined).
1 targets failed
runMain subprocess failed

So, we can do all of this configuring in a single simple command:

$ ./feedletter set-config --web-api-protocol https --web-api-host-name play.feedletter.org --time-zone America/New_York
[49/49] runMain 
+-----------------------+-------------------------------------------------------+
¦ Configuration Key     ¦ Value                                                 ¦
+-----------------------+-------------------------------------------------------+
¦ ConfirmHours          ¦ 48                                                    ¦
¦ DumpDbDir             ¦ throws com.mchange.feedletter.db.ConfigurationMissing ¦
¦ MailBatchDelaySeconds ¦ 600                                                   ¦
¦ MailBatchSize         ¦ 100                                                   ¦
¦ MailMaxRetries        ¦ 5                                                     ¦
¦ MastodonMaxRetries    ¦ 10                                                    ¦
¦ TimeZone              ¦ America/New_York                                      ¦
¦ WebApiBasePath        ¦ /                                                     ¦
¦ WebApiHostName        ¦ play.feedletter.org                                   ¦
¦ WebApiPort            ¦ None                                                  ¦
¦ WebApiProtocol        ¦ https                                                 ¦
¦ WebDaemonInterface    ¦ 127.0.0.1                                             ¦
¦ WebDaemonPort         ¦ 8024                                                  ¦
+-----------------------+-------------------------------------------------------+

11. Add feeds to watch

Let's check out the add-feed command:

$ ./feedletter add-feed --help
[49/49] runMain 
Usage:
    feedletter add-feed --ping <feed-url>
    feedletter add-feed [--min-delay-minutes <minutes>] [--await-stabilization-minutes <minutes>] [--max-delay-minutes <minutes>] [--recheck-every-minutes <minutes>] <feed-url>

Add a new feed from which mail or notifications may be generated.

Options and flags:
    --help
        Display this help text.
    --ping
        Check feed as often as possible, notify as soon as possible, regardless of (in)stability.
    --min-delay-minutes <minutes>
        Minimum wait (in miunutes) before a newly encountered item can be notified.
    --await-stabilization-minutes <minutes>
        Period (in minutes) over which an item should not have changed before it is considered stable and can be notified.
    --max-delay-minutes <minutes>
        Notwithstanding other settings, maximum period past which an item should be notified, regardless of its stability.
    --recheck-every-minutes <minutes>
        Delay between refreshes of feeds, and redetermining items' availability for notification.

When we add feeds, we also define how "finalization" of feed items will be defined. Items will never be notified or considered final prior to min-delay-minutes. Even after this period has passed, they will not be considered final unless they have been stable (the item has been unchanged) for at least await-stabilization-minutes, or until max-delay-minutes has passed. (max-delay-minutes is a failsafe, in case feed items never stabilize due to a changing timestamp or such.)

Feeds will be polled every recheck-every-minutes minutes.

If --ping (and only --ping) is set, feedletter will poll at its maximum frequency and notify immediately, irrespective of the item's (in)stability.

All of the (non-ping) values have defaults. Let's have our application watch the blog Lawyers, Guns, and Money, whose feed is at https://www.lawyersgunsmoneyblog.com/feed:

$ ./feedletter add-feed  https://www.lawyersgunsmoneyblog.com/feed
[49/49] runMain 
+---------+-------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+
¦ Feed ID ¦ Feed URL                                  ¦ Min Delay Mins ¦ Await Stabilization Mins ¦ Max Delay Mins ¦ Recheck Every Mins ¦ Added                       ¦ Last Assigned               ¦
+---------+-------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+
¦ 1       ¦ https://www.lawyersgunsmoneyblog.com/feed ¦ 30             ¦ 15                       ¦ 180            ¦ 10                 ¦ 2024-01-27T17:03:47.452533Z ¦ 2024-01-27T17:03:47.452533Z ¦
+---------+-------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+

So, by default, this feed will wait at least 30 minutes before notifying, and require a post to have been stable for at least 15 minutes. After 180 minutes, it will be considered final no matter what. It will be checked every approximately 10 minutes.

If you don't like these values, you can change them any time with the ./feedletter alter-feed command.

I am not republishing these blogs without permission. That would be icky. I'm using these feeds for demonstration purposes. I'll be their only e-mail subscriber.

By the time you read this tutorial, play.feedletter.org will have been sadly retired.

Let's add another feed to watch, Atrios' Eschaton blog, whose feed URL is https://www.eschatonblog.com/feeds/posts/default?alt=rss. I'm just going to stick with the default timings for now:

$ ./feedletter add-feed https://www.eschatonblog.com/feeds/posts/default?alt=rss

[49/49] runMain 
+---------+----------------------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+
¦ Feed ID ¦ Feed URL                                                 ¦ Min Delay Mins ¦ Await Stabilization Mins ¦ Max Delay Mins ¦ Recheck Every Mins ¦ Added                       ¦ Last Assigned               ¦
+---------+----------------------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+
¦ 1       ¦ https://www.lawyersgunsmoneyblog.com/feed                ¦ 30             ¦ 15                       ¦ 180            ¦ 10                 ¦ 2024-01-27T17:03:47.452533Z ¦ 2024-01-27T17:03:47.452533Z ¦
¦ 2       ¦ https://www.eschatonblog.com/feeds/posts/default?alt=rss ¦ 30             ¦ 15                       ¦ 180            ¦ 10                 ¦ 2024-01-27T17:04:55.092686Z ¦ 2024-01-27T17:04:55.092686Z ¦
+---------+----------------------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+

12. Define "subscribables" to feeds

Once the application is watching feeds, we can define various kinds of "subscribables" to them.

A subscribable is a subscription type. We use the made-up word to distinguish a thing you can subscribe to (a "subscribable") from an individual's subscription.

E-mail subscribables are by default one-post-per-newsletter, but they can also be defined as daily digests, weekly compendia, or bundles of every n posts, for an n you choose.

Let's take a look at the ./feedletter define-email-subscribable command:

$ ./feedletter define-email-subscribable --help
[49/49] runMain 
Usage: feedletter define-email-subscribable --feed-id <feed-id> --name <name> --from <e-mail address> [--reply-to <e-mail address>] [--compose-untemplate <fully-qualified-name>] [--confirm-untemplate <fully-qualified-name>] [--removal-notification-untemplate <fully-qualified-name>] [--status-change-untemplate <fully-qualified-name>] [--each | --daily [--time-zone <id>] | --weekly [--time-zone <id>] | --num-items-per-letter <num>] [--extra-param <key:value>]...

Define a new email subscribable, a mailing lost to which users can subscribe.

Options and flags:
    --help
        Display this help text.
    --feed-id <feed-id>
        The ID of the RSS feed to be watched.
    --name <name>
        A name for the new subscribable.
    --from <e-mail address>
        The email address from which emails should be sent.
    --reply-to <e-mail address>
        E-mail address to which recipients should reply (if different from the 'from' address).
    --compose-untemplate <fully-qualified-name>
        Fully qualified name of untemplate that will render notifications.
    --confirm-untemplate <fully-qualified-name>
        Fully qualified name of untemplate that will ask for e-mail confirmations.
    --removal-notification-untemplate <fully-qualified-name>
        Fully qualified name of untemplate that be mailed to users upon unsubscription.
    --status-change-untemplate <fully-qualified-name>
        Fully qualified name of untemplate that will render results of GET request to the API.
    --each
        E-mail each item.
    --daily
        E-mail a compilation, once a day.
    --time-zone <id>
        ID of a time zone for determining the beginning and end of the period.
    --weekly
        E-mail a compilation, once a week.
    --num-items-per-letter <num>
        E-mail every fixed number of posts.
    --extra-param <key:value>
        An extra parameter your notification renderers might use.
1 targets failed
runMain subprocess failed

The only required elements are --feed-id <id>, --name <a name you choose name>, and --from <e-mail address>. If you set only these, feedletter will use its default style (you'll not have set any custom "untemplates"), it will have no "reply to" address distinct from the "from" address you've given, and it will be of type --each, that is one e-mail per post.

If you set the flag --daily you'll send daily digests. If you set the flag --weekly, then weekly compendia. If you set --num-items-per-letter <num>, you'll send an e-mail every num posts.

More rarely, you can set any number of --extra-param items. These can be passed through to custom untemplates, to configure your own styles and themes as you see fit.

Lawyers, Guns, and Money tends to post long-ish essays, so the default --each type is probably appropriate.

Let's recall (scroll up!) that when we added that feed, it was given ID 1. So let's create a subscribable:

$ ./feedletter define-email-subscribable --name lgm --feed-id 1 --from feedletter@feedletter.org
[49/49] runMain 

-*-*-*-

Subscribable Name:    lgm
Feed ID:              1
Subscription Manager: {
    "composeUntemplateName": "com.mchange.feedletter.default.email.composeUniversal_html",
    "statusChangeUntemplateName": "com.mchange.feedletter.default.email.statusChange_html",
    "confirmUntemplateName": "com.mchange.feedletter.default.email.confirm_html",
    "from": {
        "addressPart": "feedletter@feedletter.org",
        "type": "Email",
        "version": 1
    },
    "removalNotificationUntemplateName": "com.mchange.feedletter.default.email.removalNotification_html",
    "extraParams": {},
    "type": "Email.Each",
    "version": 1
}
An email subscribable to feed with ID '1' named 'lgm' has been created.

We can create more than one subscribable to a single feed! Let's also make a daily roundup option for Lawyers, Guns, and Money:

$ ./feedletter define-email-subscribable --name lgm-daily --feed-id 1 --from feedletter@feedletter.org --daily
[49/49] runMain 

-*-*-*-

Subscribable Name:    lgm-daily
Feed ID:              1
Subscription Manager: {
    "composeUntemplateName": "com.mchange.feedletter.default.email.composeUniversal_html",
    "statusChangeUntemplateName": "com.mchange.feedletter.default.email.statusChange_html",
    "confirmUntemplateName": "com.mchange.feedletter.default.email.confirm_html",
    "from": {
        "addressPart": "feedletter@feedletter.org",
        "type": "Email",
        "version": 1
    },
    "removalNotificationUntemplateName": "com.mchange.feedletter.default.email.removalNotification_html",
    "extraParams": {},
    "type": "Email.Daily",
    "version": 1
}
An email subscribable to feed with ID '1' named 'lgm-daily' has been created.

Atrios' Eschaton blog publishes frequent, sometimes very short posts. Let's create a subscribable that sends out groups of three. Recall from above that its feed ID was 2. So...

$ ./feedletter define-email-subscribable --name atrios-three --feed-id 2 --from feedletter@feedletter.org --num-items-per-letter 3
[49/49] runMain 

-*-*-*-

Subscribable Name:    atrios-three
Feed ID:              2
Subscription Manager: {
    "composeUntemplateName": "com.mchange.feedletter.default.email.composeUniversal_html",
    "statusChangeUntemplateName": "com.mchange.feedletter.default.email.statusChange_html",
    "numItemsPerLetter": 3,
    "confirmUntemplateName": "com.mchange.feedletter.default.email.confirm_html",
    "from": {
        "addressPart": "feedletter@feedletter.org",
        "type": "Email",
        "version": 1
    },
    "removalNotificationUntemplateName": "com.mchange.feedletter.default.email.removalNotification_html",
    "extraParams": {},
    "type": "Email.Fixed",
    "version": 1
}
An email subscribable to feed with ID '2' named 'atrios-three' has been created.

13. Enable feedletter as a systemd daemon.

Let’s define a feedletter.service file right here in our installation directory, just because it seems convenient. We edit /home/feedletter/feedletter-local/feedletter.service:

[Unit]
Description=Feedletter RSS-To-Mail-Etc Service
After=syslog.target network.target

[Service]
Type=forking
PIDFile=/home/feedletter/feedletter-local/feedletter.pid
User=feedletter
Group=feedletter
WorkingDirectory=/home/feedletter/feedletter-local

ExecStart=/home/feedletter/feedletter-local/feedletter daemon --fork

TimeoutStopSec=90
Restart=on-failure
RestartSec=10s
StandardError=journal
StandardOutput=journal
StandardInput=null

[Install]
WantedBy=multi-user.target

Now we setup the symlinks that would make this a permanent systemd service. First we exit to get back to root, then…

$ exit
logout
# cd /etc/systemd/system/
# ln -s /home/feedletter/feedletter-local/feedletter.service 
# systemctl enable feedletter
Created symlink /etc/systemd/system/multi-user.target.wants/feedletter.service → /home/feedletter/feedletter-local/feedletter.service.

Now let's actually start our new service, and check its logs:

# systemctl start feedletter
# journalctl -u feedletter --follow
Jan 27 17:11:53 feedletter-play systemd[1]: Starting feedletter.service - Feedletter RSS-To-Mail-Etc Service...
Jan 27 17:11:59 feedletter-play systemd[1]: feedletter.service: Can't open PID file /home/feedletter/feedletter-local/feedletter.pid (yet?) after start: No such file or directory
Jan 27 17:12:02 feedletter-play feedletter[37405]: Jan 27, 2024 5:12:02 PM com.mchange.v2.log.MLog
Jan 27 17:12:02 feedletter-play feedletter[37405]: INFO: MLog clients using java 1.4+ standard logging.
Jan 27 17:12:06 feedletter-play systemd[1]: Started feedletter.service - Feedletter RSS-To-Mail-Etc Service.
Jan 27 17:12:07 feedletter-play feedletter[37405]: 2024-01-27@17:12:07 [INFO] [com.mchange.feedletter.Daemon] Spawning daemon fibers.
Jan 27 17:12:07 feedletter-play feedletter[37405]: 2024-01-27@17:12:07 [INFO] [com.mchange.feedletter.Daemon] Starting web API service on interface '127.0.0.1', port 8024.

It all looks good!

Occasionally I've had problems at first seeing log entries using journalctl. I'd see messages like

No journal files were found.
-- No entries --

The fix is to run

# systemctl restart systemd-journald.service

and then to restart the feedletter service.

14. Let users subscribe to your subscribables!

The feedletter services has a simple API that, for now, uses (abuses) the HTTP GET method. Here’s an example of an HTML form that would allow subscription to our new newsletter:

    <form id="subscribe-form" action="https://play.feedletter.org/v0/subscription/create" method="GET">
      <input type="hidden" name="subscribableName" value="lgm">
      E-mail: <input type="text" name="addressPart"><br>
      Display Name: <input type="text" name="displayNamePart"> (Optional)<br>
      <input name="main-submit" value="Subscribe!" type="submit">
    </form>

As of feedletter v0.0.8, you can use method="POST" in subscribe forms.

Using method="GET" (and therefore also simulating form submission by pasting a URL) remain supported as well.

(You can see live examples of feedletter subscription forms on the subscribe page of this site!)

We will fake hitting the form above just by pasting the following URL into our browser:

https://play.feedletter.org/v0/subscription/create?subscribableName=lgm&addressPart=swaldman@mchange.com&displayNamePart=Steve

We are immediately informed of our success: Screenshot of 'Subscription Created' page And, you've got mail!

Screenshot of e-mail requesting subscription confirmation

We hit the confirm link and we're done:

Screenshot of 'Subscription confirmed!' page

We've made two more subscribables we'll want to test, whose let's-fake-a-form URLs will be

https://play.feedletter.org/v0/subscription/create?subscribableName=lgm-daily&addressPart=swaldman@mchange.com&displayNamePart=Steve
https://play.feedletter.org/v0/subscription/create?subscribableName=atrios-three&addressPart=swaldman@mchange.com&displayNamePart=Steve

We go through the same (faked) submit then confirm steps for each of these.

And now we're subscribed! We just have to wait for the mail to roll in.

15. Tweak the newsletter styles

But what will this mail actually look like? We can sneak a peek.

We will want to have two terminal windows open, logged into our feedletter host. In one terminal, we will run a single-page webserver that simulates the HTML e-mails we will receive.

In the second terminal, we can edit an untemplate, until we have the look we want. Then we can update our subscribable to use our perfected untemplate to generate its mails.

Let's get the simulation server running. It's easy to run, but we won't be able to see what it's serving if we don't open up a port on our server to serve it through. We'll use port 45612. Making that available on Ubuntu is just

# ufw allow 45612
Rules updated
Rules updated (v6)

Now we just become user feedletter again, and check out the feedletter-style command.

$ ./feedletter-style --help
[50/50] runMainBackground 
Watching for changes to 14 paths and 9 other values... (Enter to re-run, Ctrl-C to exit)
Usage:
    feedletter-style [--secrets <propsfile>] compose-multiple
    feedletter-style [--secrets <propsfile>] compose-single
    feedletter-style [--secrets <propsfile>] confirm
    feedletter-style [--secrets <propsfile>] removal-notification
    feedletter-style [--secrets <propsfile>] status-change

Iteratively edit and review the untemplates through which your posts will be notified.

Options and flags:
    --help
        Display this help text.
    --secrets <propsfile>
        Path to properties file containing SMTP, postgres, c3p0, and other configuration details.

Environment Variables:
    FEEDLETTER_SECRETS=<path>
        Path to properties file containing SMTP, postgres, c3p0, and other configuration details.

Subcommands:
    compose-multiple
        Style a template that composes a multiple items.
    compose-single
        Style a template that composes a single item.
    confirm
        Style a template that asks users to confirm a subscription.
    removal-notification
        Style a template that notifies users that they have subscribed.
    status-change
        Style a template that informs users of a subscription status change.

You can style the infrastructure — confirmation and removal e-mails, web pages that inform users that their subscription status has changed.

But mostly you'll want to style the compose untemplates. For subscribables that mail just one e-mail at a time, you'll want compose-single. For subscribables that will pull together multple posts into a single mail, you'll want compose-multiple.

The ./feedletter-style command never terminates. You have to type <ctrl-c> to quit out.

This is because it's designed to be terminated and restarted each time you change underlying templates or css. A mill process runs perpetually, watching for chages and restarting whatever command you last tried.

Let's try compose-single. Our subscribable lgm sends just one post per-email. Let's try to style it.

$ ./feedletter-style compose-single --help
[50/50] runMainBackground 
Watching for changes to 14 paths and 9 other values... (Enter to re-run, Ctrl-C to exit)
Usage: feedletter-style compose-single --subscribable-name <name> [--untemplate-name <fully-qualified-name>] [--first | --random | --guid <string>] [--e-mail <address> [--display-name <name>] | --sms <number> | --masto-instance-name <name> --masto-instance-url <url>] [--within-type-id <string>] [--interface <interface>] [--port <num>]

Style a template that composes a single item.

Options and flags:
    --help
        Display this help text.
    --subscribable-name <name>
        The name of an already defined subscribable that will use this template.
    --untemplate-name <fully-qualified-name>
        Fully name of an untemplate to style.
    --first
        Display first item in feed.
    --random
        Choose random item from feed to display
    --guid <string>
        Choose guid of item to display.
    --e-mail <address>
        The e-mail address to subscribe.
    --display-name <name>
        A display name to wrap around the e-mail address.
    --sms <number>
        The number to which messages should be sent.
    --masto-instance-name <name>
        A private name for this Mastodon instance.
    --masto-instance-url <url>
        The URL of the Mastodon instance
    --within-type-id <string>
        A subscription-type specific sample within-type-id for the notification.
    --interface <interface>
        The interface on which to bind an HTTP server, which will serve the rendered untemplate.
    --port <num>
        The port on which to run a HTTP server, which will serve the rendered untemplate.

There's a lot here, but note that the only required option is --subscribable-name. We've opened port 45612, so we'll also want to hit the --port option. Let's try running the ./feedletter-style compose-single command for subscribable lgm:

$ ./feedletter-style compose-single --subscribable-name lgm --port 45612
[50/50] runMainBackground 
Watching for changes to 14 paths and 9 other values... (Enter to re-run, Ctrl-C to exit)
Starting single-page webserver on interface '0.0.0.0', port 45612...

Great. Now let's see how our newsletter looks, with its HTML served on http://play.feedletter.org:45612/. Not so good!

Screenshot of web-served lgm newsletter via ./feedletter-style compose-single, with a badly formatted image

(Update: As of feedletter v0.0.8 you can also style newsletters by e-mail, in addition to hitting a development webserver with a browser.)

By default, we just pulled the first item (and most recent, since blogs are usually reverse-chronological) from the feed. We can also pull a random item off the feed to view with --random, or a particular item identified by its <guid> element in the feed with --guid <guid>.

Let's see how we can restyle this post to make it a bit better.

We have been using the built-in default untemplate to compose our items. We cannot modify that.

But our feedletter installation directory contains a copy of this untemplate that we can deploy and tweak.

To do so, we'll have to create a folder for this file under untemplate. We’ll call it tutorial. Then...

$ mkdir untemplate/tutorial/
$ cp sample/defaultCompose.html.untemplate untemplate/tutorial/lgmCompose.html.untemplate

feedletter now has access to this untemplate, under a name you can find by calling ./feedletter list-untemplates:

$ ./feedletter list-untemplates
[42/49] compile 
[info] compiling 2 Scala sources to /home/feedletter/feedletter-local/out/compile.dest/classes ...
[info] done compiling
[49/49] runMain 
+---------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
¦ Untemplate, Fully Qualified Name                              ¦ Input Type                                                                                          ¦
+---------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
¦ com.mchange.feedletter.default.email.composeUniversal_html    ¦ com.mchange.feedletter.style.ComposeInfo.Universal                                                  ¦
¦ com.mchange.feedletter.default.email.confirm_html             ¦ com.mchange.feedletter.style.ConfirmInfo                                                            ¦
¦ com.mchange.feedletter.default.email.item_html                ¦ scala.Tuple2[com.mchange.feedletter.style.ComposeInfo.Universal,com.mchange.feedletter.ItemContent] ¦
¦ com.mchange.feedletter.default.email.removalNotification_html ¦ com.mchange.feedletter.style.RemovalNotificationInfo                                                ¦
¦ com.mchange.feedletter.default.email.statusChange_html        ¦ com.mchange.feedletter.style.StatusChangeInfo                                                       ¦
¦ com.mchange.feedletter.default.email.style_css                ¦ scala.collection.immutable.Map[java.lang.String,scala.Any]                                          ¦
¦ tutorial.lgmCompose_html                                      ¦ com.mchange.feedletter.style.ComposeInfo.Universal                                                  ¦
+---------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+

Now we can ask feedletter-style to show us what this post would look like using our "new" untemplate to render it:

$ ./feedletter-style compose-single --subscribable-name lgm --untemplate-name tutorial.lgmCompose_html --port 45612
[50/50] runMainBackground 
Watching for changes to 14 paths and 9 other values... (Enter to re-run, Ctrl-C to exit)
Starting single-page webserver on interface '0.0.0.0', port 45612...

Initially it looks exactly the same, because it is just a copy of the default untemplate!

But now we can just modify that file, untemplate/tutorial/lgmCompose.html.untemplate, hit reload, and play!

This is why we needed a second terminal window. We edit the template in one terminal while ./feedletter-style is running in the other. After each edit and save, we hit reload to see our changes. (We may have to wait 10-15 secs!)

Occasionally the autoreload glitches out, in which case you should manually <ctrl-c> and rerun your ./feedletter-style command.

If you see error messages when you rerun, you may have hit compilation errors (an untemplate is transformed into a Scala source code, which is then compiled), which you will have to resolve. You can ask for help!)

Our new untemplate has a section that looks like this:

<html>
  <head>
    <style>
      <( style_css() )>
      /* add extra CSS styling here! */
    </style>

Let's go ahead and add some CSS! We'll edit it to...

<html>
  <head>
    <style>
      <( style_css() )>
      /* add extra CSS styling here! */
      img {
        width: 100%;
        height: auto;
      }
    </style>

We save, and hit reload on our browser still pointed at http://play.feedletter.org:45612/, and see...

Screenshot of web-served lgm newsletter with a better laid-out image.

Much better!

If we are very picky, we see that at the end of our post, there is a line that doesn't logically belong in the post, and should be italicized or something.

Screenshot of web-served lgm newsletter with a better laid-out image.

If we view the source, we'll find it's the last <p> element in <div class="item-contents">. So we modify our styling as follows:

<html>
  <head>
    <style>
      <( style_css() )>
      /* add extra CSS styling here! */
      img {
        width: 100%;
        height: auto;
      }
      div.item-contents p:last-of-type {
        font-style: italic;
      }
    </style>

Looks better!

Screenshot of web-served lgm newsletter with a better laid-out image.

We can keep editing all we like. We add the --random flag and run our ./feedletter-style command over and over to make sure that posts in general render well.

When we are happy, we want to tell our subscription to use the new untemplate.

Remember, the name of the untemplate we've been editing was tutorial.lgmCompose_html.

Let's check that command out:

$ ./feedletter set-untemplates --help
[49/49] runMain 
Usage: feedletter set-untemplates --subscribable-name <name> [--compose-untemplate <fully-qualified-name>] [--confirm-untemplate <fully-qualified-name>] [--removal-notification-untemplate <fully-qualified-name>] [--status-change-untemplate <fully-qualified-name>]

Update the untemplates used to render subscriptions.

Options and flags:
    --help
        Display this help text.
    --subscribable-name <name>
        The name of an already-defined subscribable.
    --compose-untemplate <fully-qualified-name>
        Fully qualified name of untemplate that will render notifications.
    --confirm-untemplate <fully-qualified-name>
        Fully qualified name of untemplate that will ask for e-mail confirmations.
    --removal-notification-untemplate <fully-qualified-name>
        Fully qualified name of untemplate that be mailed to users upon unsubscription.
    --status-change-untemplate <fully-qualified-name>
        Fully qualified name of untemplate that will render results of GET request to the API.
1 targets failed
runMain subprocess failed

Okay. So we run...

$ ./feedletter set-untemplates --subscribable-name lgm --compose-untemplate tutorial.lgmCompose_html
[49/49] runMain 
Updated Subscription Manager: {
    "composeUntemplateName": "tutorial.lgmCompose_html",
    "statusChangeUntemplateName": "com.mchange.feedletter.default.email.statusChange_html",
    "confirmUntemplateName": "com.mchange.feedletter.default.email.confirm_html",
    "from": {
        "addressPart": "feedletter@feedletter.org",
        "type": "Email",
        "version": 1
    },
    "removalNotificationUntemplateName": "com.mchange.feedletter.default.email.removalNotification_html",
    "extraParams": {},
    "type": "Email.Each",
    "version": 1
}

And we are done! We have restyled our newsletter.

We could (and should!) do the same with our other subscriptions (using ./feedletter-style compose-multiple). We could also do much more elaborate things then just mess with the stylesheet. Our compose untemplate was really the definition of a pretty arbitrary Scala function that accepted a ComposeInfo.Single object and produced a String (embedded in an untemplate.Result).

Learn more about untemplates here.

The default compose untemplate actually accepts a ComposeInfo.Universal, a parent type of both ComposeInfo.Single and ComposeInfo.Multiple. So we can fix up the glitches we know about already in our lgm-daily subscribable just by setting for it the same compose untemplate:

$ ./feedletter set-untemplates --subscribable-name lgm-daily --compose-untemplate tutorial.lgmCompose_html
[49/49] runMain 
Updated Subscription Manager: {
    "composeUntemplateName": "tutorial.lgmCompose_html",
    "statusChangeUntemplateName": "com.mchange.feedletter.default.email.statusChange_html",
    "confirmUntemplateName": "com.mchange.feedletter.default.email.confirm_html",
    "from": {
        "addressPart": "feedletter@feedletter.org",
        "type": "Email",
        "version": 1
    },
    "removalNotificationUntemplateName": "com.mchange.feedletter.default.email.removalNotification_html",
    "extraParams": {},
    "type": "Email.Daily",
    "version": 1
}

If we take a look at that with compose-multiple..

$ ./feedletter-style compose-multiple --subscribable-name lgm-daily --port 45612
[50/50] runMainBackground 
Watching for changes to 14 paths and 9 other values... (Enter to re-run, Ctrl-C to exit)
Starting single-page webserver on interface '0.0.0.0', port 45612...

We'll find that it looks pretty good!

16. Advanced: Customize the content

feedletter supports a variety of customizers, including

  • subject customizers
  • contents customizers
  • "MastoAnnouncement" customizers"
  • template params customizers (see templating note below)

For each subscribable, you can define just one of each kind of customizer, but customers can perform any number of steps internally.

For an example, we'll build a content customizer. Both of our feeds frequently embed YouTube videos as iframe HTML elements in their blog posts. Unfortunately, mail clients generally do not render this form of embedded content, leaving awkward empty-spaces in and sometimes mangling the formatting of our newsletters.

So let's build a content customizer that replaces these with well-behaved div elements containing links to the resources that would have been in the iframe. We'll include a class="embedded" attribute on the div elements, so that we will be able to style them however we want.

Writing customizers in writing Scala code. We'll use the excellent jsoup library to manipulate HTML. We'll give ourselves space to work by creating a tutorial package in our installation's src directory, and then exiting a file called core.scala inside that.

$ mkdir src/tutorial
$ emacs src/tutorial/core.scala

First, we write a function that takes post HTML, and transforms the iframe elements into the div elements we're after. Then we embed that in the form of a Customizer.Contents, which is a function that accepts some metainformation and the original contents of a feed as ItemContents objects, and then outputs transformed contents.

Here is what all that looks like:

package tutorial

import org.jsoup.Jsoup
import org.jsoup.nodes.{Document,Element}

import scala.jdk.CollectionConverters.*

import com.mchange.feedletter.*
import com.mchange.feedletter.style.Customizer

private def createDivEmbedded( link : String ) : Element =
  val div = new Element("div").attr("class","embedded")
  val a = new Element("a").attr("href",link)
  val linkText =
    if link.toLowerCase.contains("youtube.com/") then
      "Embedded YouTube video"
    else
      "Embedded item"
  a.append(linkText)
  div.appendChild(a)
  div

def iframeToDivEmbedded( html : String ) : String =
  val doc = Jsoup.parseBodyFragment( html )
  val iframes = doc.select("iframe").asScala
  iframes.foreach: ifr =>
    val src = ifr.attribute("src").getValue()
    ifr.replaceWith( createDivEmbedded(src) )
  doc.body().html()

val IframelessCustomizer : Customizer.Contents =
  ( subscribableName : SubscribableName, subscriptionManager : SubscriptionManager, withinTypeId : String, feedUrl : FeedUrl, contents : Seq[ItemContent] ) =>
    contents.map: ic =>
      ic.article match
        case Some( html ) => ic.withArticle( iframeToDivEmbedded( html ) )
        case None => ic

Once we have IframelessCustomizer defined, to "install" it, we just register it as the Customizer.Contents for each of our feeds in our installation's PreMain object.

We modify the default src/PreMain.scala, just inserting three Customizer.Contents.register(...) lines (and the import that brings in the name Customizer).

import com.mchange.feedletter.{UserUntemplates,Main}
import com.mchange.feedletter.style.{AllUntemplates,StyleMain}

import com.mchange.feedletter.style.Customizer

object PreMain:
  def main( args : Array[String] ) : Unit =
    AllUntemplates.add( UserUntemplates )
    Customizer.Contents.register("lgm", tutorial.IframelessCustomizer)
    Customizer.Contents.register("lgm-daily", tutorial.IframelessCustomizer)
    Customizer.Contents.register("atrios-three", tutorial.IframelessCustomizer)
    val styleExec =
      sys.env.get("FEEDLETTER_STYLE") match
        case Some( s ) => s.toBoolean
        case None      => false
    if styleExec then StyleMain.main(args) else Main.main(args)

Once the customizers are registered, they will be called whenever the application generates content for the named subscribable.

We can verify that our customizer does as we expect by using ./feedletter-style to preview newsletter output. (See above).

$ ./feedletter-style compose-multiple --subscribable-name atrios-three --port 45612
[50/50] runMainBackground 
Watching for changes to 14 paths and 9 other values... (Enter to re-run, Ctrl-C to exit)
Starting single-page webserver on interface '0.0.0.0', port 45612...

We can find one of Atrios' "Rock on." posts, which used to render blank in mail clients, but now render like...

Screenshot of a transformed-to-div iframe

Of course we can style that div and link however we like.

Re: "TemplateParams" customizers

Confusingly, feedletter newsletters are rendered with two kinds of templating.

  • "untemplates" render newsletter HTML.
  • but that HTML can itself be a template, by including case-insensitive constructs like %PercentDelimitedKey% which get filled in just prior to notification.

The role of the second round of templating is to add subscriber-specific customizations, which might commonly include a particular subscriber's name and e-mail, as well as an unsubscribe link specific to that subscriber.

Each notification is rendered by an untemplate just once, but any %Key% left in that rendering can be filled in differently for each subscriber.

Template-params customizers let you add key-value pairs to the built-in set of available substitutions for these last-minute, per-subscriber customizations.

Conclusion

This was a lot!

It probably seems intimidating.

But if you know how to self-host systemd daemon processes, much of the above should have been familiar. Setting up a feedletter server should take one to two hours of your time.

Defining new feeds and subscribables, once the server is set up, becomes just a 5 minute operation.

One feedletter instance can host as many feeds and subscribables as you like.

Restyling your subscribables, or writing customizers and bespoke untemplates for them, can take longer. Developing custom front-ends is time-consuming detail work.

I'd love it if you gave feedletter a try!

2023-12-06

APIs against dependent types in Scala


Scala supports instance-dependent types, which is very cool! So I can define...

class Human( name : String ):
  case class Tooth( num : Int ):
    override def toString(): String = s"${name}'s #${num} tooth"
    
  val teeth = Set.from( (1 to 32).map( Tooth.apply ) )
  def brush( myTeeth : Set[Tooth] ) : Unit = println(s"fluoride goodness for ${name}")
  
val me = new Human("Steve")
val you = new Human("Awesome")

me.brush( me.teeth )
//me.brush( you.teeth ) // gross! doesn't compile. (as it should not!)

My teeth and your teeth are different types, even though they are of the same class. The identity of the enclosing instance is a part of the type.

And we see here how that can be useful! Often inner classes represent internal structures that should mostly be managed by their enclosing instance. It's good that the compiler pushes back against code in which you might brush my teeth or pump my heart!

But sometimes inner instances are not so internal, or even if they are, an external thing might have business interacting with it. The virtual human we are modeling might have need of a dentist or a cadiologist.

Scala's type system doesn't prevent external things from accessing inner class instances, it just demands you do it via a correct type.

I know of two ways to define external APIs against instance-dependent types. First, Scala supports projection types, like Human#Teeth. Where an ordinary dot-separated path would have required me to identify some particular instance, Human#Teeth matches the tooth of any human.

A second way to hit instance-dependent types from an external API is to require the caller to identify the instance in the call, and then let the type of a later argument to the same call include the identified instance. I think it's kind of wild that Scala supports this. It's an example where the type of arguments to a statically declared function is effectively determined at runtime. You don't even need separate argument lists, although I think I prefer them.

class Dentist:
  def checkByProjection( tooth : Human#Tooth ) : Unit = println( s"Found ${tooth} (by projection)" )
  def checkByIdentifying( human : Human)( tooth : human.Tooth ) : Unit = println( s"Found ${tooth} (by identification)" )

val d  = new Dentist

// API by projection
d.checkByProjection( me.teeth.head )
d.checkByProjection( you.teeth.head )

// API by identification
d.checkByIdentifying( me )( me.teeth.head )
d.checkByIdentifying( you )( you.teeth.head )

// d.checkByIdentifying( me )( you.teeth.head ) // does not compile, as it should not
// d.checkByIdentifying( you )( me.teeth.head ) // does not compile, as it should not

I've used projection types a lot, over the eons. I know some people think that any need for external APIs against inner types is code smell or something. But I've found a variety of places where they seem to make sense, and the "do it right" workarounds (e.g. define some instance-independent abstract base type for the inner things, and write external APIs against that) just create busy work and maintenance complexity.

Nevertheless, in some corner cases, projection types aren't completely supported, and my sense is that much of the Scala community considers them icky (like brushing someone else's teeth).

Sometimes you need to write APIs against inner types by identification anyway, because you need to know stuff about the enclosing instance (which inner instances don't disclose unless they declare an explicit reference).

But sometimes you don't need to be told the identity of the outer instance (because it's not relevant to what you are doing, or because the inner instance discloses a reference explicitly).

Are projection types icky and it best to just standardize on requiring explicit identification of enclosing instances?

Or are projection types a cool trick we should delight in using?

Enquiring minds want to know!


(This blog doesn't support comments yet, but you can reply to this post on Mastodon.)

2023-11-14

(Library + Script) vs (Application + Config File)


TL; DR:

For Scala apps, instead of installing applications and writing separate config files, why not do config like this?

#!/usr/bin/env -S scala-cli shebang

//> using dep "com.example::cool-app:1.0.0"

val config = coolapp.Config(
  name = "Fonzie",                    // the name of your installation
  apparel = coolapp.Apparel.Leather,  // see elements defined in coolapp.Apparel
  gesture = coolapp.Gesture.ThumbsUp, // see elements defined in coolapp.Gesture
  reference = "Very dated, old man.", // a string to help users identify your character
  port = 8765                         // the port on which the app will run
)

coolapp.start( config )

Once upon a time, I spent a very great deal of time supporting and integrating multiple config formats into my work. I used to describe c3p0 as a configuration project attached to a connection pool.

Lately, though, I find I am skipping any support of config files. I mostly write Scala, and Scala case classes strike me as a pretty good configuration format.

  • Since you can intitialize case classes with named arguments, key = value, they can be made literate and intuitive.

  • They support rich comments, because the Scala language supports comments.

  • With simple string or integer values, they are as simple as most config formats.

Case-class config is extremely flexible, because your values are specified in a general purpose programming language, and can include variables or functions. And you get compile-time feedback for misconfigurations.

When I first became enamored with case-classes-as-config, I wrote a special purpose bootstrap app that would compile a file containing a case-class-instance-as-config, then use Java reflection to load it from a container.

val podcast : Podcast =
    Podcast(
      mainUrl                = "https://superpodcast.audiofluidity.com/",
      title                  = "Superpodcast",
      description            = """|<p>Superpodcast is the best podcast you've ever heard.</p>
                                  |
                                  |<p>In fact, you will never hear it.</p>""".stripMargin,
      guidPrefix             = "com.audiofluidity.superpodcast-",
      shortOpaqueName        = "superpodcast",
      mainCoverImageFileName = "some-cover-art.jpg",
      editorEmail            = "asshole@audiofluidity.com",
      defaultAuthorEmail     = "asshole@audiofluidity.com",
      itunesCategories       = immutable.Seq( ItunesCategory.Comedy ),
      mbAdmin                = Some(Admin(name="Asshole", email="asshole@audiofluidity.com")),
      mbLanguage             = Some(LanguageCode.EnglishUnitedStates),
      mbPublisher            = Some("Does Not Exist, LLC"),
      episodes               = episodes
    )

In more recent projects, I've just used either scala-cli or mill as a runner. Sometimes I've left the definition of a stub case-class instance in the src directory for users to fill in, as in fossilphant. Other times I've defined abstract main classes, asking users to extend them by overriding a method that supplies config as a case class instance, as in unify-rss.

package com.mchange.unifyrss

import scala.collection.*

import zio.*

abstract class AbstractDaemonMain extends ZIOAppDefault:

  def appConfig : AppConfig

  override def run =
    for
      mergedFeedRefs   <- initMergedFeedRefs( appConfig )
      _                <- periodicallyResilientlyUpdateAllMergedFeedRefs( appConfig, mergedFeedRefs )
      _                <- ZIO.logInfo(s"Starting up unify-rss server on port ${appConfig.servicePort}")
      exitCode         <- server( appConfig, mergedFeedRefs )
    yield exitCode

So far, I've just instantiated these with concrete objects in Scala source files.

But it strikes me that a natural refinement would be to design libraries with entry points that accept a case-class-config object as an argument, and expect users to deploy them as e.g. scala-cli scripts. Just something like:

#!/usr/bin/env -S scala-cli shebang

//> using dep "com.example::cool-app:1.0.0"

val config = coolapp.Config(
  name = "Fonzie",                    // the name of your installation
  apparel = coolapp.Apparel.Leather,  // see elements defined in coolapp.Apparel
  gesture = coolapp.Gesture.ThumbsUp, // see elements defined in coolapp.Gesture
  reference = "Very dated, old man.", // a string to help users identify your character
  port = 8765                         // the port on which the app will run
)

coolapp.start( config )

There is a bit of ceremony, and a bit that might intimidate people not accustomed to Scala syntax and tools. But "standard" config file formats get complicated and intimidating too. Here users get quick feedback if they don't pick a valid value without developers having to write special validation logic. Users are still just deploying a text file, as they would with ordinary config.

If your priority is 100% user experience, then using a standard (or new and improved, ht Bill Mill) config file format, then hand-writing informative, fail-fast validation logic is going to be a better way to go.

But your priority should not always be user experience! Not all software development should take the form of a "product" developed at a high cost that will then be amortized over sales to or adoption by a very large number of users.

Software is a form of collaboration, and often that collaboration will be more productive and evolve more quickly when "users" are understood to be reasonably capable and informed, so developers don't expand the scope of their work and their maintenance burden in order to render the application accessible to the most intimidated potential users.

Obviously it depends what you are doing! But if there is going to be a config file at all, you are already collaborating with a pretty restricted set of people who are okay with setting up and editing an inevitably arcane text file.

For many applications and collaborations, maintainability at moderate cost in time and money and speed of evolution, are important. For these applications, when written in an expressive, strongly-typed language like Scala, defining config as a data structure in a script, that then executes an app defined as an entry point to a library, strike me as a pretty good way to go.

2023-10-09

Contributing to mill


I'm a big fan of Scala build tools, both sbt and mill. I've done some pretty big projects intimately based on sbt. Recently I spend a lot of time in mill because it's very well suited to static-site generators, and because I've had better success getting mill builds to call into Scala 3 code, as some of my site-generation tools require. I've contributed to both projects.

There are a few hints I want to give myself for when I contribute to mill.

Build and run

The trick is just

$ ./mill -i installLocal

in the mill repository. If the build succeeds, the a mill-release executable appears in the target/ directory of the repository. One can test and play with that.

scalafmt

mill wants code contributions to pass a scalafmt check before merging. You build mill with mill of course, and mill makes this check easy.

To check formatting...

$ mill mill.scalalib.scalafmt.ScalafmtModule/checkFormatAll __.sources

To have mill go ahead and reformat your code...

$ mill mill.scalalib.scalafmt.ScalafmtModule/reformatAll __.sources

I have now twice, embarrassingly, forgotten to do this.

???

I'll probably update this entry in place over time, if I find more hints I want to keep.

2023-10-07

Referring to Scala collections


TL; DR: Prefer

import scala.collection.{immutable,mutable}

Over the eons, I developed a habit of using the following import

import scala.collection.*

to set up my use of Scala collections.

Then I can refer to collections in a very clear longhand. For example:

val alphamap = immutable.Map( "A" -> "Apple", "B" -> "Baby", "C" -> "Candy" )

or, much less frequently:

val scratchpad = mutable.Map( "Title" -> "Untitled" )

I like referring explicitly to collections as mutable.Thing or immutable.Thing. Yes it's more typing, but it's very clear.

But recently the practice has caused some hassles, and I've realized it's not great.

Consider this little scala-cli application:

//> using scala 3.3.1

trait Hat:
  def tickets : Set[String]

object HatApp:
  import scala.collection.*
  import scala.util.Random

  val hat = new Hat:
    def tickets : Set[String] = Set("Alice", "Bob", "Carol")

  @main  
  def winner = println( Random.shuffle(hat.tickets.toList).head )

Try running it, and oops!

Compiling project (Scala 3.3.1, JVM)
[error] ./unqualified-collections.scala:11:9
[error] error overriding method tickets in trait Hat of type => Set[String];
[error]   method tickets of type => collection.Set[String] has incompatible type
[error]     def tickets : Set[String] = Set("Alice", "Bob", "Carol")
[error]         ^
Error compiling project (Scala 3.3.1, JVM)

The issue is that Set is defined in four places:

  • unqualified in scala.Predef
  • as scala.collection.Set
  • as scala.collection.immutable.Set
  • as scala.collection.mutable.Set.

The convenient, simple, unqualified Set is, very sensibly, just a type alias for scala.collection.immutable.Set.

However, after I've imported scala.collection.*, unqualified Set now refers to scala.collection.Set, the base trait for both mutable and immutable sets, rather than the immutable trait it originally referred to.

A scala.collection.Set is not a subtype of scala.collection.immutable.Set or, equivalently, the predef's unqualified Set.

So even though my definition of a Hat looks trivially conformant, it is not.

It's easy to fix this just by using the longhand syntax I prefer

  val hat = new Hat:
    def tickets : immutable.Set[String] = immutable.Set("Alice", "Bob", "Carol")

But it's a bit surprising that a contract that was defined in terms of unqualified Set has to be implemented in terms of immutable.Set. If a trait was defined in terms of unqualified Set, it's more straightforward to just implement it in terms of that.

So lately I've taken to changing how I make collections available for myself. I still prefer to be able to refer to them in explicit, clear, longhand. But instead of

import scala.collection.*

I now prefer

import scala.collection.{immutable,mutable}

This way, I can refer to collections explicitly as immutable.Whatever or mutable.Whatever, but when I refer to unqualified collection names, I haven't shadowed the predef definitions with rarely desired ambidextrous trait definitions under scala.collection.

So now I have

//> using scala 3.3.1

trait Hat:
  def tickets : Set[String]

object HatApp:
  import scala.collection.{immutable,mutable}
  import scala.util.Random

  val hat = new Hat:
    def tickets : Set[String] = Set("Alice", "Bob", "Carol")

  @main  
  def winner = println( Random.shuffle(hat.tickets.toList).head )

and everything works as expected.

The import is superfluous, gratuitous, unnecessary in this case. Since the only collection I touch is the unqualified Set, I could just have omitted any import.

But collections are very common to use. In real applications, one very often wants to set up ones files for clear, easy access to them. I think this is a good syntax to use, that lets on both write out clear collection names and avoid surprising ambiguities.

2023-09-17

Taking control of podcasts via RSS


TL; DR: I did it, and like it!

But it was not so straightforward that anyone can just ditch their podcast app for an RSS reader and have a good experience.

screenshot of final merged RSS feed in a podcast app.


I’m an avid podcast listener, so the podcast subscription list I curate is important to me. My phone is an iPhone and my laptop a Mac, so initially I just used Apple’s Podcasts app. Podcasts were conveniently synced between my phone and computer. That was nice.

But I don’t have Apple CarPlay, and didn’t like the UI for finding podcasts when I was driving. I was doing more clicking around and searching on the phone than seemed prudent. So, I looked at other podcast apps, and was pretty happy with Castro for a while.

Apple — to its vendor-lock-in shame — no longer supports exports of podcasts to OPML from the Podcasts app. It used to, when Podcasts were part of iTunes. There's a workaround here that I haven’t tried. So migrating to Castro meant a lot of manual resubscribing.

Castro is a fine app. It is not evil — it supports OPML import and export. But it’s iPhone only, and I want access to my feeds on my computer too. And I just want to feel like I own my own darned subscription list, and podcasts are published as RSS feeds, so why can't I just subscribe to them with my RSS reader?

So I did!

I use Inoreader and I am, overall, a big fan. I feel a bit of cognitive dissonance over that — in general I am trying to disentangle myself from centralized platforms as an architecture, and Inoreader is a centralized platform. I could use a local app like NetNewsWire (my first RSS reader!), which now will sync between phone and computer.

But I’m subscribed to more than 1000 feeds, I wonder if that won’t be a lot, especially for my phone. I like Inoreader's “monitored keywords" feeds. In general I’ve been impressed by Inoreader. It feels like a power tool. They are not a dominant platform in their space, so they're less likely to enshittify than a monopolist. For now, I am happy to be a paying customer of theirs.

Migrating by OPML worked easily, both the Castro export and Inoreader import. I end up with a podcasts folder in Inoreader. I can browse that folder like any other collection of feeds, and play podcast audio files from each post. Great!

But there were two hitches:

Not a great sort of sort
Not a great sort of sort!
  1. Inoreader sorts new articles in reverse chronological order using the time your feed receives the article, rather than the time of the article's publication. So when you subscribe to a feed, a folder that contains it along with other feeds will show all of the new feed's posts at the top.

    I suspect this is performance motivated — Inoreader builds and caches your feeds in advance, and RSS article publication dates are neither reliable nor stable. If it tried to cache publication-sorted feeds, it would end up frequently, expensively reconstructing them.

    Nevertheless, the effect of this has always been a bit annoying. Nearly all my feeds are organized into folders. When I subscribe to a new blog, the top of that folder gets monopolized.

    For blogs, this is not a big deal. Blogs typically keep just a few posts in their RSS, maybe the last five or ten articles. And when reading, scrolling down is not a problem.

    Podcast "blogs" sometimes have tens of articles in their RSS. You have to scroll a lot farther down. And I often want to listen to podcasts while driving. I can’t afford a lot of messing around to get past an archive of my last subscription.

  2. While you can play episodes in Inoreader’s mobile app, it’s not a great podcast app. Playback takes over the screen of your phone, and stops if switch to anywhere else. It won’t remember where you were when you go back.

    I think there’s an opportunity for Inoreader to become a better audio playback app and become the podcast app for RSS lovers. But it isn’t there yet.

So.

I wanted to re-sort my feed by publication time rather than feed-saw-it time, and I wanted podcasts to end up in a richer audio app.

Inoreader power tools to the rescue! Inoreader lets you publish the folders you curate as new own RSS feeds. What if I subscribed to this one feed of feeds from a proper podcasts app?!

That works! You get a good listening experience, can listen even when you switch out of the app, can resume episodes where you left off.

But…

  1. It does not typically cause the feed to get sorted by publication date. Podcast apps use the ordering in the feed itself; and

  2. The feed contains a jumbled mix of many podcasts' episode titles, with no information about which podcast each episode is from.

It’s here that I start to get a bit obsessive.

I’ve done a fair amount of work serving and transforming RSS and generating podcast feeds. What if I let the RSS feed server that I’ve already built and deployed subscribe to my podcast feed, sort the episodes by publication time, and then re-serve them?

Since I have the RSS, I can just inject the podcast names into the episode titles, so my items in my feed look like “Left Anchor: Finland's Cooperative Culture”, where Left Anchor is the podcast name, and the rest is the episode title.

From SubscribedPodcasts.scala:

  private val PrefixTransformations = Map("Podcasts" -> "TAP")

  private def prependFeedTitleToItemTitles(rssElem: Elem): Elem =
    val feedPrefix =
      val queryResult = (rssElem \ "channel").map(_ \ "title")
      if queryResult.nonEmpty then
        val rawPrefix = queryResult.head.text.trim
        val goodPrefix = PrefixTransformations.getOrElse(rawPrefix, rawPrefix)
        (goodPrefix + ": ")
      else
        ""
    val rule = new RewriteRule:
      override def transform(n: Node): Seq[Node] = n match
        case elem: Elem if elem.label == "item" => prefixTitlesOfItemElem(feedPrefix, elem)
        case other => other
    val transform = new RuleTransformer(rule)
    transform(rssElem).asInstanceOf[Elem]

Since my RSS server is in the business of unifying feeds, I used another Inoreader power tool — serving OPML so you can subscribe to subscription lists! I had my app subscribe to my list of feeds, periodically refresh that from Inoreader, then load and and merge all the feeds itself. That way I can control how feeds are merged. (For example, I am very careful about preserving XML namespaces in merged feeds.)

From InterfluidityMain.scala

  val subscribedPodcastsMetaSources = immutable.Seq(
    MetaSource.OPML(URL("https://www.inoreader.com/reader/subscriptions/export/user/1005956602/label/Podcasts"), eachFeedTransformer = SubscribedPodcasts.bestAttemptEmbellish),
    MetaSource.OPML(URL("https://www.inoreader.com/reader/subscriptions/export/user/1005956602/label/Podcasts+HF"), eachFeedTransformer = SubscribedPodcasts.bestAttemptEmbellish),
  )

Now any podcast app that lets you subscribe via a simple RSS url (most, but not all of them!) can subscribe to my feed.

I was done!

But I was vain.

I didn’t like the look of my feed. There were no pretty cover graphics, just the text name of each feed. And my gigafeed itself had no cover image. So…

If I was transforming XML to modify titles, I might as well transform it to add images. Unless an episode has an episode-specific image defined (usually they don’t), I take the cover image of the feed and make it the cover image of the episode. Now, when you look at my all-my-podcasts feed in a podcast app that supports episode images, you see the cover image for the podcast next to each episode.

From SubscribedPodcasts.scala:

  private def copyItunesImageElementsToItems(rssElem: Elem): Elem =
    val mbItunesFeedImage =
      val queryResult = (rssElem \ "channel").flatMap(_ \ "image").filter(_.asInstanceOf[Elem].prefix == "itunes")
      if queryResult.nonEmpty then Some(queryResult.head) else None
    val mbRegularFeedImage =
      val queryResult = (rssElem \ "channel").flatMap(_ \ "image").filter(_.asInstanceOf[Elem].prefix == null)
      if queryResult.nonEmpty then Some(queryResult.head) else None
    val mbFeedImage = mbItunesFeedImage orElse mbRegularFeedImage.map: regularImageElem =>
      val url = (regularImageElem \ "url").head.text.trim
      Element.Itunes.Image(href = url).toElem
    mbFeedImage.fold(rssElem): feedImage =>
      val rule = new RewriteRule:
        override def transform(n: Node): Seq[Node] = n match
          case elem: Elem if elem.label == "item" =>
            if (elem \ "image").isEmpty then
              elem.copy(child = elem.child :+ feedImage.asInstanceOf[Elem])
            else
              elem
          case other => other
      val transform = new RuleTransformer(rule)
      transform(rssElem).asInstanceOf[Elem]

Messing around in midjourney, I "prompted” a cover image for my overall feed of feeds, and transformed the almost-final merged feed to include that image.

Now everything is very pretty. You can see what it looks like in Podcast Republic. (This is also the image at the top of the post.)

I noticed that some apps were undesirably segregating episodes based on alleged “seasons”, putting episode from the “latest” season near the top. Obviously, there can be no consistency of seasons, since I am taking episodes from a kaleidoscope of different shows. So, I add yet another transformation to feeds before merging them, one which strips any <itunes:season> elements.

The RewriteRule API of Scala’s standard XML library performs abysmally. I transform each feed three times (modify the title, add an episode image, strip seasons), and then I transform the final feed once (to insert my cover image).

I think I could, and should, combine the transformations into a single pass that performs all three per-feed, pre-merge transformations. But it's conceptually easier to just run three passes. Even though processing a single feed can take up to 10 seconds, my ZIO-based app trivially parallelizes the transformations. Plus, reloads/reconstructions of the megafeed happen only once every 30 minutes.

So, although I feel a bit of professional embarrassment over the very remediable poor performance of feed reconstruction, it has no practical cost, and I haven’t (yet) bothered to fix it. Updated feeds replace prior feeds atomically, so there's no downtime while a new feed is under construction.

Anyway, it was all a bit much, a bit more than I had bargained for when, almost on a whim, I set out to RSS-ify my podcast management.

But now it’s done. I manage and subscribe to podcasts in Inoreader. A bit omphaloskeptically, I resubscribe from Inoreader to the re-sort of those feeds performed by my server. I listen straight off of Inoreader on my laptop. On the phone, I bounce between several apps — mostly Podcast Republic and Podcast Guru — to listen to whatever I’m into. (I still like Castro, but I've left my old setup alone there, just in case.) Each app sees the same feed, synced to Inoreder. Information the apps themselves generate, like for how long an episode has been listened to or whether it’s already completed, does not get synced between apps. I don’t find that to be a problem.

Inoreader supports tags, and will export an RSS feed of posts with a given tag. I’ve created a tag called “Queued”, and I have my podcast apps subscribe to that too. So I can browse on my desktop, tag episodes I may be particularly interested in, and find those quickly in a second feed each podcast app subscribes to. In general, I subscribe to two feeds in each app, my gigafeed that merges all of my podcast subscriptions, and my Queued feed which offers just a few episodes that I’ve selected.

So far it's working pretty well!

A couple of quick miscellaneous tips:

  • podnews.net is a great resource for finding podcast RSS feeds. Just search by name, if necessary restricting to "The Podcast Index". When you find a podcast, you'll see a gazillion icons for apps and platforms, but the very last one will be the podcast's clean, beautiful, old-fashioned RSS feed.

  • I subscribe to some very high frequency podcasts, like "NPR News Now", which comes out each hour. A wonderful feature of Inoreader is it caches the full history of your feeds. But here this becomes a problem. To prevent the history of high frequency feeds from drowning out eveything in my Inoreader podcasts folder I segregate high-frequency feeds into a "Podcasts HF" folder. When I merge feeds, I draw on the OPML from this folder as well as from my main podcasts folder. NPR's actual feed always includes only the single most recent episode, so it doesn't overwhelm my merged feed, which loads the feed to merge from NPR, not Inoreader folder.

You can subscribe to my subscriptions if you want, the URL is https://www.interfluidity.com/unify-rss/subscribed-podcasts.rss.

2023-08-30

Suppressing bloop for scala-cli managed services


I am a huge fan of scala-cli.

Among its many virtues, it reproduces the ergonomics of an interpreted language for compiled, super typesafe, Scala. (Java too!)

Recently, I've written custom services that I execute using scala-cli run, and then deploy directly as systemd units.

Here's a snippet of how I used to do this:

...

[Service]
Type=simple
User=unify-rss
Group=unify-rss
WorkingDirectory=/home/unify-rss/development/gitproj/unify-rss
ExecStart=scala-cli run main
...

I noticed, however, where I expected one long-running java process for a service's dedicated user, I ended up with two! In addition to my own application, a service called bloop.Bloop was running as well.

JVMs are not exactly lightweight, and I don't want double the fun when running a service.

bloop is a Scala build service that many different editors, build systems, and other tools use to efficiently, incrementally, compile and run scala code. It runs as a persistent background process, which external tools tools can trigger with commands to build or execute the codebase, recompiling only what is necessary given what has changed.

This is great for fast iteration during development, but unwieldy (and potentially increases a threat surface area) during deployment of persistent services.

Fortunately, scala-cli, you can turn bloop off. scala-cli then falls back to the plain old Scala compiler to rebuild your application. It retains its core ergonomics: You can edit, then run, without any sort of compile / assemble / publish ceremony. The first run after a change might take a bit longer than it would have with bloop.

Here is a snippet (slightly simplified) of my current systemd unit.

...

[Service]
Type=simple
User=unify-rss
Group=unify-rss
WorkingDirectory=/home/unify-rss/development/gitproj/unify-rss
ExecStart=scala-cli run --server=false main
...

Note the --server=false argument to scala-cli run.

Executing the service works just the same as before, rebuilding if necessary. But now no bloop service squanders precious server-side memory.

My application is small enough that the additional build time is not an issue. The only visible difference when running scala-cli is that, when bloop is enabled and something has changed, I would see a message like

Compiling project (Scala 3.3.0, JVM)
Compiled project (Scala 3.3.0, JVM)

With the --server=false flag gone (it defaults to true), that message disappears.

But changes still recompile, and everything works great.


Note: the --server=false argument has to come after the run subcommand. Otherwise...

$ scala-cli --server=false run main
[error]  run is not a scala-cli sub-command and it is not a valid path to an input file or directory.
Try viewing the relevant help to see the list of available sub-commands and options.
  scala-cli --help

The error message is unhelpfully mistaken. run in fact is a valid scala-cli subcommand. But --server is not a valid command-line option to the base scala-cli command, it is a valid command-line option to scala-cli run.

$ scala-cli --version
Scala CLI version: 1.0.4
Scala version (default): 3.3.0
2023-08-23

Getting started with HedgeDoc


Motivation

I'm really trying to put together a synchronous collaboration environment, self-hosting as much as possible, using third-party platforms as little as possible. My first step was getting Jitsi up and running.

But conversations want notes. They sometimes become collaborations that should generate documents.

I have, with various interlocutors, used Zoom's chat session as a kind of weirdly structured notes environment. Zoom does support one essential feature, "Save Chat", so that the notes you've shared, disguised as remarks, can be retained. Jitsi does offer chat, but it's a bit unwieldy and lacks that essential save feature. Jitsi chat is really not pervertable to a notes application.

I've been looking for something better, and today I found HedgeDoc, which looks perfect, and pretty amazing! Give it a look. Really.

So, I decided to install HedgeDoc on my new collaboration server, the same Digitial Ocean droplet on which I just installed Jitsi.

(I don't think HedgeDoc will be super resource intensive, though I am a bit worried that adding bells and whistles might slow down the videoconferencing core on what already is virtualized and underpowered hardware.)

Trepidation

HedgeDoc is a node.js application. In general, I find node applications intimidating. Their builds seem sprawling with little JSON and YML and lock files I don't understand. They use JSON for human-edited config files, but JSON is persnicketty and allows no comments. Life is full of persnicketty config files, but when they are not JSON, they tend to be lavish with hints in comments and suggestions you can comment out or comment in. Some people argue (ht Bill Mill) that the cleanliness that results from a hard separation between configuration data and documentation is a feature not a bug. I myself am a bug, though, and prefer informative clutter to sterile helplessness.

Anyway, maybe some day I will become a Node-knower and have opinions about yarn vs npm and stuff. But that day has not yet come. In the meantime, I just run mysterious incantations from the installation docs and marvel at just how many dependencies seem to be flowing in.

Installation

I went for HedgeDoc's Manual Installation. My first step was to download node itself. The version that came in with apt install nodejs was too old, so I found some instructions on how to (easily, lazily) get node.js v16 onto my Ubuntu 22.04 server.

I took that slowly, downloading the script to a file before executing it to at least glance at what I'd be running as root.

I'm not a big fan of curl -sL https://whoknowswhat.com/script.sh | bash - style installs. But any half competent malicious script-kiddie could have gotten past my cursory inspection. Of course I end up treating most of the dependecies I download as pure black boxes. I hope that deb.nodesource.com is trustworthy!

I was also going to need a Postgres database. That was just apt install postgresql.

Once Postgres was installed I made a user hedgedoc, and could perform most of the build as that not-so-terribly powerful persona. hedgedoc was going to be the user under whose aegis the service would be run, so I needed it to have access to a database under Postgres. So, something like...

# su postgres
$ createuser hedgedoc
$ psql
postgres=# CREATE DATABASE hedgedoc;
postgres=# ALTER DATABASE hedgedoc OWNER TO hedgedoc;
postgres=# <ctrl-D to exit psql>
$ <ctrl-D to exit user postgres>
# su hedgedoc
$ psql
hedgedoc=> ALTER USER hedgedoc WITH PASSWORD 'not-actually-this';

That last bit actually came later, as I was trying to figure out why the app couldn't connect to the database. Even though under postgres' default config passwordless psql was fine, the node app did need a password to connect over localhost.

Anyway, in real life, setting up the password came later, but we might as well get it over with now.

OK. I love postgres. That was the easy stuff. Now the hard part. The node app.

The instructions ask us to set up some node dependencies with npm in order to get yarn to work. (They are frenemies I guess.)

# npm install -g node-gyp
# npm install -g npm install -g corepack
# corepack enable

Then, as user hedgedoc, we clone the application repository and check out tag 1.9.9, the latest release. Then, following the instructions, we run a setup script.

$ git clone https://github.com/hedgedoc/hedgedoc.git
$ git checkout 1.9.9
$ cd hedgedoc
$ bin/setup

Stuff happens.

When the script is done, we have a file called config.json (just a copy of repository file config.json.example).

I then did my best to configure the application. This was... hard. The configuration is very elaborate. I had to iterate through some snags to get it right. Plus, the example file is divided into three separate configs, one each for test, development, and production. I only edited production, but the existence of multiple configurations became the source of later confusion. (See below.)

Anyway, with a first-draft config in hand, we can complete the build instructions:

$ yarn install --immutable
$ yarn build

Hooray!

Consternation

HedgeDoc was built. The next step was getting it to work. This was an iterative process with a few hitches.

Hitch 1: Postgress Password

First, following the instructions, I run (as hedgedoc)

$ NODE_ENV=production yarn start

I get a lot of messages indicating that the app is failing to connect to the database. My database config at this point looked like

{
    "production": {
        "db": {
            "username": "hedgedoc",
            "password": "",
            "database": "hedgedoc",
            "host": "127.0.0.1",
            "port": "5432",
            "dialect": "postgres"
        }
    }
}    

Since I could authenticate to psql locally as hedgedoc without a password under the default authentication policy (see /etc/postgresql/14/main/pg_hba.conf), I wondered if passwordless would be okay. Nope.

So I gave hedgedoc a password (ALTER USER hedgedoc WITH PASSWORD 'not-actually-this';, see above), and retried NODE_ENV=production yarn start. It started up beautifully.

Hitch 2: Proxy through nginx

This wasn't a hitch really. It was planned and expected. But at this point, I still couldn't see hedgedoc, which was getting served on localhost only at port 3000. The HedgeDoc docs very helpfully include a sample nginx.conf for proxying the service.

Before I could do much with this, I had to decide at what URL I wanted the app to be served. I decided I'd make a separate virtual host for it, notes.interfluidity.com.

So, I updated my DNS for interfluidity with a CNAME record pointing back to loiter.interfluidity.com, then used certbot / Let's Encrypt to get certificates for the new name (briefly stopping nginx so I could authenticate with certbot's internal web service):

# systemctl stop nginx
# certbot certonly -d notes.interfluidity.com
# systemctl start nginx

Now I could copy the nginx proxy setup from the docs to /etc/nginx/sites-available/notes.interfluidity.com.conf, and edit it to fill in the new certificate locations. Initially this failed, because the reference config contained references to other ssl config I didn't have.

    include options-ssl-nginx.conf;
    ssl_dhparam ssl-dhparams.pem;

I replace the first line with SSH config I have from elsewhere, and just omit (as I usually do, is this bad?) the ssl_dhaparam directive.

I also needed to add a stanza to redirect traffic from port 80 (regular http) to https at port 443.

Ultimately, /etc/nginx/sites-available/notes.interfluidity.com.conf looks like this.

Hitch 3: Registering users hangs

This one befuddled me for a while. I don't want my HedgeDoc installation to be an open utility for anyone one the internet. So I have it configured to not permit web registation of new users. In config.json, at this point, I had

{
    "test": {
        "db": {
            "dialect": "sqlite",
            "storage": ":memory:"
        },
        "linkifyHeaderStyle": "gfm"
    },
    "development": {
        "loglevel": "debug",
        "db": {
            "dialect": "sqlite",
            "storage": "./db.hedgedoc.sqlite"
        },
        "domain": "localhost",
        "urlAddPort": true
    },
    "production": {
        ...
        "email": true,
        "allowEmailRegister": false,
        "allowAnonymous": false,
        "allowAnonymousEdits": true,
	...
    }
}    

Googling around, absent web registration of e-mails or using a third-party to authenticate, the way to set up new users is a utility bundled in the HedgeDoc distribution, bin/manage_user.

$ bin/manage_users 
You did not specify either --add or --del or --reset!

Command-line utility to create users for email-signin.

Usage: bin/manage_users [--pass password] (--add | --del) user-email
	Options:
		--add 	Add user with the specified user-email
		--del 	Delete user with specified user-email
		--reset Reset user password with specified user-email
		--pass	Use password from cmdline rather than prompting

But when I try

$ bin/manage_users --pass NotGonnaTellYou --add swaldman@mchange.com

I find that the script... just hangs.

I have to <ctrl-c> to kill it. It provides no messages or information.

Reviewing the script, its first few lines are

#!/usr/bin/env node                                                                                                                                                                      

// First configure the logger, so it does not spam the console                                                                                                                           
const logger = require('../lib/logger')
logger.transports.forEach((transport) => transport.level = 'warning')

The trick to making bin/manage_users informative was just commenting out the lines about the logger, so that it DOES spam the console.

#!/usr/bin/env node                                                                                                                                                                      

// First configure the logger, so it does not spam the console                                                                                                                           
//const logger = require('../lib/logger')
//logger.transports.forEach((transport) => transport.level = 'warning')

Messages then result, which renders our problem easy to diagnose. The script tries and fails to hit an sqlite database, because it was using the development rather than production config. (See the early config.json fragment above.)

To solve this, it's just...

$ NODE_ENV=production bin/manage_users --pass NotGonnaTellYou --add swaldman@mchange.com 

With the script's logger still commented out to encourage verbosity, this yields...

023-08-23T00:42:46.127Z warn: 	Neither 'domain' nor 'CMD_DOMAIN' is configured. This can cause issues with various components.
Hint: Make sure 'protocolUseSSL' and 'urlAddPort' or 'CMD_PROTOCOL_USESSL' and 'CMD_URL_ADDPORT' are configured properly.
2023-08-23T00:42:46.130Z warn: 	Session secret not set. Using random generated one. Please set `sessionSecret` in your config.json file. All users will be logged out.
2023-08-23T00:42:46.137Z error: 	uncaughtException: Dialect needs to be explicitly supplied as of v4.0.0

I configure

{
    ...
    "production": {
        "domain": "notes.interfluidity.com",
	...,
        "sessionSecret": "Not-Gonna-Tell-You-64-Chars-Long-Though-I-Don't-Think-That's-Obligatory",
	...
    }
}

Then

$ NODE_ENV=production bin/manage_users --pass NotGonnaTellYou --add swaldman@mchange.com 

succeeds.

2023-08-23T00:42:59.833Z debug: 	dmp worker process started
Using password from commandline...
Created user with email swaldman@mchange.com

But I should have paid more attention to that hint!

Hint: Make sure 'protocolUseSSL' and 'urlAddPort' or 'CMD_PROTOCOL_USESSL' and 'CMD_URL_ADDPORT' are configured properly.

Hitch 4: Login form won't submit

When I run it from the command line, the web application is now served at notes.interfluidity.com. I've succeeded at registering myself as a user. I hit the sign-in button, and a login form appears. but when I hit submit, nothing at all happens.

To get any information at all about this, I needed to go to the Javascript console of my browser.

There, only there, I could see log messages about the problem, noting security policy violations. The form was only allowed to hit the app that served it, the app that served it was https://notes.interfluidity.com/. But the form was trying to hit http://notes.interfluidity.com/.

Spot the difference? It's https vs http.

The solution was to add a config parameter...

{
    ...
    "production": {
	...,
        "protocolUseSSL": true,
	...
    }
}

Now I can login!

Consummation

At this point, everything basically works. After the problems introduced by accidentally hitting the development config, I decide to clean away the unused configurations (debug, development) from config.json. The final (well, current) version is below.

I want HedgeDoc to be a permanent, systemd managed service, so I copy and modify the unit file example the documentation helpfully supplies. I have to uncomment the After=postgresql.service line, modify the WorkingDirectory to point to the location of my HedgeDoc build (which is not in opt), and comment out ProtectHome=true (because my HedgeDoc build is in fact in user hedgedoc's home directory).

Then it's just the usual, a symlink to the unit file from /etc/systemd/system, and then...

# systemctl enable hedgedoc
# systemctl start hedgedoc

(It took a couple of rounds of editing the unit file to get the startup to succeed. Mostly I had to figure out to comment out ProtectHome=true.)

HedgeDoc offers a lot of integrations. One feature which I like is you can export your markdown documents into GitHub gists. I followed these instructions, and gist export worked with no trouble.

That's it! Everything seems golden!

Security considerations

HedgeDoc's documentation is admirably security conscious, and its defaults are unusually tight. I made a few choices that involved weakening those defaults:

  • I've configured csp.allowFraming to true. I intend for my "collaboration server" to take the form of a palette of multiple apps embedded on a webpage in iframes. Each frame would be maximizable, or you could view work with multiple frames at once. So I really want HedgeDoc to work from an iframe.

    • The docs "strongly recommend disabling this option, as it increases the attack surface of XSS attacks." (Their emphasis!) If users do choose to enable framing, the docs recommend serving over https and setting a cookiePolicy of none, which we do. So identification is by SSL session information, and there is no cookie to steal. I feel pretty okay with this, but what am I missing?
  • As discussed above, I don't set ssl_dhparam) in my SSH config. I've never encountered this before, but now I wonder whether I shouldn't get in the habit of setting this up across the sites I maintain. How much does it matter?

  • I often build services beneath the home directory of the user I create to run them. They are not run on machines on which users are likely to store sensitive personal information in home directories, but given the existence of the systemd ProtectHome restriction, perhaps it's a bad habit that I should revise?

I also remain concerned about the promiscuity of the node dependency ecosystem, and the possibility of supply chain attacks. I'm concerned about that across ecosystems: The JVM world in which I typically develop carries some of the same risks.

But the problem is arguably worse in the node ecosystem. I'm not running in docker containers or chroot or anything like that.

The Digital Ocean droplet I am running on is pretty low-stakes in terms of the information it stores, but it wouldn't be great if some third party gave themselves access and could join a botnet or spy on videoconferences or read all the meeting notes.

Running npm -g install as root, which I did do, feels partcularly ill-advised and dirty. I think I'll try to avoid that in the future.

Update 2023-10-20

I use my HedgeDoc's instance as a notes platform for weekly meetings, and I want the notes to be created in advance, so that people can suggest agenda items in advance. I want to automate this.

HedgeDoc does offer an API by which one can automate creation of new notes. So, the first thing I did was get comfortable using that by writing a script to create a new note.

However, I hit another snag. I want the notes to be freely editable upon creation (so that the people my automaton notifies about the new notes can add their agenda items). As far as I can tell, HedgeDoc's API does not offer any means of setting note permissions.

Fortunately, in HedgeDoc's config file, there is a defaultPermission key. Until now, I've not been using it, so the default defaultPermission defaulted to editable.

I've just inserted this key explicitly in the config file, with the value freely. It seems to work! New notes I make (via the API or in the app) are now freely editable by default, as I want.

It would be better if there were an API to reset permissions. I'd love to automate setting notes to locked after my meetings.


Appendix

final config.json

{
    "production": {
        "domain": "notes.interfluidity.com",
        "protocolUseSSL": true,
        "email": true,
        "allowEmailRegister": false,
        "host": "localhost",
        "allowAnonymous": false,
        "allowAnonymousEdits": true,
        "allowFreeURL": true,
        "loglevel": "info",
        "hsts": {
            "enable": true,
            "maxAgeSeconds": 31536000,
            "includeSubdomains": true,
            "preload": true
        },
        "csp": {
            "enable": true,
            "directives": {
            },
            "upgradeInsecureRequests": "auto",
            "addDefaults": true,
            "addDisqus": false,
            "addGoogleAnalytics": false,
            "allowFraming": true
        },
        "cookiePolicy": "none",
        "db": {
            "username": "hedgedoc",
            "password": "Not-Gonna-Tell-You",
            "database": "hedgedoc",
            "host": "127.0.0.1",
            "port": "5432",
            "dialect": "postgres"
        },
        "sessionSecret": "Not-Gonna-Tell-You",
        "github": {
            "clientID": "Not-Gonna-Tell-You",
            "clientSecret": "Not-Gonna-Tell-You"
        }
    }
}

nginx.conf

# modified from https://docs.hedgedoc.org/guides/reverse-proxy/

map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
}
server {
    listen 80;
    listen [::]:80;
    server_name notes.interfluidity.com;

    location / {
        return 301 https://$host$request_uri;
    }
}
server {
        server_name notes.interfluidity.com;

        location / {
                proxy_pass http://127.0.0.1:3000;
                proxy_set_header Host $host; 
                proxy_set_header X-Real-IP $remote_addr; 
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
                proxy_set_header X-Forwarded-Proto $scheme;
        }

        location /socket.io/ {
                proxy_pass http://127.0.0.1:3000;
                proxy_set_header Host $host; 
                proxy_set_header X-Real-IP $remote_addr; 
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
        }

    listen [::]:443 ssl http2;
    listen 443 ssl http2;
    ssl_certificate /etc/letsencrypt/live/notes.interfluidity.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/notes.interfluidity.com/privkey.pem;

    # Mozilla Guideline v5.4, nginx 1.17.7, OpenSSL 1.1.1d, intermediate configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:10m;  # about 40000 sessions
    ssl_session_tickets off;
}