2025-01-14 ⇒

Syndicating RSS to Mastodon and BlueSky with feedletter

feedletter is a service I wrote mostly to convert RSS newsfeeds into e-mail newsletters.

If you subscribe to this blog by e-mail, or to drafts.interfluidity.com, feedletter is the service to which you subscribed, and which mails you my blogposts.

I wrote an elaborate tutorial documenting how to set up newsletter subscriptions about a year ago.

But feedletter can also syndicate RSS announcements (or posts, length permitting) to Mastodon and BlueSky.

I thought I'd do a quick tutorial on how it works.

Table of contents

  1. Prerequisites
  2. Create user feedletter
  3. Install feedletter
  4. Prepare the postgres database
  5. Create the /etc/feedletter directory
  6. Create a Mastodon app password
  7. Create /etc/feedletter/feedletter-secrets.properties and configure Mastodon access
  8. Configure BlueSky access
  9. Add c3p0 (database) configuration to /etc/feedletter/feedletter-secrets.properties
  10. Add a "secret salt" to /etc/feedletter/feedletter-secrets.properties
  11. Check out the feedletter app
  12. Initialize the feedletter database
  13. Add a feed you wish to syndicate
  14. Define your subscribables
  15. Subscribe the target Mastodon and BlueSky accounts to the subscribables.
  16. Test in the terminal
  17. Enable feedletter as a systemd daemon
  18. Conclusion

1. Prerequisites

You'll need a systemd-based server on the internet, git, postgres, and a Java 17+ JVM installed.

2. Create user feedletter

We'll create a passwordless user:

# adduser --disabled-password feedletter

3. Install feedletter

We just clone a git repository. We'll make it become /opt/feedletter, and make sure our new user feedletter owns it:

# cd /opt
# git clone https://github.com/swaldman/feedletter-install.git feedletter
Cloning into 'feedletter'...
remote: Enumerating objects: 81, done.
remote: Counting objects: 100% (81/81), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 81 (delta 40), reused 64 (delta 23), pack-reused 0 (from 0)
Receiving objects: 100% (81/81), 12.32 KiB | 4.10 MiB/s, done.
Resolving deltas: 100% (40/40), done.
# chown -R feedletter:feedletter feedletter

4. Prepare the postgres database

# su - postgres
$ createdb feedletter
$ createuser feedletter
$ psql
psql (16.6 (Ubuntu 16.6-0ubuntu0.24.10.1))
Type "help" for help.

postgres=# ALTER DATABASE feedletter OWNER TO feedletter;
ALTER DATABASE
postgres=# ALTER USER feedletter WITH PASSWORD 'not-actually-this';
ALTER ROLE
postgres=# \q
$ exit

5. Create the /etc/feedletter directory

# mkdir /etc/feedletter
# chown feedletter:feedletter /etc/feedletter

6. Create a Mastodon app password, and place it in /etc/feedletter/feedletter-secrets.properties

In the Mastodon account to which you intend to syndicate announcements, go to Settings > Development, and click the "New Application" button.

Define an application name. We'll use "feedletter-syndication".

Be sure to enable write permissions!

Finally click the Submit button.

You should see your new application listed now in the applications page. Click on it. On the resulting page, note the field "Your access token". You'll need its value in the next step.

7. Create /etc/feedletter/feedletter-secrets.properties and configure Mastodon access

We'll first become user feedletter, then open a text editor on /etc/feedletter/feedletter-secrets.properties.

# su - feedletter
$ touch /etc/feedletter/feedletter-secrets.properties
$ chmod 600 /etc/feedletter/feedletter-secrets.properties
$ emacs /etc/feedletter/feedletter-secrets.properties

feedletter will insist that /etc/feedletter/feedletter-secrets.properties has restrictive permissions.

It will refuse to run if you skip the chmod bit.

$ chmod 600 /etc/feedletter/feedletter-secrets.properties`

Now we'll add an entry to our properties file in the form

feedletter.masto.access.token.<arbitrary-name>=<access-token>`:

Just to be dumb, we'll use "arbitrary-name" as our, um, arbitrary name.

feedletter.masto.access.token.arbitrary-name=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Your access token probably is not all XXXXXXXXXX!

8. Configure BlueSky access

In the BlueSky account you intend to syndicate to, go to Settings > Privacy and security > App passwords.

Click "Add App Password".

Give your app a name. We'll use "feedletter-syndication" again.

Your app password will appear.

Add a new line to /etc/feedletter/feedletter-secrets.properties, in the form feedletter.bsky.identifier.<identifier>=<app-password>.

This is a bit different from Mastodon!

In Mastodon, a totally arbitrary name went at the end of the key. Here it is the user identifier, which usually a DNS handle (but it can also be a atproto did).

The account we're syndicating to will be testable-bsky.bsky.social

So /etc/feedletter/feedletter-secrets.properties should look something like this, albeit with a real access tokens / passwords.

feedletter.masto.access.token.arbitrary-name=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
feedletter.bsky.identifier.testable-bsky.bsky.social=xxxx-xxxx-xxxx-xxxx

9. Add c3p0 (database) configuration to /etc/feedletter/feedletter-secrets.properties

Remember when we set the Postgres password a while back? We'll use it now.

Postgres by default runs on port 5432, so let's add some database access and authentication information.

feedletter uses the c3p0 connection pooling library.

Edit /etc/feedletter/feedletter-secrets.properties so that it looks something like this:

feedletter.masto.access.token.arbitrary-name=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
feedletter.bsky.identifier.testable-bsky.bsky.social=xxxx-xxxx-xxxx-xxxx
c3p0.jdbcUrl=jdbc:postgresql://localhost:5432/feedletter
c3p0.user=feedletter
c3p0.password=not-actually-this

10. Add a "secret salt" to /etc/feedletter/feedletter-secrets.properties

/etc/feedletter/feedletter-secrets.properties should contain a "secret salt", whose value is entirely arbitrary but should be kept secret. We'll add that, so /etc/feedletter/feedletter-secrets.properties now looks like:

feedletter.masto.access.token.arbitrary-name=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
feedletter.bsky.identifier.testable-bsky.bsky.social=xxxx-xxxx-xxxx-xxxx
c3p0.jdbcUrl=jdbc:postgresql://localhost:5432/feedletter
c3p0.user=feedletter
c3p0.password=not-actually-this
feedletter.secret.salt=This is an arbitrary string.

11. Check out the feedletter app

As user feedletter, cd into /opt/feedletter and type

$ ./feedletter --help

The first time you do this, it will download a lot of stuff!

Eventually you will see a help message, like this:

Usage:
     feedletter [--secrets <propsfile>] add-feed
     feedletter [--secrets <propsfile>] alter-feed
     feedletter [--secrets <propsfile>] daemon
     feedletter [--secrets <propsfile>] db-dump
     feedletter [--secrets <propsfile>] db-init
     feedletter [--secrets <propsfile>] db-migrate
     feedletter [--secrets <propsfile>] define-bluesky-subscribable
     feedletter [--secrets <propsfile>] define-email-subscribable
     feedletter [--secrets <propsfile>] define-mastodon-subscribable
     feedletter [--secrets <propsfile>] drop-feed-and-subscribables
     feedletter [--secrets <propsfile>] drop-subscribable
     feedletter [--secrets <propsfile>] edit-subscribable
     feedletter [--secrets <propsfile>] export-subscribers
     feedletter [--secrets <propsfile>] list-config
     feedletter [--secrets <propsfile>] list-feeds
     feedletter [--secrets <propsfile>] list-items-excluded
     feedletter [--secrets <propsfile>] list-subscribables
     feedletter [--secrets <propsfile>] list-subscribers
     feedletter [--secrets <propsfile>] list-untemplates
     feedletter [--secrets <propsfile>] send-test-email
     feedletter [--secrets <propsfile>] set-config
     feedletter [--secrets <propsfile>] set-extra-params
     feedletter [--secrets <propsfile>] set-untemplates
     feedletter [--secrets <propsfile>] subscribe
 
 Manage e-mail subscriptions to and notifications from RSS feeds.
 
 Options and flags:
     --help
         Display this help text.
     --secrets <propsfile>
         Path to properties file containing SMTP, postgres, c3p0, and other configuration details.
 
 Environment Variables:
     FEEDLETTER_SECRETS=<path>
         Path to properties file containing SMTP, postgres, c3p0, and other configuration details.
 
 Subcommands:
     add-feed
         Add a new feed from which mail or notifications may be generated.
     alter-feed
         Alter the timings of an already-defined feed.
     daemon
         Run daemon that watches feeds and sends notifications.
     db-dump
         Dump a backup of the database into a configured directory.
     db-init
         Initialize the database schema.
     db-migrate
         Migrate to the latest version of the database schema.
     define-bluesky-subscribable
         Define a BlueSky subscribable, a source from which BlueSky feeds can receive automatic posts..
     define-email-subscribable
         Define a new email subscribable, a mailing lost to which users can subscribe.
     define-mastodon-subscribable
         Define a Mastodon subscribable, a source from which Mastodon feeds can receive automatic posts..
     drop-feed-and-subscribables
         Removes a feed, along with any subscribables defined upon it, from the service.
     drop-subscribable
         Removes a subscribable from the service.
     edit-subscribable
         Edit an already-defined subscribable.
     export-subscribers
         Dump subscriber information for a subscribable in CSV format.
     list-config
         List all configuration parameters.
     list-feeds
         List all feeds the application is watching.
     list-items-excluded
         List items excluded from generating notifications.
     list-subscribables
         List all subscribables.
     list-subscribers
         List all subscribers to a subscribable.
     list-untemplates
         List available untemplates.
     send-test-email
         Send a brief email to test your SMTP configuration.
     set-config
         Set configuration parameters.
     set-extra-params
         Add, update, or remove extra params you may define to affect rendering of notifications and messages.
     set-untemplates
         Update the untemplates used to render subscriptions.
     subscribe
         Subscribe to a subscribable.

This might be surrounded by some noise from the mill process used to launch it. It might end with Subprocess failed.

That's all fine.

12. Initialize the feedletter database

As user feedletter, from within /opt/feedletter, type

$ ./feedletter db-init

That's all!

13. Add a feed you wish to syndicate

As user feedletter, from within /opt/feedletter, type

$ ./feedletter add-feed --help

You should see

 Usage:
     feedletter add-feed --ping <feed-url>
     feedletter add-feed [--min-delay-minutes <minutes>] [--await-stabilization-minutes <minutes>] [--max-delay-minutes <minutes>] [--recheck-every-minutes <minutes>] <feed-url>
 
 Add a new feed from which mail or notifications may be generated.
 
 Options and flags:
     --help
         Display this help text.
     --ping
         Check feed as often as possible, notify as soon as possible, regardless of (in)stability.
     --min-delay-minutes <minutes>
         Minimum wait (in miunutes) before a newly encountered item can be notified.
     --await-stabilization-minutes <minutes>
         Period (in minutes) over which an item should not have changed before it is considered stable and can be notified.
     --max-delay-minutes <minutes>
         Notwithstanding other settings, maximum period past which an item should be notified, regardless of its stability.
     --recheck-every-minutes <minutes>
         Delay between refreshes of feeds, and redetermining items' availability for notification.

feedletter is general very cautious about announcing / e-mailing posts. Since posts are often edited soon after they are published, by default it waits a period of time, and then checks to make sure the post has been stable a while before announcing or e-mailing. By default min-delay-minutes is 30, await-stabilization-minutes is 15, max-delay-minutes is 180, and recheck-every-minutes is 10.

It's probably best to use cautious settings like this, so you don't announce your links while you are still re-editing them. (I don't think I'm the only writer who inevitably finds important edits just after hitting publish.)

For this tutorial, I'm going to use a feed of all the articles in my newsreader, inoreader. Its URL looks like

https://www.inoreader.com/stream/user/0000000000/tag/all-articles,

although 0000000000 is not in fact my user id, I'm altering that for privacy. You should use whatever feed you are interested in syndicating.

To use the defaults, you could just type

$ ./feedletter add-feed https://www.inoreader.com/stream/user/0000000000/tag/all-articles

For the purposes of our little experiment here, I want articles announced as soon as feedletter sees them, even if they have not "stabilized". For that, it will just be

$ ./feedletter add-feed --ping https://www.inoreader.com/stream/user/0000000000/tag/all-articles

You should see something like

+---------+-------------------------------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+
¦ Feed ID ¦ Feed URL                                                          ¦ Min Delay Mins ¦ Await Stabilization Mins ¦ Max Delay Mins ¦ Recheck Every Mins ¦ Added                       ¦ Last Assigned               ¦
+---------+-------------------------------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+
¦ 1       ¦ https://www.inoreader.com/stream/user/0000000000/tag/all-articles ¦ 0              ¦ 0                        ¦ 0              ¦ 0                  ¦ 2025-01-11T05:00:47.128363Z ¦ 2025-01-11T05:00:47.128363Z ¦
+---------+-------------------------------------------------------------------+----------------+--------------------------+----------------+--------------------+-----------------------------+-----------------------------+

The numbers will be different if you (wisely!) don't use the --ping flag.

Using --ping will cause the feed to be checked very frequently, currently about once a minute. Some servers may grow cross! You may be blocked or rate-limited!

You can use ./feedletter alter-feed to set a gentler recheck-every-minutes.

14. Define your subscribables

feedletter is a layered application. First you tell it what feeds to watch, then you define different kinds of "subscribables" — e.g. email, Mastodon, BlueSky — to each of which multiple recipients can subscribe in order to receive notifications.

Typically an e-mail subscribable will have many — 100s, 1000s — of destinations.

BlueSky and Mastodon "subscribable" often have just one subscriber, the accounts to which entries will be syndicated. You can attach as many subscribers as you want, though, if you want to post announcements to many accounts.

Let's start with our Mastodon subscribable:

$ ./feedletter define-mastodon-subscribable --help

You'll see something like

 Usage: feedletter define-mastodon-subscribable --feed-id <feed-id> --name <name> [--extra-param <key:value>]...
 
 Define a Mastodon subscribable, a source from which Mastodon feeds can receive automatic posts..
 
 Options and flags:
     --help
         Display this help text.
     --feed-id <feed-id>
         The ID of the RSS feed to be watched.
     --name <name>
         A name for the new subscribable.
     --extra-param <key:value>
         An extra parameter your notification renderers might use.

We saw the feed-id in the output to ./feedletter add-feed (and if we forget it, we can always run ./feedletter list-feeds).

The subscribable name can be anything we want, but by default it will appear in announcements. I'm going to call this subscribable stevefeeds-masto:

$ ./feedletter define-mastodon-subscribable --feed-id=1 --name=stevefeeds-masto
 
 -*-*-*-
 
 Subscribable Name:    stevefeeds-masto
 Feed ID:              1
 Subscription Manager: {
     "extraParams": {},
     "type": "Mastodon.Announce",
     "version": 1
 }
 A Mastodon subscribable to feed with ID '1' named 'stevefeeds-masto' has been created.

Let's do the same, and make a BlueSky subscribable:

$ ./feedletter define-bluesky-subscribable --feed-id=1 --name=stevefeeds-bsky

 -*-*-*-
 
 Subscribable Name:    stevefeeds-bsky
 Feed ID:              1
 Subscription Manager: {
     "extraParams": {},
     "type": "BlueSky.Announce",
     "version": 1
 }
 A BlueSky subscribable to feed with ID '1' named 'stevefeeds-bsky' has been created.

15. Subscribe the target Mastodon and BlueSky accounts to the subscribables.

Let's check out the subscribe subcommand.

$ ./feedletter subscribe --help

 Usage:
     feedletter subscribe --subscribable-name <name> --e-mail <address> [--display-name <name>] [--unconfirmed]
     feedletter subscribe --subscribable-name <name> --sms <number> [--unconfirmed]
     feedletter subscribe --subscribable-name <name> --masto-instance-name <name> --masto-instance-url <url> [--unconfirmed]
     feedletter subscribe --subscribable-name <name> --bsky-identifier <identifier> [--bsky-entryway-url <name>] [--unconfirmed]
 
 Subscribe to a subscribable.
 
 Options and flags:
     --help
         Display this help text.
     --subscribable-name <name>
         The name of an already-defined subscribable.
     --e-mail <address>
         The e-mail address to subscribe.
     --display-name <name>
         A display name to wrap around the e-mail address.
     --sms <number>
         The number to which messages should be sent.
     --masto-instance-name <name>
         A private name for this Mastodon instance.
     --masto-instance-url <url>
         The URL of the Mastodon instance
     --bsky-identifier <identifier>
         An account identifier, usually a DNS name or value beginning with 'did:'.
     --bsky-entryway-url <name>
         The base URL of the bluesky service.
     --unconfirmed
         Mark the subscription unconfirmed.

Recall that when we placed our Mastodon access token in feedletter-secrets.properties, we gave it a name. We chose "arbitrary-name", because it's lovely. That becomes our masto-instance-name. The URL of the instance that hosts our accounts will be masto-instance-url. So we just...

./feedletter subscribe --subscribable-name stevefeeds-masto --masto-instance-name arbitrary-name --masto-instance-url https://mas.to/

For BlueSky, at least for now, we don't really need to provide a URL, there's just one. (In the future, hopefully there will be more BlueSky-like atproto services, for which you would provide and entryway-url). All you need to provide is bsky-identifier, the DNS handle or did associated with your BlueSky account.

./feedletter subscribe --subscribable-name stevefeeds-bsky --bsky-identifier testable-bsky.bsky.social

16. Test in the terminal

Finally, all of our setup is done!

We can get feedletter checking out our feed and syndicating announcements just by running what's usually the daemon process in the terminal.

$ ./feedletter daemon

Jan 13, 2025 5:33:34 PM com.mchange.v2.log.MLog 
INFO: MLog clients using java 1.4+ standard logging.
2025-01-13@17:33:36 [INFO] [com.mchange.feedletter.Daemon] feedletter-0.1.0-13-60e19c daemon (re)starting. 
2025-01-13@17:33:36 [INFO] [com.mchange.feedletter.Daemon] Spawning daemon fibers. 
2025-01-13@17:33:37 [INFO] [com.mchange.feedletter.Daemon] Starting web API service on interface '127.0.0.1', port 8024. 
timestamp=2025-01-13T17:33:37.596959114Z level=INFO thread=#zio-fiber-1822226262 message="Starting the server..." location=com.mchange.feedletter.Daemon.webDaemon file=Daemon.scala line=189
timestamp=2025-01-13T17:33:37.714089350Z level=INFO thread=#zio-fiber-1822226262 message="Server started" location=com.mchange.feedletter.Daemon.webDaemon file=Daemon.scala line=189
2025-01-13@17:33:50 [FINE] [com.mchange.feedletter.db.PgDatabase] Deleting any as-yet-unassigned items that have been deleted from feed with ID 1 
2025-01-13@17:33:50 [INFO] [com.mchange.feedletter.db.PgDatabase] Updated/assigned all items from feed with ID 1, feed URL 'https://www.inoreader.com/stream/user/0000000000/tag/all-articles' 
2025-01-13@17:33:59 [FINE] [com.mchange.feedletter.db.PgDatabase] Deleting any as-yet-unassigned items that have been deleted from feed with ID 1 
2025-01-13@17:33:59 [INFO] [com.mchange.feedletter.db.PgDatabase] Updated/assigned all items from feed with ID 1, feed URL 'https://www.inoreader.com/stream/user/0000000000/tag/all-articles' 
2025-01-13@17:34:03 [FINE] [com.mchange.feedletter.db.PgDatabase] Deleting any as-yet-unassigned items that have been deleted from feed with ID 1

Great! This is what it looks like when the daemon is watch feeds. (In this case, just one feed.)

Eventually, when a new item is posted, you'll see entries like

2025-01-13@17:34:42 [FINE] [com.mchange.feedletter.db.PgDatabase] Added new item, feed ID 1, guid 'http://www.inoreader.com/article/3a9c6e7789e019a2'. 
2025-01-13@17:34:42 [FINE] [com.mchange.feedletter.db.PgDatabase] Item with GUID 'http://www.inoreader.com/article/3a9c6e7789e019a2' from feed with ID 1 has been assigned in subscribable 'stevefeeds-bsky' with assignable identifier 'http://www.inoreader.com/article/3a9c6e7789e019a2'. 
2025-01-13@17:34:42 [FINE] [com.mchange.feedletter.db.PgDatabase] Item with GUID 'http://www.inoreader.com/article/3a9c6e7789e019a2' from feed with ID 1 has been assigned in subscribable 'stevefeeds-masto' with assignable identifier 'http://www.inoreader.com/article/3a9c6e7789e019a2'. 
2025-01-13@17:34:42 [FINE] [com.mchange.feedletter.db.PgDatabase] Item with GUID 'http://www.inoreader.com/article/3a9c6e7789e019a2' from feed with ID 1 has been assigned (or refused assignment) in all subscribables to that feed.

In English, a new item has been found, and it's been "assigned" to the various subscribables watching the feed, here one that distributes to mastodon, and another that distributes to Mastodon and another that distributes to BlueSky. (You can also make various forms of e-mail subscribables!)

Some subscribables collect many posts before distributing them. (For example, a weekly digest e-mail.)

But for Mastodon and BlueSky post announcements, each post should be distributed immediately following its observation. So soon after we see an add message, we'll see messages like...

2025-01-13@17:34:42 [FINE] [com.mchange.feedletter.db.PgDatabase] Queued Mastodon post for distribution. Content: [stevefeeds-masto] New Post: Thousands sign petition calling on ad titan WPP to rethink its 4-day RTO demand, by pthompson@insider.com (Polly Thompson) https://www.businessinsider.com/wpp-return-to-office-thousands-sign-petition-workers-2025-1 
2025-01-13@17:34:42 [FINE] [com.mchange.feedletter.db.PgDatabase] Assignable (item collection) defined by subscribable name 'stevefeeds-masto', within-type-id 'http://www.inoreader.com/article/3a9c6e7789e019b1' has been deleted. 
2025-01-13@17:34:42 [FINE] [com.mchange.feedletter.db.PgDatabase] Cached values of items fully distributed have been cleared. 
2025-01-13@17:34:42 [INFO] [com.mchange.feedletter.db.PgDatabase] Completed assignable 'http://www.inoreader.com/article/3a9c6e7789e019b1' with subscribable 'stevefeeds-masto'. 
2025-01-13@17:34:42 [INFO] [com.mchange.feedletter.db.PgDatabase] Cleaned away data associated with completed assignable 'http://www.inoreader.com/article/3a9c6e7789e019b1' in subscribable 'stevefeeds-masto'.

After "completion", all data about the post is deleted, except the RSS guid and link are retained to prevent reannouncing the same post should it reappear in the RSS.

Once queued for distribution, you will see messages about postings to Mastodon and BlueSky.

Since on very active feeds, you can get a lot of posts at once when they are republished, feedletter spaces postings to a single account by one minute (currently hardcoded, eventually configurable), though it sends its posting streams to all subscribed accounts in parallel.

So you'll see stuff like this:

2025-01-13@17:52:48 [INFO] [com.mchange.feedletter.db.PgDatabase] Posted BlueSky notification to (https://bsky.social/,testable-bsky.bsky.social) 
2025-01-13@17:53:13 [FINE] [com.mchange.feedletter.db.PgDatabase] Deleting any as-yet-unassigned items that have been deleted from feed with ID 1 
2025-01-13@17:53:13 [INFO] [com.mchange.feedletter.db.PgDatabase] Updated/assigned all items from feed with ID 1, feed URL 'https://www.inoreader.com/stream/user/0000000000/tag/all-articles' 
2025-01-13@17:53:15 [INFO] [com.mchange.feedletter.db.PgDatabase] Posted Mastodon notification to (https://mas.to/,arbitrary-name)

And we see posts!

Sometimes things do go wrong!

2025-01-13@17:56:49 [WARNING] [com.mchange.feedletter.db.PgDatabase] Failed attempt to post to Bluesky destination 'https://bsky.social/', retried = 0 (maxRetries: 10) 
requests.RequestFailedException: Request to https://bsky.social/xrpc/com.atproto.repo.createRecord failed with status code 400
{"error":"InvalidRequest","message":"Invalid app.bsky.feed.post record: Record/text must not be longer than 300 graphemes"}
	at requests.Requester$$anon$1.readBytesThrough(Requester.scala:360)
	at geny.Readable.writeBytesTo(Writable.scala:93)
	at geny.Readable.writeBytesTo$(Writable.scala:91)
        ...
2025-01-13@17:56:49 [WARNING] [com.mchange.feedletter.db.PgDatabase] Attempt to post BlueSky notification to (https://bsky.social/,testable-bsky.bsky.social) failed or was skipped. Postable: BskyPostable(6,[stevefeeds-bsky] New Post: In DISSENT's winter issue, I make the case for left-YIMBYism in a debate with Brian Callaci and Sandeep Vaheesan https://www.dissentmagazine.org/article/supply-and-the-housing-crisis-a-debate/, by @resnikoff.bsky.social - Ned Resnikoff https://bsky.app/profile/resnikoff.bsky.social/post/3lfnb24zzbc2c,https://bsky.social/,testable-bsky.bsky.social,0,List())

feedletter will retry failed postings up to 10 times by default (the retry count is reconfigurable), then give up. This issue — the announcement is just too long to fit in BlueSky's 300 character limit — will not be cured, so that's what will happen. (I'll probably modify the BlueSky subscribable to just reject too-long posts, but for now they are just tried, retried, and eventually dropped.)

17. Enable feedletter as a systemd daemon

We just place a systemd unit file /opt/feedletter, and then symlink it from /etc/systemd/system.

[Unit]
Description=Feedletter RSS-To-Mail-Etc Service
After=syslog.target network.target

[Service]
Type=forking
PIDFile=/opt/feedletter/feedletter.pid
User=feedletter
Group=feedletter
WorkingDirectory=/opt/feedletter

#
# feedletter daemon behaves as a traditional forking service
# when given the --fork flag
#
# its launcher, mill, will fork a feedletter process,
# write a PID file for that process in the base working directory
# and then exit
#
ExecStart=/opt/feedletter/feedletter daemon --fork

TimeoutStopSec=90
Restart=on-failure
RestartSec=10s
StandardError=journal
StandardOutput=journal
StandardInput=null

[Install]
WantedBy=multi-user.target

Then it's just

# systemctl enable feedletter
# systemctl start feedletter

We verify that the service is running.

# journalctl -u feedletter --follow
Jan 13 20:15:55 temp-feedletter systemd[1]: feedletter.service: Consumed 18.609s CPU time, 222M memory peak.
Jan 13 20:15:55 temp-feedletter systemd[1]: Starting feedletter.service - Feedletter RSS-To-Mail-Etc Service...
Jan 13 20:15:56 temp-feedletter feedletter[70034]: =========================================== runMainDaemon PreMain daemon --fork ======================================
Jan 13 20:15:56 temp-feedletter feedletter[70034]: ======================================================================================================================
Jan 13 20:15:59 temp-feedletter feedletter[70034]: [62/62] =================================== runMainDaemon PreMain daemon --fork =================================== 3s
Jan 13 20:15:59 temp-feedletter feedletter[70034]: ======================================================================================================================
Jan 13 20:16:00 temp-feedletter systemd[1]: Started feedletter.service - Feedletter RSS-To-Mail-Etc Service.
Jan 13 20:16:01 temp-feedletter feedletter[70086]: Jan 13, 2025 8:16:01 PM com.mchange.v2.log.MLog
Jan 13 20:16:01 temp-feedletter feedletter[70086]: INFO: MLog clients using java 1.4+ standard logging.
Jan 13 20:16:03 temp-feedletter feedletter[70086]: 2025-01-13@20:16:03 [INFO] [com.mchange.feedletter.Daemon] feedletter-0.1.0-13-60e19c daemon (re)starting.
Jan 13 20:16:03 temp-feedletter feedletter[70086]: 2025-01-13@20:16:03 [INFO] [com.mchange.feedletter.Daemon] Spawning daemon fibers.
Jan 13 20:16:03 temp-feedletter feedletter[70086]: 2025-01-13@20:16:03 [INFO] [com.mchange.feedletter.Daemon] Starting web API service on interface '127.0.0.1', port 8024.
Jan 13 20:16:04 temp-feedletter feedletter[70086]: timestamp=2025-01-13T20:16:04.337801942Z level=INFO thread=#zio-fiber-614230192 message="Starting the server..." location=com.mchange.feedletter.Daemon.webDaemon file=Daemon.scala line=189
Jan 13 20:16:04 temp-feedletter feedletter[70086]: timestamp=2025-01-13T20:16:04.470643106Z level=INFO thread=#zio-fiber-614230192 message="Server started" location=com.mchange.feedletter.Daemon.webDaemon file=Daemon.scala line=189
Jan 13 20:16:41 temp-feedletter feedletter[70086]: 2025-01-13@20:16:41 [FINE] [com.mchange.feedletter.db.PgDatabase] Deleting any as-yet-unassigned items that have been deleted from feed with ID 1
Jan 13 20:16:41 temp-feedletter feedletter[70086]: 2025-01-13@20:16:41 [INFO] [com.mchange.feedletter.db.PgDatabase] Updated/assigned all items from feed with ID 1, feed URL 'https://www.inoreader.com/stream/user/0000000000/tag/all-articles'

Looks Good!

18. Conclusion

That's it!

It seems like a lot, but it's much easier done than said. Give it a try. It won't bite.

It is also possible to customize the announcement messages that feedletter syndicates, or to include post text, length permitting. If that would be useful to you, Let me know and I'll add a quick tutorial on that.

2025-01-06 ⇒

Supporting all-item RSS

In addition to single-item RSS for each post, this blog now offers all-item-rss. Check it out.. It's effectively a full-content archive of the site, in the form of easy-to-parse RSS.

drafts.interfluidity.com supports both of these features as well. (Here's the all-items-feed, an example single-item-feed.)

The channel element of these feeds includes metadata describing them. iffy:curation describes how many and what sort of items are included in the feed. iffy:completeness describes how complete each item is, i.e. whether it include full content, or even potentially embeds media.

Here's an example from an all-item feed:

<?xml version='1.0' encoding='UTF-8'?>
<rss version="2.0" xmlns:iffy="http://tech.interfluidity.com/xml/iffy/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    ...
    <iffy:curation>
      <iffy:all/>
    </iffy:curation>
    <iffy:completeness>Content</iffy:completeness>
    ...
  </channel>
</rss>

Here is an example from a single-item feed:

<?xml version='1.0' encoding='UTF-8'?>
<rss version="2.0" xmlns:iffy="http://tech.interfluidity.com/xml/iffy/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    ...
    <iffy:curation>
      <iffy:single/>
    </iffy:curation>
    <iffy:completeness>Content</iffy:completeness>
    ...
  </channel>
</rss>

An ordinary syndicatin feed that includes curation and completeness metadata looks like this:

<?xml version='1.0' encoding='UTF-8'?>
<rss version="2.0" xmlns:iffy="http://tech.interfluidity.com/xml/iffy/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    ...
    <iffy:curation>
      <iffy:recent last="5"/>
    </iffy:curation>
    <iffy:completeness>Content</iffy:completeness>
    ...
  </channel>
</rss>

All-item and single-item feeds, like syndication feeds, are discoverable in <link rel="alternate"> elements of the HTML pages of the blog.

But they are not intended to be discovered by feed readers, which expect short feeds of recent posts for syndication. In order to prevent discovery of these feeds by feed readers, we give them special MIME type in HTML link elements only. These are still RSS feeds. Their "true" MIME type is still application/rss+xml.

But in HTML link elements — only in HTML link elements — I use more specific, less standard MIME types, to throw unwitting feed readers off the scent. These MIME types are application/x-all-item-rss+xml and application/x-single-item-rss+xml.

Here's the HTML link element to a site's all-item feed:

  <link rel="alternate" type="application/x-all-item-rss+xml" title="drafts.interfluidity.com - all items" href="https://drafts.interfluidity.com/all-item-feed/index.rss">

Here's the HTML link element from a post permalink to that post's single-item feed:

  <link rel="alternate" type="application/x-single-item-rss+xml" title="Segmentation fault" href="index.rss">

While ordinary feed readers should ignore these feeds, clients resolving iffy:item-ref tags might find single-item and all-item feeds very useful.

2024-12-22 ⇒
2024-11-15 ⇒

Updated: The 'iffy' XML namespace

A significant update of The 'iffy' XML namespace was made on 2024-11-15 @ 01:10 AM EST.

→ Document iffy:curation, and curation type elements iffy:all, iffy:recent, iffy:selection, iffy:single.

The post was originally published 2024-05-13 @ 04:10 AM EDT.

This is an automatically generated post.
2024-11-12 ⇒

Supporting single-item RSS

This blog now generates single-item RSS alongside each blog post.

This is in preparation for how I mean to support blog-to-blog comments, most directly.

But in general it is offered as an alternative to approaches like microformats.

Microformats exist and other people use them. I feel a bit bad not to adopt them. But my preference is to have meta-data more cleanly separated from displayed HTML, even if it is "POSH". It seems to me like too many concerns are mixed.

If microformats were very widely adopted — particularly if the JVM ecosystem already had well-supported, widely used microformats parsers — I might swallow my own preference and run with them. But I don't think that's the case. I'm going to have to do my own parsing, and I have good tools for that with XML and RSS.

unstatic SimpleBlog instances now support generating single-item RSS. Just override

val generateSingleItemRss : Boolean

to true, and they will appear.

By default, each entry's single-item RSS is placed by converting each whatever.html to whatever.rss, under the same path. (If the leaf of the path does not end with .html, then .rss is simply appended to the full path.)

If you prefer a different scheme, you can override

def singleItemRssSiteRootedFromPermalinkSiteRooted( permalinkSiteRooted : Rooted ) : Rooted

where Rooted is a unstantic.UrlPath.Rooted.

Single-item RSS and the HTML "permalink" for the item should mutually refer to one another.

In the HTML header tag, we add a <link> tag with rel="alternate" to point to the single-item RSS.

We don't want feed readers to subscribe to single-item RSS, so, a bit clumsily, we make up a special content type that feed readers won't recognize, application/x-single-item-rss+xml.

Here's the link you'll find in the source of this post.

<link rel="alternate" type="application/x-single-item-rss+xml" title="Supporting single-item RSS" href="index.rss">

The single-item RSS links back to the HTML as a matter of course, via the standard <link> element within the sole <item>.

Check out the single-item RSS for this post!

Note: If an <atom:link rel="self" /> tag is included in the <channel> element of a single-item RSS stream, the type should be the standard type="application/rss+xml", and definitely not application/x-single-item-rss+xml. (I recommend <atom:link rel="self" /> always be included in the <channel> of an RSS stream, whenever there exists a stable link to the stream.) Single-item RSS is just standard RSS for all purposes other than a feed-reader deciding whether to treat a feed as subscribable.

2024-11-02 ⇒
2024-08-16 ⇒

Updated: Neonix

A significant update of Neonix was made on 2024-08-16 @ 11:20 AM EDT.

→ Add update regarding new location of Bill Mill's "modern unix tool list"

The post was originally published 2024-06-06 @ 03:35 PM EDT.

This is an automatically generated post.
2024-07-01 ⇒

Updated: The 'iffy' XML namespace

A significant update of The 'iffy' XML namespace was made on 2024-07-01 @ 02:40 PM EDT.

→ Major revision of iffy:synthetic, generalize iffy:initial, iffy:update, iffy:update-history, define iffy:uid.

The post was originally published 2024-05-13 @ 04:10 AM EDT.

This is an automatically generated post.
2024-06-22 ⇒

Condensing updates that appear in digests

In the previous post, I claimed to be "finished" my exploration of handling updates.

Ha!

I don't like how my update notifications appear in digests — in e-mail newsletters that gather up all the posts over a period of time.

There are two problems:

  1. Update notifications appear in digests as separate, cookie-cutter posts, which bury and dilute "real" posts.

    On websites, we can make synthetic update posts visually distinct to help guide visitors past them. We could do that in HTML mail digests too, but I think update posts will still put people off actually reading new material when they are going quickly through their e-mail, particularly when updates are more recent and therefore come before the "real" posts.

  2. Updates appear even for posts that are included in the current digest.

    feedletter digests always include the most recently seen version of a post, so from the digest reader's perspective, the "update" notification tells them about nothing that isn't already in the post.

To address this, I mean to do a bunch of things.

  1. I'm going to modify the definition of iffy:synthetic, reversing course on my my current admonition that

    Applications that include iffy:synthetic as a direct child of channel SHOULD NOT also mark individual items as iffy:synthetic, unless there is some meaningful sense in which some items are more synthetic than others. It serves no purpose to mark every item of a feed iffy:synthetic when the channel is already so marked.

    Never mind.

    Until now, iffy:synthetic has been basically just a marker, so there was no point in marking posts twice. Marking at the channel level implied the syntheticness and the types at the item level.

    But now we'll reconceive of iffy:synthetic as also the carrier of the data on which the item content (in description, content:encoded, or atom:content) is based.

    The item content, of course, remains the post's HTML, whether it is a synthetic post or not. But we'll want the information we used to compose synthetic posts to be available, in an iffy:type-dependent way, inside item > iffy:synthetic. So even when a channel is marked synthetic, with a type that implies the type of elements, we will want explicitly to include an iffy:synthetic element in each item.

    For synthethic items of type UpdateAnnouncement — the relevant case here — we've already defined an element that carries most the relevant data, iffy:update.

    We'll also need the guid of the post updated. So we'll bring back iffy:original-guid, with a slightly modified role.

  2. I'll define a feedletter content Customizer that

    1. filters away upates to items already in the post;
    2. combines the remaining updates into a single notification post; and
    3. places that notification post at the end of the digest.

So. We have a plan!

Let's see if it works.

2024-06-20 ⇒

Updated: Sprouted

A significant update of Sprouted was made on 2024-06-20 @ 05:50 PM EDT.

→ Add bold update crediting Maggie Appleton for some of these ideas.

The post was originally published 2024-06-20 @ 03:05 PM EDT.

This is an automatically generated post.