Minimum supported Tailscale client version: v1.64.0

Database integrity improvements

This release includes a significant database migration that addresses longstanding issues with the database schema and data integrity that has accumulated over the years. The migration introduces a schema.sql file as the source of truth for the expected database schema to ensure new migrations that will cause divergence does not occur again.

These issues arose from a combination of factors discovered over time: SQLite foreign keys not being enforced for many early versions, all migrations being run in one large function until version 0.23.0, and inconsistent use of GORM's AutoMigrate feature. Moving forward, all new migrations will be explicit SQL operations rather than relying on GORM AutoMigrate, and foreign keys will be enforced throughout the migration process.

We are only improving SQLite databases with this change - PostgreSQL databases are not affected.

Please read the PR description for more technical details about the issues and solutions.

SQLite Database Backup Example:

# Stop headscale
systemctl stop headscale

# Backup sqlite database
cp /var/lib/headscale/db.sqlite /var/lib/headscale/db.sqlite.backup

# Backup sqlite WAL/SHM files (if they exist)
cp /var/lib/headscale/db.sqlite-wal /var/lib/headscale/db.sqlite-wal.backup
cp /var/lib/headscale/db.sqlite-shm /var/lib/headscale/db.sqlite-shm.backup

# Start headscale (migration will run automatically)
systemctl start headscale

DERPMap update frequency

The default DERPMap update frequency has been changed from 24 hours to 3 hours. If you set the derp.update_frequency configuration option, it is recommended to change it to 3h to ensure that the headscale instance gets the latest DERPMap updates when upstream is changed.

Autogroups

This release adds support for the three missing autogroups: self (experimental), member, and tagged. Please refer to the documentation for a detailed explanation.

autogroup:self is marked as experimental and should be used with caution, but we need help testing it. Experimental here means two things; first, generating the packet filter from policies that use autogroup:self is very expensive, and it might perform, or straight up not work on Headscale installations with a large number of nodes. Second, the implementation might have bugs or edge cases we are not aware of, meaning that nodes or users might gain more access than expected. Please report bugs.

Node store (in memory database)

Under the hood, we have added a new datastructure to store nodes in memory. This datastructure is called NodeStore and aims to reduce the reading and writing of nodes to the database layer. We have not benchmarked it, but expect it to improve performance for read heavy workloads. We think of it as, "worst case" we have moved the bottle neck somewhere else, and "best case" we should see a good improvement in compute resource usage at the expense of memory usage. We are quite excited for this change and think it will make it easier for us to improve the code base over time and make it more correct and efficient.

BREAKING

• Remove support for 32-bit binaries #2692
• Policy: Zero or empty destination port is no longer allowed #2606

Changes

• Database schema migration improvements for SQLite #2617
  • IMPORTANT: Backup your SQLite database before upgrading
  • Introduces safer table renaming migration strategy
  • Addresses longstanding database integrity issues
• Add flag to directly manipulate the policy in the database #2765
• DERPmap update frequency default changed from 24h to 3h #2741
• DERPmap update mechanism has been improved with retry, and is now failing conservatively, preserving the old map upon failure. #2741
• Add support for autogroup:member, autogroup:tagged #2572
• Fix bug where return routes were being removed by policy #2767
• Remove policy v1 code #2600
• Refactor Debian/Ubuntu packaging and drop support for Ubuntu 20.04.

Changelog

• 0512f7c57ec0e0e721132dc4efdd371de9952fcd .github/ISSUE_TEMPLATE: add node number to environment
• 05996a50483d576722875be8e8c30cb9b285d8d7 .github/workflow: only run a few selected postgres tests
• f6c4b338fd8cf21cf9401b3e213778a86702c4a8 .github/workflows: add generate check
• 5ba712041809d3416d09ccc0bcc77d2357ffd31f .github/workflows: prettier
• 4a8d2d9ed30d17db77b5703a0aeaef9c11e05eee .github/workflows: reduce integration retry to 3
• 7f8b14f6f3e0267be5a6905abff2e4495b9f5a31 .github/workflows: remove integration retry
• e949859d33fc0888818088d5047673561bdf1b81 Add DERP docs
• bd35fcf338d678314fc5ef674d318f441fdf6fb6 Add FAQ entry about policy migration in the database
• 30d12dafed210316431a57349adfdb2128078feb Add FAQ entry about the recommended upgrade path
• bcd80ee7733fe7d9a8cee9a96a0857046c805697 Add debugging and troubleshooting guide
• 76ca7a2b5005da9c8c03f8353f73975e987b55a1 Add headscale-console
• 98fc0563acfd6b58b059e3b0f5e38134138ea998 Bump version in docs
• 33e9e7a71f8fe5925fca8597a10a9b16192ff3d7 CLAUDE: split into agents
• 3f72ee9de8d38e28bc4059fbf3762809e07a0c00 Clarify SIGHUP log message (#2661)
• 51c6367bb13c63112cb5836c98906add127638e1 Correctly document the default for dns.override_local_dns
• 2f3c365b68fe23dbd53110b505f8a8a70d100c0e Describe how to remove a DERP region
• 49b3468845576c3db0970edbb9ac2a9be78ee576 Do not ignore config-example.yml
• c15aa541bb9e8d834a3e2d9c12d8edacae7d8502 Document HEADSCALE_CONFIG
• b50e10a1be94898e3f237fee35e9f57eea06eda8 Document breaking change for dns.override_local_dns
• 30cec3aa2b422a9d8184e47a747598fbe2f9f569 Document ports in use
• c04e17d82ee5925df13727a3343822a5ef85af9d Document valid log levels
• cd704570be8672e16a036a88b56b45b4b82d7c80 Drop support for Ubuntu 20.04
• 43c9c50af4d0010a0f2cd665769bb3033f16790c Drop syslog.target and systemd-managed /var/run
• be337c6a33b64687c23c227038a1d45ad86469f3 Enable derp.server.verify_clients by default
• e73b2a9fb9db82ea5dd1a1a5d554a585188b6b21 Ensure that a username starts with a letter (#2635)
• fa619ea9f3f73200bf17b5d5abb3ecded1aa349f Fix CHANGELOG for autogroup:member and autogroup:tagged (#2733)
• 086fcad7d958e1f94b172e2dd25f89ccdc3e1955 Fix Internal server error on /verify (#2735)
• bad783321e8ea0c77bd00933bfd5d18ec8e05bca Fix /machine/map endpoint vulnerability (#2642)
• 46c59a3fff6ca23e3f2676541e76fd85aad7fbb6 Fix command in bug report template
• a8f2eebf6637049756b23c9ffa8c698cf54f6b4d Fix config param name in TLS doc
• e7fe645be5a6f22ca8b3b1dded79de681ac0d980 Fix invocation of golangci-lint (#2703)